Skeema Versions Save

Declarative pure-SQL schema management for MySQL and MariaDB

v1.11.2

1 month ago
  • MySQL 8.3 is now marked as supported (31239e7)
  • MariaDB 11.3 is now marked as supported (c05a1fa)
    • In MariaDB 11.3+, modifying a column's data type from inet4 to inet6 is no longer considered unsafe
  • Several improvements and edge case fixes for Docker workspaces, especially when using an arm64 system (e.g. Apple Silicon)
    • When copying the sql_mode from the target database server and applying it to the Dockerized instance, if the flavors/versions of the two databases differ, incompatible sql_mode values are now removed automatically (686294e)
    • When using an arm64 client system with a MySQL 5.x requested flavor, utf8mb4 columns now use the old MySQL 5.x collation default in the Dockerized instance, despite it actually running MySQL 8.0 for arm64 compatibility (48d2074)
    • When using an arm64 client system with a MySQL 5.x requested flavor, utf8mb3 columns could cause an uncaught panic due to a logic bug, which is now fixed (58e9338, #226)
    • Percona Server 8.0, 8.1, 8.2 images are now available regardless of CPU architecture; previously 8.0 was only available for amd64, and 8.1+ was unavailable entirely (f7753c1, e50f754, da0e8e1)
    • If the requested flavor value is a release series newer than the current version of Skeema, a non-fatal warning is now logged; previously this caused a fatal error (a45dbf2)
  • Logging improvement: word wrapping is now disabled automatically if STDERR is a narrow terminal window below 80 characters in width (4ab585c)
  • MariaDB bug fix: when using foreign key constraints in MariaDB 10.6+, Skeema's workspace cleanup step could sometimes stall due to database server bug MDEV-32899; this condition is now avoided by using a low innodb_lock_wait_timeout and retries during workspace cleanup (a5ed51d)
  • MariaDB bug fix: in lint-reserved-word, erroneous handling of two MariaDB-specific reserved words has been corrected (0c0b8a8)
  • Premium edition bug fix: when using SSH tunnel functionality to connect to a server that only has an RSA host key, previously the rsa-sha2-512 or rsa-sha2-256 algorithms would not be attempted by Skeema during the handshake. This is now fixed, thanks to a community contribution to our open source knownhosts wrapper package. (https://github.com/skeema/knownhosts/pull/6)

Thank you to all code contributors and issue reporters!

An installation guide and full documentation are available on our website skeema.io.

v1.11.1

5 months ago
  • MySQL 8.2 is now supported (0346508d)
    • The new SET_ANY_DEFINER privilege is now properly handled by the strip-definer=auto Premium option
  • MariaDB 11.2 is now supported (36113596f)
    • MariaDB 11.2 extends its built-in online schema change support to include ALTER TABLE ... ALGORITHM=COPY, LOCK=NONE. As always, Skeema users can enforce a minimum algorithm/lock for generated ALTER TABLE statements by using Skeema's alter-algorithm and alter-lock options. However, users should be aware that the database server's built-in OSC can cause replication lag. Enabling MariaDB's binlog_alter_two_phase server variable may help reduce or prevent this lag.
  • Performance improvement for skeema diff and skeema push: the verify option, which is enabled by default, now ignores cosmetic table discrepancies which are excluded from ALTER TABLE generation anyway. This provides a major performance benefit in environments with many auto-increment tables or range-partitioned tables, among other situations. (bcd46d4)
  • Performance improvements for the workspace=docker option (22d4b9b, c029294, 766efc3)
    • Newly-created containerized databases now use a smaller buffer pool, use smaller redo log files, disable performance_schema, disable the adaptive hash index, and disable separate log writer threads. These changes improve performance for the workspace workload, and reduce system resource consumption.
    • On Linux systems, when docker-cleanup=destroy is configured, the ephemeral containerized database will now automatically use a tmpfs mount for its data directory. This greatly improves performance by avoiding disk writes for the workspace workload.
    • These changes do not affect pre-existing containers which were created from prior use of workspace=docker. You can manually remove older containers (for example docker rm -v -f skeema-mysql-8.0) if you wish for Skeema to recreate them with the new tuned settings.
  • Two problematic variants of CREATE TABLE statements are now simply ignored by all Skeema commands entirely; a warning is logged in these situations (62c33e0)
    • CREATE TABLE ... SELECT statements in the filesystem previously caused their entire subdir to be skipped with a fatal error
    • CREATE TABLE ... WITH SYSTEM VERSIONING (MariaDB-only database feature) statements in the filesystem previously would be erroneously removed by skeema pull (#187, #206)
  • Bug fix for MariaDB 10.10+: tables using the new UCA-14.0.0 collations, such as utf8mb4_uca1400_ai_ci, are now introspected properly. Previously, these collations were not handled correctly by Skeema because they are represented differently in information_schema.collations than all other collations. (ee25133)
  • Internal improvement for Skeema developers/contributors: integration test suite output is now substantially less noisy, suppressing all output from subtests which pass (5d3d1b5)

v1.11.0

7 months ago

See the Skeema v1.11 release announcement blog post for a summary of the most important changes in this release.

  • Spatial indexes and SRID column attributes are now fully supported for diff operations. (#97, 77ab3de, 5e357db)
  • MySQL 8.1 is now supported. (23520ea, f4e480f)
  • MariaDB 11.1 is now supported. (865aa62)
  • Premium edition improvement for triggers: whenever skeema push needs to perform multiple trigger manipulations in the same schema, the corresponding parent tables are now briefly write-locked, to prevent data inconsistencies from writes which occur in the middle of the push operation.
    • Even though MySQL lacks CREATE OR REPLACE syntax to atomically modify a trigger, this improvement now makes it safe to modify existing triggers, so allow-unsafe is no longer necessary in this situation.
  • The workspace=docker option now shells out to an external Docker CLI binary, instead of bundling a Golang Docker client library inside of Skeema. This change reduces the size of the skeema binary by 25-40%, and avoids spurious false-positive security warnings related to dependencies of the Docker client library. (#186, 2a2d8e3, 233c73e)
  • New option lax-column-order provides a way to ignore differences in column ordering in skeema diff and skeema push. (#178, c6de431)
  • New option lax-comments provides a way to ignore COMMENT clause differences in skeema diff and skeema push. (76a7323)
  • When skeema diff or skeema push detects multiple unsafe changes, all such unsafe changes are now logged, instead of just the first one. Additionally, linting still occurs if unsafe changes are present, allowing the user to see all problematic unsafe statements and linter violations at the same time. (01a93d5)
  • Several additional column modification edge cases are now considered unsafe by skeema diff and skeema push. (905ee14)
    • Changing the collation of a column that appears in a uniqueness constraint
    • Adding, removing, or changing the SRID of a spatial column in MySQL 8+
    • Converting the column type between TIME and TIMESTAMP
  • Several small improvements when modifying existing routines when using skeema diff or skeema push
    • Changes to the parameters or return type of a routine are now considered unsafe, requiring allow-unsafe to proceed even in MariaDB (where atomic CREATE OR REPLACE syntax normally makes other types of routine modifications safe). (a3ecc18)
    • When modifying a routine's characteristics (COMMENT clause, SECURITY characteristic, SQL DATA characteristic) and no other properties of the routine, Skeema now emits ALTER PROCEDURE / ALTER FUNCTION, which is safe and atomic even in MySQL. (6fab6d8)
    • In the output for skeema diff and skeema push, if the last statement for a schema required use of a nonstandard DELIMITER, the output will now also include a DELIMITER ; line before moving on to the next schema. This is beneficial for users who pipe skeema diff STDOUT to another program and then add additional SQL statements afterwards. (605b735)
  • Linter annotation messages now backtick-wrap the name of the problematic table, routine, view, or trigger. (2dd1fb6)
  • Premium edition improvement for Windows build: when Skeema needs to invoke external commands with PowerShell, its -EncodedCommand option is now used in order to prevent problems with tricky nested escaping of quoted parameters.
  • Premium edition Aurora bug fix: the latest release of Aurora for MySQL 5.7 (Aurora v2.12.0, from 2023-07-25) was not properly detected as flavor=aurora. This version of Aurora self-reports its MySQL server version as 5.7.40, whereas all previous Aurora v2 versions reported it as MySQL 5.7.12. Skeema's Aurora detection logic has now been adjusted to allow for dynamic 5.7 versions, similar to how the logic for Aurora v3 (MySQL 8.0.x) already worked.
  • Premium edition bug fix: when using Skeema Premium's ssh tunnel functionality on a server with an IPv6 address, if the SSH configuration specifies StrictHostKeyChecking=no or StrictHostKeyChecking=ask, Skeema could previously write the knownhosts entry incorrectly, in part due to an upstream Golang bug. Workarounds are now in place, thanks to community contributions to our open source knownhosts wrapper package. (https://github.com/skeema/knownhosts/pull/2, https://github.com/skeema/knownhosts/pull/4)

Upgrade notice for users of Skeema's workspace=docker option:

  • Skeema v1.11+ no longer bundles a built-in Docker client. A docker command-line client binary must be available in your $PATH to use workspace=docker. If you use this setting while also running skeema inside of a container, you must ensure that skeema's container has a working docker binary prior to upgrading. See #186 for more information.

v1.10.1

10 months ago
  • MariaDB 11.0 is now supported. (963f0e3, 2b112b6)
  • skeema diff and skeema push can now generate supported ALTER TABLE statements even for tables that already have unsupported features, as long as the new modification to the table does not affect those unsupported features. (#214, 7b5e86d)
    • For example, these commands can now add columns or indexes to pre-existing tables that already happen to use unsupported features such as spatial indexes, sub-partitioning, application time periods, etc. (Previously, Skeema could not generate an ALTER TABLE for such tables under any circumstances.)
    • In these cases, Skeema always verifies that the generated DDL has the correct effect by testing it in a workspace, even if skip-verify has been configured.
    • This verification step is conservative, and some edge cases may fail verification, particularly on MySQL 8 tables with non-default collations. In that situation, the operation remains marked as unsupported and the table is skipped with a warning message.
  • When using workspace=docker, Skeema will attempt to automatically use a session sql_mode equivalent to that of your "real" database server (assuming a database host is configured and reachable). This solves common strict-mode problems on database systems that often use a nonstandard sql_mode, such as AWS RDS. (cfc11a0)
  • Bug fix: On Mac and Windows client systems, a change in Skeema v1.10.0 inadvertently caused skeema init and skeema pull to always downcase the name of any newly-created .sql files. This has now been fixed, and the casing of the original table/object will be retained when naming the .sql file. (f0e52b7)
    • Regardless of this change, users can always freely rename .sql files or move statements between .sql files in the same directory. Skeema never requires that you keep statements in the exact location that skeema init or skeema pull put them in.
  • Bug fix: a regression in Skeema v1.8.1 inadvertently reduced the effectiveness of the strip-definer option specifically for procs/funcs, when using an unprivileged database user and workspace=temp-schema. In that situation, if the CREATE FUNCTION or CREATE PROCEDURE statement in the .sql file contained a DEFINER clause, workspace errors could result. This has now been fixed, and DEFINER clauses are now properly stripped from procs and funcs. (The bug did not affect views or triggers; those already worked properly and this remains the case.)
  • Bug fix: On Windows, when skeema pull needs to delete a .sql file because the corresponding table/object no longer exists in the database, file locking errors could sometimes occur on Skeema v1.10.0. This has now been fixed. (061f478)
  • Improvements to logging and error messages in several situations:
    • When host and/or schema are not supplied in the appropriate location, or host-wrapper doesn't output any hostnames (0825d1a, 1055b2d)
    • Common mistakes with the password option (80635d9)
    • Workspace locking conflicts and timeouts (f8f0745)
    • Problems with the connect-options configuration (4d218c1)

v1.10.0

1 year ago

For a summary of the most important changes in this release, see the Skeema v1.10.0 release announcement blog post.

  • The Premium edition CLI now supports management of seed data using INSERT statements, for populating new tables or spinning up new environments. (#22)
    • The new inserts option controls how skeema push handles INSERT statements found in your *.sql files. The default setting, inserts=seed, will execute INSERTs whenever the target table is empty.
    • skeema pull's new update-seed-inserts option provides a mechanism for selectively dumping row data from an environment using view logic, automatically creating or updating INSERT statements in your *.sql files.
  • All logic for processing .sql files has been completely rewritten. The new implementation provides an 8x to 40x speedup on parsing time, while also using substantially less memory. This was motivated by the requirement to handle arbitrarily-large seed data INSERT statements in the Premium edition, but the performance improvements are generally beneficial to all users, and are present in the Community edition as well. (e625c97)
  • MariaDB 10.11 is now marked as supported by Skeema. (5b0cc10)
  • New linter rule lint-pk-type provides a way to restrict which column data types may be used in primary keys. The list of allowed data types is configurable using the new allow-pk-type companion option. (#204, #205)
  • New linter rule lint-reserved-word flags names of identifiers (tables, columns, routines, etc) which match reserved words in any known version of your database server. (#120, #207, 4c2ddd0)
  • Linter improvement: lint-has-time now provides more descriptive messages that differ for timestamp vs datetime/time. The timestamp message now includes a warning about Y2K38 problems with this type, in addition to other improvements. (#209, #210)
  • skeema diff --brief now automatically disables linter checks and omits all INFO-level logging. These changes are designed to make the output more easily human-readable, while still surfacing any unusual errors or warnings. (310164a)
  • If STDERR isn't a terminal (e.g. being piped to another program or redirected to a file), all error/warning linter annotation output from skeema lint, skeema diff, and skeema push now omits embedded newlines. Previously, each annotation message could span multiple lines, which made automated parsing and processing more difficult. Now, each individual annotation message will consist of a single line when STDERR isn't a terminal. (#211)
  • Bug fix: previously, skeema init would hang for a long period if a large .sql file was present in the working directory, for example mysqldump output from a large database. The contents of the working directory no longer affect skeema init. (#190, 2481bed)
  • Bug fix: Partitioned tables using the LIST COLUMNS partitioning method were previously unsupported for diff operations in many cases. Now most LIST COLUMNS tables are supported, as long as the partitioning key column names don't conflict with MySQL keywords. (#199, 3313bfc)
  • Bug fix: On MySQL 8, if a table contained at least one BINARY or VARBINARY column with a literal (non-expression) DEFAULT value, previously the table could be marked as unsupported for diff operations due to a MySQL information_schema bug. Skeema now implements a workaround for this MySQL bug. (0177ef9)

Thank you to all code contributors and issue reporters!

v1.9.0

1 year ago

For a summary of the most important changes in this release, see the Skeema v1.9.0 release announcement blog post.

  • Environment variables may be used as option values in .skeema option files for options user, password, host, schema, port, socket, and temp-schema. This provides a flexible mechanism for dynamic configuration. (16b54e4, #101)
  • Premium CLI now includes the ability to offload the temp-schema workspace to a different database server, via the new temp-schema-environment option. The option value is set to the name of an environment (.skeema file section) which can override some or all connectivity options (host, port, socket, user, password, host-wrapper, SSH options, SSL options, etc) for determining what host to place the temp-schema workspace on. (#94)
  • Improvements to the various ignore-* options (ignore-table, ignore-proc, ignore-func, ignore-view, ignore-trigger)
    • The ignore-proc and ignore-func options, which previously were only available in the Premium edition (since v1.6.0-premium one year ago), are now available in the Community edition as well. (0de3c57)
    • The ignore options now take effect earlier in the processing flow for all commands. Workspaces skip ignored statements entirely now, improving performance and ensuring that execution cannot be affected by SQL errors from ignored statements. (b35f065)
  • Several MariaDB-specific improvements and fixes
    • MariaDB 10.10 is now fully supported, including special conversion logic for the new inet4 column type. (53de093)
    • MariaDB's November 2022 point releases (MariaDB 10.3.37, 10.4.27, 10.5.18, 10.6.11, 10.7.7, 10.8.6, 10.9.4) are now supported by Skeema. These MariaDB releases unexpectedly changed the output logic for collations in SHOW CREATE TABLE, which broke compatibility with a core safety check in Skeema. (f2dcaf8, #193, #194, #195)
    • When modifying existing procs/funcs in MariaDB, atomic CREATE OR REPLACE syntax is now used. This functionality was originally introduced in Skeema v1.6.0-premium one year ago, but is now present in the Community Edition as well. (412a326)
  • When interacting with multiple hosts (running skeema from a parent directory), it is now possible to interactively prompt for different passwords for each host, by configuring the bare password option with no =value in the same .skeema file as the host. (ab4a84c, 5e35143, b004616)
    • Previously, bare password option with no =value was only supported in global option files. This still works, and still prompts for a single "global" password rather than host-specific ones.
    • For an alternative mechanism for passing different passwords to each host, also see the new ENV variable support, mentioned above.
  • When using workspace=docker with MySQL 8, whenever Skeema creates a new ephemeral containerized DB, binary logging will be disabled automatically, despite MySQL 8 normally defaulting to enabling binlogging. This improves workspace performance, and prevents potential issues with non-deterministic CREATE FUNCTION definitions. (2804a44)

v1.8.2

1 year ago
  • MariaDB 10.9 is now supported. (9262e29)
  • When using an interactive password prompt (--password or -p with no value), the prompt text is now normally displayed on STDERR instead of STDOUT, so that it is visible even if STDOUT is being redirected to a file. This also helps ensure that STDOUT only contains valid SQL, which can be piped/redirected to other client programs as-is if desired. (c0267f4, #188)
  • MySQL 8 cross-schema foreign key safety improvement: when using the default of workspace=temp-schema, if any foreign keys cross schema boundaries (the referenced "parent" table is in a different database than the "child" table defining the FK constraint), this can cause metadata locking conflicts in MySQL 8, since the database extends metadata locks to both sides of the FK. Skeema now automatically applies a low session-level lock_wait_timeout for its workspace=temp-schema operations in MySQL 8 to limit any performance impact from such metadata locking conflicts. Note that only cross-schema FKs (and not same-schema FKs) are affected by the root issue, and only under MySQL 8 and workspace=temp-schema. See commit message for more details. (762da89)
  • Bug fix: a nil pointer panic could occur if the network connection to the DB was extremely unstable/flaky (initial connections succeeded, but some subsequent connection attempts in the same operation failed in one particular code path). Skeema will now gracefully handle this situation with a normal error message. (d07d106)

v1.8.1

1 year ago
  • Tables with explicit TABLESPACE innodb_system or TABLESPACE innodb_file_per_table clauses are now supported for diff operations. These clauses permit overriding the server's innodb_file_per_table setting on an individual table basis. This syntax is only available in MySQL 5.7+, and is relatively rare. (9ed3f60)
  • MySQL 8 bug fix: under some rare conditions, tables can report different next AUTO_INCREMENT value in information_schema vs SHOW CREATE TABLE, even with information_schema_stats_expiry=0 which Skeema automatically uses for its sessions. When this occurred, tables were unsupported for diff operations. To solve this, Skeema now relies on the SHOW CREATE TABLE value exclusively, and these tables are now properly supported for diffs. (677d8de)
  • Internal concurrency improvements should provide performance benefits in some scenarios, such as running Skeema with high network latency, a large count of procs/funcs, and/or operating on multiple schemas at once. (f9a6bb2)
  • When Skeema is unable to generate diff DDL for a table, the extra information logged by --debug now makes it clearer which side of the diff was problematic. A similar improvement has been made to the output of --verify in the rare case of verification failing. (#52, 677d8de)
  • Whenever skeema pull or skeema add-environment needs to rewrite a .skeema file, some option values are now rewritten differently:
    • String or enum type options that have been set to an empty-string value are now rewritten to use the "skip" prefix instead. For example, lint-pk='' is now rewritten as skip-lint-pk. These configurations have the exact same effect (and in this specific example both are equivalent to lint-pk=ignore), but the empty-string value was less clear. (cf19abd)
    • Premium CLI bug fix: If a .skeema file contained strip-definer with no value (implying =true), previously these commands would incorrectly rewrite this as strip-definer= (empty value implying false). Now fixed.
  • In a sharded environment in which shards are running different server versions (e.g. some shards on MySQL 8.0.28 and others on 8.0.30), differences in the server's metadata representation of the legacy three-byte "utf8" vs "utf8mb3" charset (and corresponding collations) are now ignored by default in skeema diff and skeema push. (894530f)
  • Some unit tests and integration tests required minor improvements to handle MySQL 8.0.30's metadata changes regarding legacy three-byte utf8mb3 collations. Aside from test suite adjustments, Skeema itself was already fully compatible with MySQL 8.0.30. (28d2b0a)

v1.8.0

1 year ago

For a summary of the most important changes in this release, see our blog post.

  • Premium CLI now includes built-in support for SSH tunnels via the new ssh and ssh-to-db options. (#119)
    • These options cover a wide range of SSH use-cases: bastion security servers / jump boxes, localhost admin database connections, running online schema change scripts from a unified Linux server, and more.
    • Fully supports several OpenSSH configuration options from ~/.ssh/config, interacting with OpenSSH ssh-agent via UNIX socket (Linux and MacOS) or named pipe (Windows), known_hosts management, automatic keep-alives, multi-hop tunnels without need for agent forwarding, and more.
  • MariaDB 10.8 is now fully supported, including descending indexes (DESC index parts), as well as functions with IN / OUT / INOUT param qualifiers (1454f21)
  • Skeema now fully supports database servers running on Windows or MacOS, regardless of object name casing or the database server's lower_case_table_names setting
    • Previously, Skeema had issues with database servers using lower_case_table_names=1 (Windows default) or lower_case_table_names=2 (MacOS default) if any table, view, or database names used uppercase characters. These issues -- which related to the database's special handling of case-insensitive filesystems -- are now fixed, and Skeema's integration testing suite covers these situations. (#65, bdc279e)
    • New linter option lint-name-case can be used to enforce an all-lowercase table/view naming scheme, if desired for portability reasons (8cb30d0)
    • Premium CLI contains view-specific fixes relating to name casing, in addition to the fixes already made in v1.7.1.
  • MySQL 8 bug fix: due to a regression in Skeema v1.7.1, tables with nonstandard default collations (meaning the table's default charset has a different default collation than the table's chosen default collation) would inadvertently cause failures in the diff verify step. This was caused by unusual/inconsistent behavior in MySQL 8's SHOW CREATE TABLE and is now fixed by using a deeper structural comparison in the verify logic. Thank you to Etsy for the report! (#184, 8bde55a)
  • skeema pull now logs several problem conditions -- query failures, file I/O errors, etc -- as an error with exit code 2, instead of a warning with exit code 1. This is more consistent with how other commands handle these conditions. (2d3d7c4)
  • Other minor internal refactors and test suite fixes (0d5d476, d4022fc)

v1.7.1

2 years ago
  • Premium CLI bug fix: When the database server uses lower_case_table_names=1 (e.g. server running on Windows, including Azure Database for MySQL) or lower_case_table_names=2 (server running on MacOS), and CREATE VIEW statements in *.sql files had view names containing uppercase characters, nil pointer panics would occur in most Skeema commands due to case sensitivity mismatches. View name casing is now handled correctly even with lower_case_table_names=1 or 2.
  • Premium CLI bug fix: In MySQL 8, if a CREATE VIEW statement supplied an explicit column list prior to the SELECT, the view was inadvertently unsupported by Skeema. This was caused by MySQL 8’s inclusion of these column list overrides in SHOW CREATE VIEW, unlike previous MySQL versions and all MariaDB versions. This view syntax (which is fairly uncommon) is now fully supported for all database versions.
  • Premium CLI license activation key timeout has been increased, and if a timeout occurs, the error is now clearer.
  • Bug fix: Tables containing generated column expressions using LIKE, or check constraints using LIKE, were inadvertently unsupported due to a logic error in Skeema’s previous prohibition of CREATE TABLE ... LIKE statements. (#181, 6cbff9c)
    • CREATE TABLE ... LIKE is now permitted for use in .sql files, and works properly in skeema diff and skeema push as long as the target table is a normal table in the same schema. The statement is evaluated dynamically, meaning any changes in the target table will be reflected properly in diffs for any tables that point to it.
    • skeema pull, skeema lint, and skeema format may overwrite CREATE TABLE ... LIKE with a normal non-dynamic CREATE TABLE. This functionality may be improved in a future release.
  • An upcoming release of MySQL 8 is expected to replace occurrences of "utf8" with "utf8mb3" in information_schema and SHOW command output. Skeema charset logic has now been improved to be more robust, in order to ensure compatibility with this upcoming change. (a03718c)
  • When manually creating or editing *.sql files, identifiers may now contain non-Latin unicode characters without requiring backticks around the identifier. Skeema’s identifier quoting rules should now match MySQL and MariaDB properly. (3e40b6c)
  • workspace=docker now logs a warning if docker-cleanup=stop or docker-cleanup=destroy fails, for example due to the Docker daemon returning an error upon attempting to stop or remove the container. (5e971e0)

An installation guide and full documentation are available on our website skeema.io.