Liquibase Percona Extension
Extension to support the tool pt-online-schema-change
from Percona Toolkit.
This extension replaces a couple of the default changes to use pt-online-schema-change
instead of SQL.
This allows to perform a non-locking database upgrade.
MySQL and MariaDB (since 4.3.2) are the only supported databases. The extension checks whether it is being run against a MySQL/MariaDB database. If not, it falls back to the default changes provided by liquibase-core.
This liquibase extension supports the following changeset formats:
The following changes are supported:
Since: liquibase-percona 1.0.0
Automatic rollback supported? yes
Example:
<changeSet id="2" author="Alice">
<addColumn tableName="person">
<column name="address" type="varchar(255)"/>
</addColumn>
</changeSet>
Corresponding command:
pt-online-schema-change --alter="ADD COLUMN address VARCHAR(255)" ...
Since: liquibase-percona 1.3.0
Automatic rollback supported? yes
Example:
<changeSet id="3" author="Alice">
<addForeignKeyConstraint constraintName="fk_person_address"
referencedTableName="person" referencedColumnNames="id"
baseTableName="address" baseColumnNames="person_id"/>
</changeSet>
Corresponding command:
pt-online-schema-change --alter="ADD CONSTRAINT fk_person_address FOREIGN KEY (person_id) REFERENCES person (id)" ...
Since: liquibase-percona 1.7.0
Automatic rollback supported? no
Example:
<changeSet id="2" author="Alice">
<addPrimaryKey tableName="person" columnNames="id, name"/>
</changeSet>
Corresponding command:
pt-online-schema-change --alter="DROP PRIMARY KEY, ADD PRIMARY KEY (id, name)" ...
Note: When the table has already a primary key, a "DROP PRIMARY KEY" statement is added to the
alter command first. By default, the pt-online-schema-change will not execute this change,
you have to set the additional option --no-check-alter
first
(see check-alter).
Make sure to read this section completely.
In order to figure out, whether a primary key exists already (and therefore the DROP PRIMARY KEY statement is needed), a database connection is required. This means, the generated migration SQL will be wrong (it only contains the ADD PRIMARY KEY statement).
Automatic rollback is not supported by this percona change (as opposed to the plain liquibase addPrimaryKey change). pt-osc usually needs a primary key or a unique key in order to operate properly. If the table has no such keys, it most likely will refuse to operate.
Since: liquibase-percona 1.3.0
Automatic rollback supported? yes
Example:
<changeSet id="2" author="Alice">
<addUniqueConstraint columnNames="id, name" tableName="person" constraintName="uq_id_name"/>
</changeSet>
Corresponding command:
pt-online-schema-change --alter="ADD CONSTRAINT uq_id_name UNIQUE (id, name)" ...
Note: pt-online-schema-change is executed with the option --nocheck-unique-key-change
. This enables to
add a unique index, but can cause data loss, since duplicated rows are ignored.
See Percona Toolkit Documentation
for more information.
Since: liquibase-percona 1.2.0
Automatic rollback supported? yes
Example:
<changeSet id="2" author="Alice">
<createIndex indexName="emailIdx" tableName="person" unique="true">
<column name="email"/>
</createIndex>
</changeSet>
Corresponding command:
pt-online-schema-change --alter="ADD UNIQUE INDEX emailIdx (email)" ...
Note: pt-online-schema-change is executed with the option --nocheck-unique-key-change
. This enables to
add a unique index, but can cause data loss, since duplicated rows are ignored.
See Percona Toolkit Documentation
for more information.
Since: liquibase-percona 1.0.0
Automatic rollback supported? no
Example:
<changeSet id="2" author="Alice">
<dropColumn tableName="person" columnName="age"/>
</changeSet>
Corresponding command:
pt-online-schema-change --alter="DROP COLUMN age" ...
Since: liquibase-percona 1.3.0
Automatic rollback supported? no
Example:
<changeSet id="4" author="Alice">
<dropForeignKeyConstraint baseTableName="address" constraintName="fk_person_address" />
</changeSet>
Corresponding command:
pt-online-schema-change --alter="DROP FOREIGN KEY _fk_person_address" ...
Since: liquibase-percona 1.3.0
Automatic rollback supported? no
Example:
<changeSet id="3" author="Alice">
<dropUniqueConstraint tableName="person" constraintName="uq_id_name"/>
</changeSet>
Corresponding command:
pt-online-schema-change --alter="DROP KEY uq_id_name" ...
Since: liquibase-percona 1.2.0
Automatic rollback supported? no
Example:
<changeSet id="3" author="Alice">
<dropIndex indexName="emailIdx" tableName="person"/>
</changeSet>
Corresponding command:
pt-online-schema-change --alter="DROP INDEX emailIdx" ...
Since: liquibase-percona 1.2.0
Automatic rollback supported? no
Example:
<changeSet id="2" author="Alice">
<modifyDataType tableName="person" columnName="email" newDataType="VARCHAR(400)"/>
</changeSet>
Corresponding command:
pt-online-schema-change --alter="MODIFY email VARCHAR(400)" ...
Since: liquibase-percona 4.19.1
Automatic rollback supported? no
Example:
<changeSet id="2" author="Alice">
<sql>ALTER TABLE person ADD COLUMN address VARCHAR(255) NULL</sql>
</changeSet>
Corresponding command:
pt-online-schema-change --alter="ADD COLUMN address VARCHAR(255) NULL" ...
Note that there are several limitations:
;
).
In that case a warning will be logged and the sql change will be executed as usual
without the percona toolkit.liquibase.percona.skipChanges
.Each change allows to enable or disable the usage of percona toolkit via the property usePercona
.
By default, the percona toolkit is used, see also the system property liquibase.percona.defaultOn
.
Example:
- changeSet:
id: 2
author: Alice
changes:
- addColumn:
tableName: person
usePercona: false
columns:
- column:
name: address
type: varchar(255)
This flag exists since liquibase-percona 1.3.0
It is supported by using the YAML format and since liquibase 3.6.0, you can use it in XML changesets, too:
<addColumn tableName="person"
xmlns:liquibasePercona="http://www.liquibase.org/xml/ns/dbchangelog-ext/liquibase-percona"
liquibasePercona:usePercona="false">
<column name="address" type="varchar(255)"/>
</addColumn>
Since liquibase-percona 4.19.1, you can use it in SQL changeset as follows:
--changeset Alice:2
--liquibasePercona:usePercona="false"
ALTER TABLE person ADD address VARCHAR(255) NULL;
Each change allows to specify options that are used when executing pt-osc. If specified, this option
overrides the system property liquibase.percona.options
. If not specified, then
the system property will be used.
Example:
- changeSet:
id: 2
author: Alice
changes:
- addColumn:
tableName: person
perconaOptions: "--alter-foreign-keys-method=auto"
columns:
- column:
name: address
type: varchar(255)
This flag exists since liquibase-percona 2.0.0.
It is supported by using the YAML format and in XML changesets:
<addColumn tableName="person"
xmlns:liquibasePercona="http://www.liquibase.org/xml/ns/dbchangelog-ext/liquibase-percona"
liquibasePercona:perconaOptions="--alter-foreign-keys-method=auto">
<column name="address" type="varchar(255)"/>
</addColumn>
Since liquibase-percona 4.19.1 this is also supported in SQL changesets:
--changeset Alice:2
--liquibasePercona:perconaOptions="--alter-foreign-keys-method=auto"
ALTER TABLE person ADD email VARCHAR(255) NULL;
According to the Liquibase Documentation for XML Changelogs
an extra XSD for the extension is not mandatory as there an "allow all" default schema (dbchangelog-ext.xsd).
But having an explicit schema helps in IDE support (e.g. autocomplete, validation).
Therefore, this extension provides an official XSD for the attributes it supports on the changes (usePercona
and perconaFlags
).
Example usage:
<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:liquibasePercona="http://www.liquibase.org/xml/ns/dbchangelog-ext/liquibase-percona"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-latest.xsd
http://www.liquibase.org/xml/ns/dbchangelog-ext/liquibase-percona https://raw.githubusercontent.com/liquibase/liquibase-percona/main/src/main/resources/dbchangelog-ext-liquibase-percona.xsd">
<changeSet id="1" author="alice">
<addColumn tableName="person" liquibasePercona:usePercona="false">
<column name="address" type="varchar(255)"/>
</addColumn>
</changeSet>
</databaseChangeLog>
If you use as the URL exactly https://raw.githubusercontent.com/liquibase/liquibase-percona/main/src/main/resources/dbchangelog-ext-liquibase-percona.xsd
,
then liquibase doesn't retrieve the XSD file from the internet but uses the locally provided file inside
liquibase-percona.jar
instead.
Otherwise, you need to set the system property liquibase.secureParsing
to false, so that liquibase
downloads the extension schema from the internet in order to validate the XML changelog file.
The extension schema is available in the repository: dbchangelog-ext-liquibase-percona.xsd.
The extension supports the following java system properties:
liquibase.percona.failIfNoPT
: true/false. Default: false.
If set to true, the database update will fail, if the command pt-online-schema-change
is not found.
This can be used, to enforce, that percona toolkit is used.
liquibase.percona.noAlterSqlDryMode
: true/false. Default: false.
When running updateSQL or rollbackSQL in order to generate a migration SQL file, the command line, that would
be executed, will be added as a comment.
In addition, the SQL statements (as produced by liquibase-core) will also be generated and output into the migration
file. This allows to simply execute the generated migration SQL to perform an update. However, the Percona toolkit
won't be used.
If this property is set to true
, then no such SQL statements will be output into the migration file.
liquibase.percona.skipChanges
: comma separated list of changes. Default: addColumn,dropColumn
in order to not use percona for adding/dropping a column.
liquibase.percona.options
: String of options. Default: --alter-foreign-keys-method=auto --nocheck-unique-key-change.
Since liquibase-percona 1.2.1. Default value changed with liquibase-percona 1.6.0.
This option allows the user to pass additional command line options to pt-online-schema-change. This e.g. can
be used in complicated replication setup to change the way slaves are detected and how their state is used.
You can also specify a percona configuration file via --config file.conf
,
see Configuration Files.
Multiple options are separated by space. If argument itself contains a space, it must be quoted with
double-quotes, e.g. --config "filename with spaces.conf"
.
liquibase.percona.defaultOn
: true/false. Default: true. Since liquibase-percona 1.3.0
This options allows to change the default behavior for the UsePercona flag. By default,
all changes, that do not explicitly specify this flag, use the value from this system property.
Setting this property to false
allows to control for each single change, whether to use Percona Toolkit
or not.
liquibase.password
: String with the password needed to connect to the database. Default: java.sql.Connection
(if that fails) or from the default liquibase.properties
file. If this property is set,
then it is used for the password when executing pt-online-schema-change
.
liquibase.percona.path
: Path to the percona toolkit directory, where the tool
pt-online-schema-change
is installed. Default: PATH
. You need to specify the bin
subfolder of the Percona Toolkit
distribution.
liquibase.percona.ptdebug
: true/false. Default: false. Since liquibase-percona 1.5.0
This option enables the debug output of pt-osc by setting the environment variable PTDEBUG
before
starting pt-osc.
liquibase.percona.keepAlive
: true/false. Default: true Since liquibase-percona 4.4.0
This option allows to disable the keepalive thread if there are any problems with it. The keepalive thread
pings the database while pt-online-schema-change is executing. This avoids that the server closes
liquibase's connection as it is idle during pt-osc. The server variable "wait_timeout" controls
when the connection is considered stale and dropped by the server. This variable is used to
determine how often the server will be pinged.
You can set these properties by using the standard java -D
option:
java -Dliquibase.percona.skipChanges=createIndex,dropColumn -jar liquibase.jar ...
Note: You'll have to call liquibase via "java -jar" as otherwise the system property cannot be set. You'll also need to make sure, that the liquibase-percona.jar file is on the classpath via the "--classpath" option.
When executing liquibase through maven, you can use the Properties Maven Plugin to set the system property. An example can be found in the "createIndexSkipped" integration test.
See CHANGELOG.md
The jar files can be downloaded manually from maven:
https://repo.maven.apache.org/maven2/org/liquibase/ext/liquibase-percona/
After extracting the zip file of liquibase, place liquibase-percona-4.26.0.jar
file in the sub directory lib
.
The shell script liquibase
/ liquibase.bat
will automatically pick this up and the extension is available.
Add the following dependency to the liquibase plugin:
<dependency>
<groupId>org.liquibase.ext</groupId>
<artifactId>liquibase-percona</artifactId>
<version>4.26.0</version>
</dependency>
You can also create a docker image which combines liquibase, liquibase-percona and percona toolkit. See Liquibase Percona Docker images.
Snapshot builds contain the latest features which are not yet available in a release.
Download: https://oss.sonatype.org/content/repositories/snapshots/org/liquibase/ext/liquibase-percona/
Enable the snapshot repository via Maven:
<project>
<pluginRepositories>
<pluginRepository>
<id>sonatype-nexus-snapshots</id>
<name>Sonatype Nexus Snapshots</name>
<url>https://oss.sonatype.org/content/repositories/snapshots</url>
<releases>
<enabled>false</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>
</project>
See also https://maven.apache.org/guides/development/guide-testing-development-plugins.html.
And just use the latest SNAPSHOT version for liquibase-percona dependency, e.g. 4.25.1-SNAPSHOT
:
<dependency>
<groupId>org.liquibase.ext</groupId>
<artifactId>liquibase-percona</artifactId>
<version>4.25.1-SNAPSHOT</version>
</dependency>
The non-locking update is achieved using triggers. First a new temporary table is created, including the added or dropped columns. Then the data is copied in chunks. While the copy is in progress, any newly created or deleted or updated rows are copied, too. This is done by adding triggers to the original table. After the copy is finished, the original table is dropped and the temporary table is renamed.
This means, that pt-online-schema-change cannot be used, if the table already uses triggers.
The command pt-online-schema-change
is searched only on the PATH
. Depending on the property
liquibase.percona.failIfNoPT
the update will fail or will just run without using pt-online-schema-change and
potentially lock the table for the duration of the update.
Simply run ./mvnw clean verify
.
You'll find the jar-file in the target/
subdirectory.
In order to execute the integration tests, run ./mvnw clean verify -Prun-its
.
Please note, that you'll need:
sudo apt-get install docker.io
.PATH
.sudo apt-get install libdbd-mysql-perl
.See the properties config_... in pom.xml
for connection details for the mysql docker instance.
To run a single integration test, execute maven like this: ./mvnw verify -Prun-its -Dinvoker.test=sharedScripts,addColumn*,dropColumn
The full error message:
Unexpected error running Liquibase: liquibase.exception.UnexpectedLiquibaseException:
java.lang.NoSuchMethodError: liquibase.ext.percona.PerconaDropColumnChange.getColumns()Ljava/util/List;
This means, you are trying to use version 1.1.1 of the extension with liquibase 3.2.x. This is an unsupported combination. For Liquibase 3.2.x, you'll need to use liquibase-percona 1.0.0
Some development has been sponsored by billforward.net, a highly flexible & unified billing platform.