Global Workflow Composition that is Scalable, Secure, and GDPR-compliant
Release candidates such as 2.8.0-rc4 don't come with release notes
Release candidates such as 2.8.0-rc3 don't come with release notes
Release candidates such as 2.8.0-rc2 don't come with release notes
Canton 2.7.6 has been released on November 29, 2023. You can download the Daml Open Source edition from the Daml Connect Github Release Section. The Enterprise edition is available on Artifactory. Please also consult the full documentation of this release.
This release contains minor bug fixes and added flexibility on how database connections can be configured.
Canton periodically checks the validity and health of its connection to the database. Those checks were previously competing with other database queries, sometimes leading to errors that would be retried but cause warnings. This is no longer the case.
The observability of the streams served over the ledger api has been improved in both logging and metrics.
debug level rather then trace
metrics.daml.lapi.streams.active are now collected always, independently of the rate limiting settingsThe sizes of the connection pools used for interactions with the database storages inside the canton nodes are determined using a dedicated formula described in a documentation article on max connection settings:
The values obtained from that formula can now be overridden using explicit configuration settings for the read, write and ledger-api connection pools sizes:
canton.participants.participant.storage.parameters.connection-allocation.num-reads
canton.participants.participant.storage.parameters.connection-allocation.num-writes
canton.participants.participant.storage.parameters.connection-allocation.num-ledger-api
Similar parameters exist also for other canton node types:
canton.sequencers.sequencer.storage.parameters.connection-allocation...
canton.mediators.mediator.storage.parameters.connection-allocation...
canton.domain-managers.domain_manager.storage.parameters.connection-allocation...
The effective connection pool sizes are reported by the canton nodes at start-up
INFO c.d.c.r.DbStorageMulti$:participant=participant_b - Creating storage, num-reads: 5, num-writes: 4
The mediator does not handle transient DB exceptions properly after it has transitioned from passive to active during an HA failover
Mediator
2.5 to 2.7.5
Mediator becomes non functional when encountering a database exception after HA failover.
After encountering an SQL exception, the application handler fails, and the mediator disconnects its sequencer client. After that the mediator stops processing messages.
Restart the mediator node.
Needs a mediator to transition from active state to passive state and active state again, and then encounter a DB error. Not extremely likely but likelihood increases over time.
Upgrade to 2.7.6 or higher
The following Canton protocol and Ethereum sequencer contract versions are supported:
| Dependency | Version |
|---|---|
| Canton protocol versions | 3, 4, 5 |
Canton has been tested against the following versions of its dependencies:
| Dependency | Version |
|---|---|
| Java Runtime | OpenJDK 64-Bit Server VM Zulu11.62+17-CA (build 11.0.18+10-LTS, mixed mode) |
| Postgres | postgres (PostgreSQL) 14.10 (Debian 14.10-1.pgdg120+1) |
| Oracle | 19.18.0 |
Release candidates such as 2.8.0-rc1 don't come with release notes
Canton 2.7.5 has been released on November 1, 2023. You can download the Daml Open Source edition from the Daml Connect Github Release Section. The Enterprise edition is available on Artifactory. Please also consult the full documentation of this release.
This is a maintenance release.
Canton cannot reconnect to a domain after a duplicated submission request.
Participant
2.3 to 2.7.4
The participant is left in a failed state.
Connecting to the domain fails with an internal error "IllegalStateException: Cannot find event for sequenced in-flight submission".
No workaround by clients possible.
Needs a submission request being sequenced twice. This should not happen under normal operations, but it did happen with the Besu driver.
Upgrade to 2.7.5
The expected KMS wrapper-key configuration value has changed from:
crypto.private-key-store.encryption.wrapper-key-id = { str = "..."}
to a simple string:
crypto.private-key-store.encryption.wrapper-key-id = "..."
The following Canton protocol and Ethereum sequencer contract versions are supported:
| Dependency | Version |
|---|---|
| Canton protocol versions | 3, 4, 5 |
Canton has been tested against the following versions of its dependencies:
| Dependency | Version |
|---|---|
| Java Runtime | OpenJDK 64-Bit Server VM Zulu11.62+17-CA (build 11.0.18+10-LTS, mixed mode) |
| Postgres | postgres (PostgreSQL) 14.9 (Debian 14.9-1.pgdg120+1) |
| Oracle | 19.18.0 |
Canton 2.3.17 has been released on October 23, 2023. You can download the Daml Open Source edition from the Daml Connect Github Release Section. The Enterprise edition is available on Artifactory. Please also consult the full documentation of this release.
This is a maintenance release of Canton. We recommend users who uses Besu to upgrade to this version to ensure that sequencer drivers properly report their Health Status that allows a HA environment to fail over.
Description : when a sequencer driver is connected to a Besu node, in case the Besu node is unable to respond within 15s the sequencer is still considered healthy as long as a connection to the node can be made.
Affected deployments : Besu deployments.
Affected versions : All canton versions supporting Besu.
Impact : The participant will be unable to change to a healthy sequencer (in a multi sequencer setup) and will be stuck until Besu problem is resolved.
Workaround : Restarting the Besu node.
Likelihood of the bug : The bug occurs very rarely and only when the Besu node is having unusual problems or under heavy load.
The following Canton protocol and Ethereum sequencer contract versions are supported:
| Dependency | Version |
|---|---|
| Canton protocol versions | 2.0.0, 3.0.0 |
| Ethereum contract versions | 1.0.0, 1.0.1 |
Canton has been tested against the following versions of its dependencies:
| Dependency | Version |
|---|---|
| Java Runtime | OpenJDK 64-Bit Server VM 18.9 (build 11.0.16+8, mixed mode, sharing) |
| Postgres | postgres (PostgreSQL) 14.9 (Debian 14.9-1.pgdg120+1) |
| Oracle | 19.15.0 |
| Besu | besu/v21.10.9/linux-x86_64/openjdk-java-11 |
| Fabric | 2.2.2 |
Canton 2.7.4 has been released on October 09, 2023. You can download the Daml Open Source edition from the Daml Connect Github Release Section. The Enterprise edition is available on Artifactory. Please also consult the full documentation of this release.
This release contains minor performance improvements related to the over-use of metrics in certain contexts as well as 2 bug fixes (one minor, one major). Please see below for more information.
canton.monitoring.metrics.report-execution-context-metrics configuration to true to enable these metrics.canton.monitoring.metrics.enabled to allow disabling of metrics collection entirely.canton.monitoring.tracing.tracer.sampler = always-on. See the tracing documentation for more information.The CommandService now respects the grpc deadlines:
If a request reaches the command processing layer with an already-expired gRPC deadline, the command will not be sent for submission.
Instead, the request is rejected with a new self-service error code REQUEST_DEADLINE_EXCEEDED, which informs the client
that the command is guaranteed to not have been sent for execution to the ledger.
This change has been introduced with the bugfix 23-027 (see below).
Participant may shutdown ungracefully if there there are still completing CommandService submissions or, in extreme cases, the Ledger API can restart during normal operations.
The participant node
No significant operational impact.
IllegalStateException("Promise already completed.") is logged.Not applicable as the effect is mostly aesthetic.
This ungraceful shutdown is only likely under heavy usage of CommandService at the same time with the participant shutdown. The likelihood of this bug triggering a Ledger API restart is very small as multiple conditions need to be met:
Upgrade when convenient.
The CommandService errors out when confronted with an expired gRPC request deadline.
The participant node
If encountered repeatedly (up to the maximum-in-flight configuration limit for the CommandService), the CommandService can appear saturated and reject new commands.
When a command request is submitted via CommandService.submitAndWait and its variants with providing a gRPC request deadline, the request can fail with an INTERNAL error reported to the client and a log message with ERROR level is logged on the participant.
Restart the participant and use a higher gRPC request deadline.
This bug is likely to happen if the gRPC request deadline is small enough for it to expire upon arriving at the participant's Ledger API.
Upgrade when convenient.
The following Canton protocol and Ethereum sequencer contract versions are supported:
| Dependency | Version |
|---|---|
| Canton protocol versions | 3, 4, 5 |
Canton has been tested against the following versions of its dependencies:
| Dependency | Version |
|---|---|
| Java Runtime | OpenJDK 64-Bit Server VM Zulu11.62+17-CA (build 11.0.18+10-LTS, mixed mode) |
| Postgres | postgres (PostgreSQL) 14.9 (Debian 14.9-1.pgdg120+1) |
| Oracle | 19.18.0 |
Canton 2.7.3 has been released on September 27, 2023. You can download the Daml Open Source edition from the Daml Connect Github Release Section. The Enterprise edition is available on Artifactory. Please also consult the full documentation of this release.
This is a maintenance release. Please check the list of bug-fixes and improvements below. Due to various improvements, we would recommend users to upgrade to it during their next maintenance window. Please note, we've skipped 2.7.2.
Previously, the canton console would use a hard-coded "CantonConsole" as an applicationId in the command submissions and the completion subscriptions performed against the ledger api.
Now, if an access token is provided to the console, it will extract the userId from that token and use it instead. A local console will use the adminToken provided in canton.participants.<participant>.ledger-api.admin-token, whereas a remote console will use the token from canton.remote-participants.<remoteParticipant>.token
This affects the following console commands:
The console command keys.secret.rotate_node_key can now accept a name for the newly generated key.
owner_to_key_mappings.rotate_key command expects a node referenceThe previous owner_to_key_mappings.rotate_key is deprecated and now expects a node reference (InstanceReferenceCommon)
to avoid any dangerous and/or unwanted key rotations.
The mediator keeps recent finalised responses cached in memory to avoid having to re-fetch late responses from the database. The cache size is now configurable via
canton.mediators.mediator.caching.finalized-mediator-requests.maximum-size = 1000 // default
The background pruning queries of the contract key journal for Postgres have been improved to reduce the load on the database by making better use of the existing indexes. In addition, a pruning related query that checks the request journal for how far it is safe to prune has also been improved for Postgres by choosing a more suitable index.
The Canton trace id was added back to some KMS audit logs where it was missing.
Console commands for ACS migration can now be used with remote nodes.
On restart of a sync domain, the participant will replay pending transactions, updating the stores in case some writes were not persisted. Within the command deduplication store, existing records are compared with to be written records for internal consistency checking purposes. This comparison includes the trace context which differs on a restart and hence can cause the check to fail, aborting the startup with an IllegalArgumentException.
Participant
An affected participant can not reconnect to the given domain, which means that transaction processing is blocked.
Upon restart, the participant refuses to reconnect to the domain, writing the following log message: ERROR c.d.c.p.s.d.DbCommandDeduplicationStore ... - An internal error has occurred. java.lang.IllegalArgumentException: Cannot update command deduplication data for ChangeIdHash
No workaround exists. You need to upgrade to a version not affected by this issue.
The error happens if a participant crashes with a particular state not yet written to the database. The bug has been present since end of Nov 21 and has never been observed before, not even during testing.
Upgrade during your next maintenance window to a patch version not affected by this issue.
rotate_keys command is dangerous and does not work when there are multiple sequencersWe allow the replacement of key A with key B, but we cannot guarantee that the node using key A will actually have access to key B.
Furthermore, when attempting to rotate the keys of a sequencer using the rotate_node_keys(domainManagerRef) method,
it will fail if we have more than one sequencer in our environment.
This occurs because they share a unique identifier (UID), and as a result, this console command not only rotates the
keys of the sequencer it is called on but also affects the keys of the other sequencers.
Modified the process of finding keys for rotation in the rotate_node_keys(domainManagerRef)
function to prevent conflicts among multiple sequencers that share the same UID.
Additionally, we have updated the console command owner_to_key_mappings.rotate_key to
expect a node reference (InstanceReferenceCommon), thereby ensuring that both the current
and new keys are associated with the correct node.
All nodes (but mostly sequencer)
A node can mistakenly rotate keys that do not pertain to it. Using rotate_node_keys(domainManagerRef) to rotate a sequencer's keys when other sequencers are present will also fail and break Canton.
When trying to rotate a sequencers' keys it catastrophically fails with a java.lang.RuntimeException: KeyNotAvailable.
For 'rotate_node_keys(domainManagerRef),' we have ensured that we filter the correct keys for rotation by checking both the authorized store and the local private key store. Additionally, we have deprecated the existing 'owner_to_key_mappings.rotate_key' and introduced a new method that requires the user to provide the node instance for which they intend to apply the key rotation. We have also implemented a validation check within this function to ensure that the current and new keys are associated with this node.
Everytime we use rotate_node_keys to rotate the keys of sequencer(s) in a multiple sequencer environment.
Upgrade the Canton console that you use to administrate the domain, in particular the sequencer and mediator, to a Canton version with the bug fix.
rotate_node_keys command when it is used to rotate keys of sequencer(s) and mediator(s)Canton has a series of console commands to rotate keys, in particular, rotate_node_keys that is used
to rotate de keys of a node.
We allow the replacement of key A with key B, but we cannot guarantee that the node using key A
will actually have access to key B.
Furthermore, when attempting to rotate the keys of a sequencer using the
rotate_node_keys(domainManagerRef) method, it will fail if we have more than one sequencer in our environment.
This occurs because they share a unique identifier (UID), and as a result, this console command not only
rotates the keys of the sequencer it is called on but also affects the keys of the other sequencers.
A node can mistakenly rotate keys that do not pertain to it.
Using rotate_node_keys(domainManagerRef) to rotate a sequencer's keys when other
sequencers are present will also fail and break Canton.
No visible symptom as the command just skips over the keys to rotate.
For rotate_node_keys(domainManagerRef), we have ensured that we filter the correct keys for
rotation by checking both the authorized store and the local private key store.
Additionally, we have deprecated the existing owner_to_key_mappings.rotate_key and introduced a new
method that requires the user to provide the node instance for which they intend to apply the key rotation.
We have also implemented a validation check within this function to ensure that the current
and new keys are associated with this node.
Everytime we use rotate_node_keys to rotate the keys of sequencer(s) in a multiple sequencer environment.
Upgrade the Canton console that you use to administrate the domain, in particular the sequencer and mediator, to a Canton version with the bug fix.
The Canton protocol includes in ViewParticipantData contracts that required to re-interpret command evaluation. These inputs are known as coreInputs. Included as part of this input contract data is contract metadata the includes details of signatories and stakeholders.
Where the evaluation of a command results in a contract key being resolved to a contract identifier but that identifier is not in turn resolved to a contract instance that distributed metadata associated with contract will incorrectly have the key maintainers as both the signatory and the stakeholders. A way to do this would be to execute a choice on a contract other than the keyed contract that only issues a lookupByKey on the keyed contract.
Protocol version 3,4
As the signatory data associated with the affected contracts is not valid, any validation based on this will also be invalid, so the command will be rejected.
This bug was discovered when the model conformance logic was extended to validate all input contract signatories. The presented signatory list being inconsistent with that expected caused a failure.
A workaround would be to ensure that whenever a contract key is resolved to a contract identifier that identifier is always resolved to a contract (even if not needed). For example following the lookupByKey if the case where a contract identifier is returned the issue a fetch command on this identifier, discarding the result.
Unlikely. Most of the time a contract key is resolved to a contract so that some action can be performed on that contract. In this situation the metadata would be correct. The only situation this has occurred observed is in test scenarios.
If affected, the issue should be observed during development and testing which can then be remediated by upgrading the environment to the appropriate version.
Transaction view decomposition is the process of taking a transaction and generating a view hierarchy whereby each view has a common set of informees. Each view additionally has a rollback scope. A child view having a rollback scope that is different from that of the parent indicates that any changes to the liveness of contracts that occurred within the child view should be disregarded upon exiting the child view. For example it would be valid for a contract to be consumed in a rolled back child view and then consumed again in a subsequent child view. As the activeness of contracts is preserved across views with the same rollback scope every source transaction rollback node should be allocated a unique scope. In certain circumstances this is not happening resulting in contract activeness inconsistency.
This problem is observed in canton protocol version 4 and fixed in version 5.
The impact of this bug is that a inconsistent transaction view hierarchy can be generated from a consistent transaction. This in turn can result in a valid transaction being rejected or in a ledger fork when an observer participant lacks input to properly processing a transaction, thereby rejecting it beside the transaction being approved. The symptoms of this would be the mediator rejecting a valid transaction request on the basis of inconsistency or the participant declaring LOCAL_VERDICT_FAILED_MODEL_CONFORMANCE_CHECK followed by an ACS_COMMITMENT_MISMATCH.
This bug only effects transactions that contain rollbacks (catched exceptions), if use of rollbacks can be avoided this bug will not occur.
To encounter this bug requires a transaction that has multiple rolled back nodes in which overlapping contracts and/or keys are used. For this reason the likelihood of encountering the bug is low and issues should be discovered during development / testing.
Only use Daml exception with protocol version >= 5.
In some cases, participant state and mediator domain state topology transactions were silently ignored when they were sent as part of a cascading topology update (which means they were sent together with a namespace certificate). As a result, the nodes had a different view on the topology state and not all daml transactions could be run.
All nodes
A participant node might consider another participant node as inactive and therefore refuse to send transactions or invalidate transactions.
A Daml transaction might be rejected with UNKNOWN_INFORMEES.
Flush the topology state by running "domain.participants.set_state(pid, Submission, Vip (and back to Ordinary))". This will run the update through the "incremental" update code path which is behaving correct, thereby fixing the topology state of the broken node.
The bug is deterministic and can be caused when using permissioned domains when the participant state is received together with the namespace delegation of the domain but without the namespace delegation of the participant.
Upgrade to this version if you intend to use permissioned domains. If you need to fix a broken system, then upgrade to a version fixing the issue and apply the work-around to "flush" the topology state.
After an Indexer restart in the Ledger API or any error causing the client transaction streams to fail, the PingService stops working.
The participant node
When the Ledger API encounters an error that leads to cancelling the client connections while the participant node does not become passive, the PingService cannot continue processing commands.
Ping commands issued in the PingService are timing out.
Additionally, the participant might appear unhealthy if configured to report health
by using the PingService (i.e. configured with monitoring.health.check.type = ping).
Restart the participant node.
This bug occurs consistently when there is an error in the Ledger API, such as a DB overloaded issue that causes the Ledger API Indexer to restart. For this bug to occur, the participant node must not transition to passive state. If it transitions to passive and then back to active, the bug should not reproduce.
If the system is subject to frequent transient errors in the Ledger API (e.g. flaky Index database) or consistently high load, update to this version in order to avoid reproducibility.
The following Canton protocol and Ethereum sequencer contract versions are supported:
| Dependency | Version |
|---|---|
| Canton protocol versions | 3, 4, 5 |
| Canton has been tested against the following versions of its dependencies: | |
| Dependency | Version |
| ---------------------------- | ---------------------------- |
| Java Runtime | OpenJDK 64-Bit Server VM Zulu11.62+17-CA (build 11.0.18+10-LTS, mixed mode) |
| Postgres | postgres (PostgreSQL) 14.9 (Debian 14.9-1.pgdg120+1) |
| Oracle | 19.18.0 |
Canton 2.3.16 has been released on September 22, 2023. You can download the Daml Open Source edition from the Daml Connect Github Release Section. The Enterprise edition is available on Artifactory. Please also consult the full documentation of this release.
Canton 2.3.16 has been released as part of Daml 2.3.16 with no additional change on Canton.
The following Canton protocol and Ethereum sequencer contract versions are supported:
| Dependency | Version |
|---|---|
| Canton protocol versions | 2.0.0, 3.0.0 |
| Ethereum contract versions | 1.0.0, 1.0.1 |
Canton has been tested against the following versions of its dependencies:
| Dependency | Version |
|---|---|
| Java Runtime | OpenJDK 64-Bit Server VM 18.9 (build 11.0.16+8, mixed mode, sharing) |
| Postgres | postgres (PostgreSQL) 14.9 (Debian 14.9-1.pgdg120+1) |
| Oracle | 19.15.0 |
| Besu | besu/v21.10.9/linux-x86_64/openjdk-java-11 |
| Fabric | 2.2.2 |