The most scalable open-source MQTT broker for IoT, IIoT, and connected vehicles
#12085 EMQX has been upgraded to leverage the capabilities of OTP version 26.1.2-2. NOTE: Docker images are still built with OTP 25.3.2.
#12189 Enhanced the ACL claim format in EMQX JWT authentication for greater versatility. The updated format now supports an array structure, aligning more closely with the file-based ACL rules.
For example:
[
{
"permission": "allow",
"action": "pub",
"topic": "${username}/#",
"qos": [0, 1],
"retain": true
},
{
"permission": "allow",
"action": "sub",
"topic": "eq ${username}/#",
"qos": [0, 1]
},
{
"permission": "deny",
"action": "all",
"topics": ["#"]
}
]
In this new format, the absence of a matching rule does not result in an automatic denial of the action. The authorization chain can advance to other configured authorizers if a match is not found in the JWT ACL. If no match is found throughout the chain, the final decision defers to the default permission set in authorization.no_match.
#12267 Added a new timeout parameter to the cluster/:node/invite interface, addressing the issue of default timeouts.
The previously set 5-second default timeout often led to HTTP API call timeouts because joining an EMQX cluster usually requires more time.
In addition, EMQX added a new API /cluster/:node/invite_async to support an asynchronous way to invite nodes to join the cluster and introduced a new cluster/invitation API to inspect the join status.
#12272 Introduced updates to the retain API in EMQX:
DELETE /retainer/messages to clean all retained messages.topic in the query string for the API GET /retainer/messages. For example, using a query string topic=t/1 filters the retained messages for a specific topic, improving the efficiency of message retrieval.#12277 Added mqtt/delayed/messages/:topic API to remove delayed messages by topic name.
#12278 Adjusted the maximum pagination size for paginated APIs in the REST API from 3000 to 10000.
#12289 Authorization caching now supports the exclusion of specific topics. For the specified list of topics and topic filters, EMQX will not generate an authorization cache. The list can be set through the authorization.cache.excludes configuration item or via the Dashboard. For these specific topics, permission checks will always be conducted in real-time rather than relying on previous cache results, thus ensuring the timeliness of authorization outcomes.
#12329 Added broker.routing.batch_sync configuration item to enable a dedicated process pool that synchronizes subscriptions with the global routing table in batches, thus reducing the frequency of cross-node communication that can be slowed down by network latency. Processing multiple subscription updates collectively, not only accelerates synchronization between replica nodes and core nodes in a cluster but also reduces the load on the broker pool, minimizing the risk of overloading.
#12333 Added a tags field for actions and connectors. Similar to the description field (which is a free text annotation), tags can be used to annotate actions and connectors for filtering and grouping.
#12072 GreptimeDB data integration now supports asynchronous data write operations to provide better performance.
#12194 Improved Kafka producer performance.
#12247 The bridges for InfluxDB have been split so they are available via the connectors and actions APIs. They are still backward compatible with the old bridge API.
#12299 Exposed more metrics to improve observability:
Montior API:
retained_msg_count field to /api/v5/monitor_current.license_quota field to /api/v5/monitor_current
retained_msg_count and node_uptime fields to /api/v5/monitor_current/nodes/{node}.retained_msg_count, license_quota and node_uptime fields to /api/v5/monitor_current/nodes/{node}.Prometheus API:
emqx_cert_expiry_at and emqx_license_expiry_at to /api/v5/prometheus/stats to display TLS listener certificate expiration time and license expiration time./api/v5/prometheus/auth endpoint to provide metrics such as execution count and running status for all authenticatiors and authorizators./api/v5/prometheus/data_integration endpoint to provide metrics such as execution count and status for all rules, actions, and connectors.Limitations:
Prometheus push gateway only supports the content in /api/v5/prometheus/stats?mode=node.
For more API details and metric type information, please see swagger api docs.
#12196 Improved network efficiency during routes cleanup.
Previously, when a node was down, a delete operation for each route to that node must be exchanged between all the other live nodes. After this change, only one match and delete operation is exchanged between all live nodes, significantly reducing the number of necessary network packets and decreasing the load on the inter-cluster network.
This optimization must be especially helpful for geo-distributed EMQX deployments where network latency can be significantly high.
#12354 The concurrent creation and updates of data integrations are now supported, significantly increasing operation speeds, such as when importing backup files.
#12396 Enhanced the user import feature in the authentication/:id/import_users Interface:
?type=plain for easier importing of users with plaintext passwords, complementing the existing functionality that supports hashed passwords.content-type: application/json, allowing HTTP Body submissions in JSON format. This extends the current capability that exclusively supports multipart/form-data for CSV files.#11902 Enhanced EMQX's capability to facilitate MQTT message bridging through the one-way Nari SysKeeper 2000 network isolation gateway.
#12232 Fixed an issue when cluster commit log table was not deleted after a node was forced to leave a cluster.
#12243 Fixed a family of subtle race conditions that could lead to inconsistencies in the global routing state.
#12269 Improved error handling in the /clients interface; now returns a 400 status with more detailed error messages, instead of a generic 500, for query string validation failures.
#12285 Updated the CoAP gateway to support short parameter names for slight savings in datagram size. For example, clientid=bar can be written as c=bar.
#12303 Fixed the message indexing in retainer. Previously, clients with wildcard subscriptions might receive irrelevant retained messages not matching their subscription topics.
#12305 Corrected an issue with incomplete client/connection information being passed into emqx_cm, which could lead to internal inconsistencies and affect memory usage and operations like node evacuation.
#12306 Fixed an issue preventing the connectivity test for the Connector from functioning correctly after updating the password parameter via the HTTP API.
#12359 Fixed an issue causing error messages when restarting a node configured with some types of data bridges. Additionally, these bridges were at risk of entering a failed state upon node restart, requiring a manual restart to restore functionality.
#12404 Fixed an issue where restarting a data integration with heavy message flow could lead to a stop in the collection of data integration metrics.
#12282 Improved the HTTP API error response for MySQL bridge creation failures. It also resolved a problem with removing MySQL Sinks containing undefined columns in their SQL.
#12291 Fixed inconsistencies in EMQX’s handling of configuration updates involving sensitive parameters, which previously led to stray "******" strings in cluster configuration files.
#12301 Fixed an issue with the line protocol in InfluxDB, where numeric literals were being stored as string types.
#12317 Removed the resource_opts.batch_size field from the MongoDB Action schema, as it is not yet supported.
resource_opts configuration schema for the GCP PubSub Producer connector so that it contains only relevant fields.
This affects the creation of GCP PubSub Producer connectors via HOCON configuration (connectors.gcp_pubsub_producer.*.resource_opts) and the HTTP APIs POST /connectors / PUT /connectors/:id for this particular connector type.#12234 Resolved compatibility issues with Open Telemetry configurations defined in emqx.conf from versions before EMQX 5.4.0, ensuring smooth integration of legacy configurations with the latest EMQX release.
#12236 Fixed client ID generation in MQTT broker data integration to comply with MQTT 3.1 specification of 23-byte limit. Client ID is now prefixed with user-assigned connector name, followed by the first 8 bytes of node name's SHA hash and pool member ID. If the resulting ID exceeds 23 bytes, additional SHA hash and truncation are applied to ensure compliance.
#12238 Resolved compatibility issue with the error format configurations introduced in the HTTP Action feature of EMQX version 5.3.2.
#12246 Stopped exposing port 11883 by default in Docker and removed it from Helm charts, as this port is no longer in use.
#12249 Fixed an issue in the /configs API where attempting to modify a read-only configuration value resulted in a garbled response message.
#12264 Fixed an issue where version 5.4 replica nodes could not join clusters with core nodes running versions earlier than 5.4 during the rolling upgrade process.
#12234 Resolved compatibility issues with Open Telemetry configurations defined in emqx.conf from versions before EMQX 5.4.0, ensuring smooth integration of legacy configurations with the latest EMQX release.
#12236 Fixed client ID generation in MQTT broker data integration to comply with MQTT 3.1 specification of 23-byte limit. Client ID is now prefixed with user-assigned connector name, followed by the first 8 bytes of node name's SHA hash and pool member ID. If the resulting ID exceeds 23 bytes, additional SHA hash and truncation are applied to ensure compliance.
#12238 Resolved compatibility issue with the error format configurations introduced in the HTTP Action feature of EMQX version 5.3.2.
#12240 Modified the /file_transfer API to return file transfer configurations in their original raw format. This change prevents the conversion of time units, such as "1h", to seconds, ensuring that callers receive the initially configured values. This modification aligns with other getter APIs, maintaining consistency in data representation.
#12241 Fixed a bug where configuring additional HTTP headers for S3 API interactions disrupted file transfers using the S3 storage backend, ensuring stable and uninterrupted file transfer operations.
#12246 Stopped exposing port 11883 by default in Docker and removed it from Helm charts, as this port is no longer in use.
#12249 Fixed an issue in the /configs API where attempting to modify a read-only configuration value resulted in a garbled response message.
#12250 Resolved an issue where the file_transfer configuration's secret_access_key value was erroneously being updated to masked stars (*****), ensuring that the original key value remains unaltered and secure.
#12256 Fixed an issue that prevented establishing connections to MySQL resources without a password.
#12264 Fixed an issue where version 5.4 replica nodes could not join clusters with core nodes running versions earlier than 5.4 during the rolling upgrade process.
#12234 Fix old (prior to EMQX 5.4.0) Open Telemetry configuration incompatibility when the config is defined in emqx.conf.
#12236 Ensure short client ID for MQTT bridges.
#12238 Compatible with the configuration of error formats introduced by HTTP Action in 5.3.2.
#12241 Fix an issue where setting up extra HTTP headers for communication with S3 API would break File Transfers using S3 storage backend.
#11994 Stop releasing packages for Windows.
#11998 Stop releasing packages for MacOS 11 (BigSur).
#12112 Stop supporting UDP multicast based clustering strategy.
#10976 Fix topic-filter overlapping handling in shared subscription.
session.subscribed and client.subscribe will now receive shared subscription in its full representation, e.g. $share/group1/topic1/#, and the share property is deleted from subopts.session.unsubscribed and client.unsubscribe will now receive shared subscription in its full representation, e.g. $share/group1/topic1/# instead of just topic1/#.share field in message SubOpts was deprecated.
ExHook Server will now receive shared subscription in its full representation, e.g. $share/group1/topic1/#, and the share property is deleted from message SubOpts.session.subscribed and session.unsubscribed rule-engine events will have shared subscriptions in their full representation for topic, e.g. $share/group1/topic1/# instead of just topic1/#.#11884 Modified the Prometheus API and configuration to implement the following improvements:
enable_basic_auth configuration for basic authentication on the scrape API endpoint, enhancing security.#11896 Introduced an enhancement for configuring sensitive authentication fields in bridges, such as passwords, tokens, and secret keys. This improvement allows the use of secrets stored as files in the file system. These secrets can be securely referenced in configuration files using the special file://
#11921 Introduced Open Telemetry Logs Handler that allows to format log events in alignment with the Open Telemetry log data model. This handler facilitates the exportation of formatted log events to a configured Open Telemetry collector or back-end, thereby enhancing log management and integration capabili
#11935 Switched to the new v2 routing store schema by default. New schema improves both subscription and routing performance, especially in scenarios with concurrent subscriptions to topic filters sharing common wildcard prefixes. However, it does come with a minor increase in memory usage. This schema also eliminates the need for a separate index, thus inconsistencies in the routing state rarely encountered in previous versions should no longer be possible.
If a cluster is rolling upgraded from older version, the cluster will continue to use v1 store until a full cluster (non-rolling) restart happens.
Users can still opt for the previous schema by configuring the broker.routing.storage_schema option to v1. However, this also requires a complete, non-rolling restart of the cluster to take effect.
#11984 Implemented Open Telemetry distributed tracing feature.
#12017 Implemented a dedicated HTTP API for the import and export of configuration and user data.
#12040 Upgraded QUIC protocol stack.
#11766 Implemented a preliminary Role-Based Access Control for the REST API. In this version, there are three predefined roles:
#12201 Support hot update of TCP/SSL/WS/WSS MQTT listeners configuration. This allows changing most of the configuration parameters without restarting the listener and disconnecting the clients. The limitations are:
bind
tcp_options.backlog
bind
tcp_options.*
ssl_options.*
#11608 Integrated LDAP bind operation as a new authenticator, providing a more flexible and secure method for user authentication.
#11773 Implemented Dashboard support for audit log management. Users can utilize this page to view all change operations performed on EMQX devices and data, such as kicking out devices, creating/deleting rules, etc.
#11778 Integrated Microsoft Entra Identity (formerly known as Azure Active Directory) support into the SAML single sign-on (SSO) process.
#11811 Improved the format for the REST API key bootstrap file to support initializing key with a role.
The new form is:api_key:api_secret:role.
role is optional and its default value is administrator.
#11852 Introduced a new GB/T 32960 gateway, enabling vehicles to connect with EMQX via the GBT32960 vehicular networking protocol.
#11883 Introduced a new JT/T808 gateway, enabling vehicles to connect with EMQX via the JT/T 808 vehicular networking protocol.
#11885 Introduced a new OCPP gateway for Electric vehicle (EV) charging stations to access EMQX through the OCPP (Open Charge Point Protocol).
#11971 Made /api/v5/load_rebalance/availability_check public, meaning it no longer requires authentication. This change simplifies the setup of load balancers.
It improved the gracefulness of the rebalance/evacuation process during the wait health check phase. The connections to nodes marked for eviction are now not prohibited during this phase. During this phase it is unknown whether these nodes are all marked unhealthy by the load balancer, so prohibiting connections to them may cause multiple unsuccessful reconnection attempts.
#12013 The data bridging design has been adjusted to split it into connectors and actions (Sinks). Connectors are used to manage the integration of data with external systems and can be reused across multiple actions, while actions are used to configure how data is processed. This design provides greater fl
The adjusted data bridges includes PostgreSQL, Timescale, and Matrix, which have now been split into connectors and actions APIs, but they remain backward compatible with the old data bridge API.
#12016 Enhanced license key management.
EMQX can now load the license key from a specified file. This is enabled by setting the license.key configuration to a file path, which should be prefixed with "file://".
Also added the ability to revert to the default trial license by setting license.key = default. This option simplifies the process of returning to the trial license if needed.
#12129 Default license renewal. Replaced old license issued in Jan 2023. New license supports up to 25 concurrent connections.
#10976 Fixed topic-filter overlapping handling in shared subscription. In the previous implementation, the storage method for subscription options did not provide adequate support for shared subscriptions. This resulted in message routing failures and leakage of routing tables between nodes during the "subscribe-unsubscribe" process with specific order and topics.
#12048 Fixed COAP gateway bug that caused it to ignore subscription options.
#12078 Upgraded grpc-erl to 0.6.12. This update addresses a potential deadlock issue where the grpc client started dependent apps lazily.
#12081 Updated gen_rpc library to version 3.3.1. The new version includes several performance improvements:
Avoiding allocating extra memory for the packets before they are sent to the wire in some cases.
Bypassing network for the local calls.
Avoid senstive data leaking in debug logs #12202
#12111 Fixed an issue when API tokens were sometimes unavailable immediately after login due to race condition.
#12121 Fixed an issue where nodes in the cluster would occasionally return a stale view when updating configurations on different nodes concurrently.
#12158 Fixed an issue when the rule engine cannot connect to Redis hosted by Upstash.
Before the fix, after establishing a TCP connection with the Redis service, the Redis driver of EMQX used Inline Commands to send AUTH and SELECT commands. However, the upstash Redis service does not support Inline Commands, which causes the rule engine to fail to connect to the `up
After the fix, the Redis driver of EMQX uses RESP (REdis Serialization Protocol) to send AUTH and SELECT commands.
#12176 Always acknowledge DISCONNECT packet to MQTT-SN client regardless of whether the connection has been successfully established before.
#12180 Fix an issue where DTLS enabled MQTT-SN gateways could not be started, caused by incompatibility of default listener configuration with the DTLS implementation.
#12219 Fix file transfer S3 config secret deobfuscation issue while performing config updates from dashboard.
#11994 Stop releasing packages for Windows.
#11998 Stop releasing packages for MacOS 11 (BigSur).
#12112 Stop supporting UDP multicast based clustering strategy.
#10976 Fix topic-filter overlapping handling in shared subscription.
session.subscribed and client.subscribe will now receive shared subscription in its full representation, e.g. $share/group1/topic1/#, and the share property is deleted from subopts.session.unsubscribed and client.unsubscribe will now receive shared subscription in its full representation, e.g. $share/group1/topic1/# instead of just topic1/#.share field in message SubOpts was deprecated.
ExHook Server will now receive shared subscription in its full representation, e.g. $share/group1/topic1/#, and the share property is deleted from message SubOpts.session.subscribed and session.unsubscribed rule-engine events will have shared subscriptions in their full representation for topic, e.g. $share/group1/topic1/# instead of just topic1/#.#11884 Modified the Prometheus API and configuration to implement the following improvements:
enable_basic_auth configuration for basic authentication on the scrape API endpoint, enhancing security.#11896 Introduced an enhancement for configuring sensitive authentication fields in bridges, such as passwords, tokens, and secret keys. This improvement allows the use of secrets stored as files in the file system. These secrets can be securely referenced in configuration files using the special file://
prefix, enhancing the security of sensitive data handling in bridge configurations.
#11921 Introduced Open Telemetry Logs Handler that allows to format log events in alignment with the Open Telemetry log data model. This handler facilitates the exportation of formatted log events to a configured Open Telemetry collector or back-end, thereby enhancing log management and integration capabilities.
#11935 Switched to the new v2 routing store schema by default. New schema improves both subscription and routing performance, especially in scenarios with concurrent subscriptions to topic filters sharing common wildcard prefixes. However, it does come with a minor increase in memory usage. This schema also eliminates the need for a separate index, thus inconsistencies in the routing state rarely encountered in previous versions should no longer be possible.
If a cluster is rolling upgraded from older version, the cluster will continue to use v1 store until a full cluster (non-rolling) restart happens.
Users can still opt for the previous schema by configuring the broker.routing.storage_schema option to v1. However, this also requires a complete, non-rolling restart of the cluster to take effect.
#11984 Implemented Open Telemetry distributed tracing feature.
#12017 Implemented a dedicated HTTP API for the import and export of configuration and user data.
#12040 Upgraded QUIC protocol stack.
#11766 Implemented a preliminary Role-Based Access Control for the REST API. In this version, there are three predefined roles:
#12201 Support hot update of TCP/SSL/WS/WSS MQTT listeners configuration. This allows changing most of the configuration parameters without restarting the listener and disconnecting the clients. The limitations are:
bind
tcp_options.backlog
bind
tcp_options.*
ssl_options.*
#12048 Fixed COAP gateway bug that caused it to ignore subscription options.
#12078 Upgraded grpc-erl to 0.6.12. This update addresses a potential deadlock issue where the grpc client started dependent apps lazily.
#12081 Updated gen_rpc library to version 3.3.1. The new version includes several performance improvements:
Avoiding allocating extra memory for the packets before they are sent to the wire in some cases.
Bypassing network for the local calls.
Avoid senstive data leaking in debug logs #12202
#12111 Fixed an issue when API tokens were sometimes unavailable immediately after login due to race condition.
#12121 Fixed an issue where nodes in the cluster would occasionally return a stale view when updating configurations on different nodes concurrently.
#12158 Fixed an issue when the rule engine cannot connect to Redis hosted by Upstash.
Before the fix, after establishing a TCP connection with the Redis service, the Redis driver of EMQX used Inline Commands to send AUTH and SELECT commands. However, the upstash Redis service does not support Inline Commands, which causes the rule engine to fail to connect to the upstash Redis service.
After the fix, the Redis driver of EMQX uses RESP (REdis Serialization Protocol) to send AUTH and SELECT commands.
#12176 Always acknowledge DISCONNECT packet to MQTT-SN client regardless of whether the connection has been successfully established before.
#12180 Fix an issue where DTLS enabled MQTT-SN gateways could not be started, caused by incompatibility of default listener configuration with the DTLS implementation.
#11884 Modified the Prometheus API and configuration to:
enable_basic_auth configuration for basic authentication on the scrape API endpoint, enhancing security#11896 Support configuring authentication-related sensitive fields in bridges (i.e. passwords, tokens, secret keys) via secrets stored as files in the file system, through special file:// prefix.
#11921 Introduced Open Telemetry Logs Handler that allows to format log events according to Open Telemetry log data model and export them to the configured Open Telemetry collector or back-end.
#11935 Switch to the new v2 routing store schema by default. New schema improves both subscription and routing performance, especially so for scenarios with concurrent subscriptions to topic filters sharing common wildcard prefixes, at the cost of slightly increased memory usage. This schema also eliminates the need for a separate index, thus inconsistencies in the routing state rarely encountered in previous versions should no longer be possible.
If a cluster is rolling upgraded from older version, the cluster will continue to use v1 store until a full cluster (non-rolling) restart happens.
The former schema can still be forced by setting broker.routing.storage_schema configuration option to v1 and conducting full non-rolling cluster restart as well.
#11984 Implemented Open Telemetry distributed tracing feature.
#12017 Implemented HTTP API for configuration and user data import/export.
#12040 Upgrade QUIC stack, more features on the way!
#12089 Added a technical preview of the new persistent session implementation based on RocksDB. Please note that this feature is in alpha stage and must not be enabled in the production systems.
Features missing in the early preview version of the new persistent session implementation:
#12201 Support hot update of TCP/SSL/WS/WSS MQTT listeners configuration, which allows changing most of the configuration parameters without restarting the listener and disconnecting the clients.
In case of TCP/SSL listeners, changes to the following parameters still require full listener restart:
bind
tcp_options.backlog
In case of WS/WSS listeners, any parameter can be freely changed without losing the connected clients. However, changing transport related parameters will cause listening socket to be re-opened, namely:
bind
tcp_options.*
ssl_options.*
#11766 Implemented a preliminary Role-Based Access Control for the REST API.
In this version, there are three predefined roles:
Administrator: This role could access all resources.
Viewer: This role can only view resources and data, corresponding to all GET requests in the REST API.
Publisher: This role is special for MQTT messages publish, it can only access publish-related endpoints.
#11773 Support audit log filter via dashboard (http api).
#11778 Support Azure Entra Id for saml single sign on.
Signed SAML AuthnRequest is not fully supported yet by Azure Entra ID.
See also for more details:
single-sign-on-saml-protocol#signature
#11795 Integrated Nari Syskeeper 2000 as a new bridge backend.
#11811 Improve the format for the REST API key bootstrap file to support initialize key with a role.
The new form is:api_key:api_secret:role.
role is optional and its default value is administrator.
#11852 Introduced a new gateway for vehicles to access EMQX through the GBT32960 protocol.
#11883 Introduced a new gateway for vehicles to access EMQX through the JT/T 808 protocol.
#11885 Introduced a new gateway for Electric vehicle (EV) charging stations to access EMQX through the OCPP (Open Charge Point Protocol).
#11971 Made /api/v5/load_rebalance/availability_check public, i.e. not requiring authentication. This simplifies load balancer setup.
Made rebalance/evacuation more graceful during the wait health check phase. The connections to nodes marked for eviction are now not prohibited during this phase. During this phase it is unknown whether these nodes are all marked unhealthy by the load balancer, so prohibiting connections to them may cause multiple unssuccessful attempts to reconnect.
#12013 The bridges for PostgreSQL, Timescale and Matrix have been split so they are available via the connectors and actions APIs. They are still backwards compatible with the old bridge API.
#12016 Enhanced license key management.
EMQX can now load the license key from a specified file. This is enabled by setting the license.key configuration to a file path, which should be prefixed with "file://".
Also added the ability to revert to the default trial license by setting license.key = default. This option simplifies the process of returning to the trial license if needed.
#12129 Default license renewal.
Replaced old license issued in Jan 2023. New license supports up to 25 concurrent connections.
#10976 Fix topic-filter overlapping handling in shared subscription. In the previous implementation, the storage method for subscription options did not provide adequate support for shared subscriptions. This resulted in message routing failures and leakage of routing tables between nodes during the "subscribe-unsubscribe" process with specific order and topics.
session.subscribed and client.subscribe will now receive shared subscription in its full representation, e.g. $share/group1/topic1/#, and the share property is deleted from subopts.session.unsubscribed and client.unsubscribe will now receive shared subscription in its full representation, e.g. $share/group1/topic1/# instead of just topic1/#.share field in message SubOpts was deprecated.
ExHook Server will now receive shared subscription in its full representation, e.g. $share/group1/topic1/#, and the share property is deleted from message SubOpts.session.subscribed and session.unsubscribed rule-engine events will have shared subscriptions in their full representation for topic, e.g. $share/group1/topic1/# instead of just topic1/#.#12048 Fix COAP gateway bug that caused it to ignore subscription options.
#12078 Upgrade grpc-erl to 0.6.12
grpc-erl 0.6.12 fixes a potential deadlock that was possible because grpc client started dependent apps lazily.
#12081 Updated gen_rpc library to version 3.3.1. The new version includes
several performance improvements:
Avoid allocating extra memory for the packets before they are sent to the wire in some cases
Bypass network for the local calls
Avoid senstive data leaking in debug logs #12202
#12111 Fix an issue where API tokens were sometimes unavailable by using sync_transaction function to ensure all updates are consistently synchronized to the replica node.
#12121 Fixed occasionally return stale view when updating configurations on different nodes concurrently
#12158 Fix the issue that the rule engine cannot connect to upstash Redis.
Before the fix, after establishing a TCP connection with the Redis service, the Redis driver of EMQX used Inline Commands to send AUTH and SELECT commands. However, the upstash Redis service does not support Inline Commands, which causes the rule engine to fail to connect to the upstash Redis service.
After the fix, the Redis driver of EMQX uses RESP (REdis Serialization Protocol) to send AUTH and SELECT commands.
#12176 Ack the DISCONNECT packet to MQTT-SN client regardless of whether the connection has been successfully established.
#12180 Fix an issue where DTLS enabled MQTT-SN gateways could not be started, caused by incompatibility of default listener configuration with the DTLS implementation.
#11884 Modified the Prometheus API and configuration to:
enable_basic_auth configuration for basic authentication on the scrape API endpoint, enhancing security#11896 Support configuring authentication-related sensitive fields in bridges (i.e. passwords, tokens, secret keys) via secrets stored as files in the file system, through special file:// prefix.
#11921 Introduced Open Telemetry Logs Handler that allows to format log events according to Open Telemetry log data model and export them to the configured Open Telemetry collector or back-end.
#11935 Switch to the new v2 routing store schema by default. New schema improves both subscription and routing performance, especially so for scenarios with concurrent subscriptions to topic filters sharing common wildcard prefixes, at the cost of slightly increased memory usage. This schema also eliminates the need for a separate index, thus inconsistencies in the routing state rarely encountered in previous versions should no longer be possible.
If a cluster is rolling upgraded from older version, the cluster will continue to use v1 store until a full cluster (non-rolling) restart happens.
The former schema can still be forced by setting broker.routing.storage_schema configuration option to v1 and conducting full non-rolling cluster restart as well.
#12017 Implemented HTTP API for configuration and user data import/export.
#12040 Upgrade QUIC stack, more features on the way!
#11766 Implemented a preliminary Role-Based Access Control for the REST API.
In this version, there are three predefined roles:
Administrator: This role could access all resources.
Viewer: This role can only view resources and data, corresponding to all GET requests in the REST API.
Publisher: This role is special for MQTT messages publish, it can only access publish-related endpoints.
#11773 Support audit log filter via dashboard (http api).
#11778 Support Azure Entra Id for saml single sign on.
Known issue: Signed SAML AuthnRequest is not fully supported yet by Azure Entra ID.
See also for more details:
single-sign-on-saml-protocol#signature
#11795 Integrated Nari Syskeeper 2000 as a new bridge backend.
#11811 Improve the format for the REST API key bootstrap file to support initialize key with a role.
The new form is:api_key:api_secret:role.
role is optional and its default value is administrator.
#11852 Introduced a new gateway for vehicles to access EMQX through the GBT32960 protocol.
#11883 Introduced a new gateway for vehicles to access EMQX through the JT/T 808 protocol.
#11885 Introduced a new gateway for Electric vehicle (EV) charging stations to access EMQX through the OCPP (Open Charge Point Protocol).
#11971 Made /api/v5/load_rebalance/availability_check public, i.e. not requiring authentication. This simplifies load balancer setup.
Made rebalance/evacuation more graceful during the wait health check phase. The connections to nodes marked for eviction are now not prohibited during this phase. During this phase it is unknown whether these nodes are all marked unhealthy by the load balancer, so prohibiting connections to them may cause multiple unssuccessful attempts to reconnect.
#12013 The bridges for PostgreSQL, Timescale and Matrix have been split so they are available via the connectors and actions APIs. They are still backwards compatible with the old bridge API.
#12016 Enhanced license key management.
EMQX can now load the license key from a specified file. This is enabled by setting the license.key configuration to a file path, which should be prefixed with "file://".
Also added the ability to revert to the default trial license by setting license.key = default. This option simplifies the process of returning to the trial license if needed.
#10976 Fix topic-filter overlapping handling in shared subscription. In the previous implementation, the storage method for subscription options did not provide adequate support for shared subscriptions. This resulted in message routing failures and leakage of routing tables between nodes during the "subscribe-unsubscribe" process with specific order and topics.
session.subscribed and client.subscribe will now receive shared subscription in its full representation, e.g. $share/group1/topic1/#, and the share property is deleted from subopts.session.unsubscribed and client.unsubscribe will now receive shared subscription in its full representation, e.g. $share/group1/topic1/# instead of just topic1/#.share field in message SubOpts was deprecated.
ExHook Server will now receive shared subscription in its full representation, e.g. $share/group1/topic1/#, and the share property is deleted from message SubOpts.session.subscribed and session.unsubscribed rule-engine events will have shared subscriptions in their full representation for topic, e.g. $share/group1/topic1/# instead of just topic1/#.#11725 Introduced the LDAP as a new authentication and authorization backend.
#11752 Changed default RPC driver from gen_rpc to rpc for core-replica database synchronization.
This improves core-replica data replication latency.
#11785 Allowed users with the "Viewer" role to change their own passwords. However, those with the "Viewer" role do not have permission to change the passwords of other users.
#11787 Improved the performance of the emqx command.
#11790 Added validation to Redis commands in Redis authorization source.
Additionally, this improvement refines the parsing of Redis commands during authentication and authorization processes. The parsing now aligns with redis-cli compatibility standards and supports quoted arguments.
#11757 Fixed the error response code when downloading non-existent trace files. Now the response returns 404 instead of 500.
#11762 Fixed an issue in EMQX's built_in_database authorization source. With this update, all Access Control List (ACL) records are completely removed when an authorization source is deleted. This resolves the issue of residual records remaining in the database when re-creating authorization sources.
#11771 Fixed validation of Bcrypt salt rounds in authentication management through the API/Dashboard.
#11780 Fixed validation of the iterations field of the pbkdf2 password hashing algorithm. Now, iterations must be strictly positive. Previously, it could be set to 0, which led to a nonfunctional authenticator.
#11791 Fixed an issue in the EMQX CoAP Gateway where heartbeats were not effectively maintaining the connection's active status. This fix ensures that the heartbeat mechanism properly sustains the liveliness of CoAP Gateway connections.
#11797 Modified HTTP API behavior for APIs managing the built_in_database authorization source. They will now return a 404 status code if built_in_database is not set as the authorization source, replacing the former 20X response.
#11965 Improved the termination of EMQX services to ensure a graceful stop even in the presence of an unavailable MongoDB resource.
#11975 This fix addresses an issue where redundant error logs were generated due to a race condition during simultaneous socket closure by a peer and the server. Previously, concurrent socket close events triggered by the operating system and EMQX resulted in unnecessary error logging. The implemented fix improves event handling to eliminate unnecessary error messages.
#11987 Fixed a bug where attempting to set the active_n option on a TCP/SSL socket could lead to a connection crash.
The problem occurred if the socket had already been closed by the time the connection process attempted to apply the active_n setting, resulting in a case_clause crash.
#11752 Changed default RPC driver from gen_rpc to rpc for core-replica database synchronization.
This improves core-replica data replication latency.
#11785 Allowed users with the "Viewer" role to change their own passwords. However, those with the "Viewer" role do not have permission to change the passwords of other users.
#11787 Improved the performance of the emqx command.
#11790 Added validation to Redis commands in Redis authorization source.
Additionally, this improvement refines the parsing of Redis commands during authentication and authorization processes. The parsing now aligns with redis-cli compatibility standards and supports quoted arguments.
#11541 Enhanced file transfer capabilities. Now, clients can use an asynchronous method for file transfer by sending commands to the $file-async/... topic and subscribing to command execution results on the $file-response/{clientId} topic. This improvement simplifies the use of the file transfer feature, particularly suitable for clients using MQTT v3.1/v3.1.1 or those employing MQTT bridging. For more details, please refer to EIP-0021.
#11757 Fixed the error response code when downloading non-existent trace files. Now the response returns 404 instead of 500.
#11762 Fixed an issue in EMQX's built_in_database authorization source. With this update, all Access Control List (ACL) records are completely removed when an authorization source is deleted. This resolves the issue of residual records remaining in the database when re-creating authorization sources.
#11771 Fixed validation of Bcrypt salt rounds in authentication management through the API/Dashboard.
#11780 Fixed validation of the iterations field of the pbkdf2 password hashing algorithm. Now, iterations must be strictly positive. Previously, it could be set to 0, which led to a nonfunctional authenticator.
#11791 Fixed an issue in the EMQX CoAP Gateway where heartbeats were not effectively maintaining the connection's active status. This fix ensures that the heartbeat mechanism properly sustains the liveliness of CoAP Gateway connections.
#11797 Modified HTTP API behavior for APIs managing the built_in_database authorization source. They will now return a 404 status code if built_in_database is not set as the authorization source, replacing the former 20X response.
#11965 Improved the termination of EMQX services to ensure a graceful stop even in the presence of an unavailable MongoDB resource.
#11975 This fix addresses an issue where redundant error logs were generated due to a race condition during simultaneous socket closure by a peer and the server. Previously, concurrent socket close events triggered by the operating system and EMQX resulted in unnecessary error logging. The implemented fix improves event handling to eliminate unnecessary error messages.
#11987 Fixed a bug where attempting to set the active_n option on a TCP/SSL socket could lead to a connection crash.
The problem occurred if the socket had already been closed by the time the connection process attempted to apply the active_n setting, resulting in a case_clause crash.
#11731 Added hot configuration support for the file transfer feature.
#11754 Improved the log formatting specifically for the Postgres bridge in EMQX. It addresses issues related to Unicode characters in error messages returned by the driver.