Nats Server Versions Save

Changelog

Downgrade compatibility note

2.10.0 brings on-disk storage changes which bring significant performance improvements. Upgrade existing server versions will handle the new storage format transparently. However, if a downgrade from 2.10.0 occurs, the old version will not understand the format on disk with the exception 2.9.22 and any subsequent patch releases for 2.9. So if you upgrade from 2.9.x to 2.10.0 and then need to downgrade for some reason, it must be back to 2.9.22+ to ensure the stream data can be read correctly.

Go Version

• 1.21.1

Dependencies

• github.com/nats-io/nats.go v1.29.0
• github.com/nats-io/jwt/v2 v2.5.2
• github.com/nats-io/nkeys v0.4.5
• github.com/klauspost/compress v1.17.0
• golang.org/x/crypto v0.13.0

Added

Accounts

• Add $SYS.REQ.USER.INFO NATS endpoint for user info (#3671)

Auth

• Authorization callout extension for delegating to external auth providers (#3719, #3784, #3799, #3864, #3987, #4501, #4544)

Builds

• Add early build support for NetBSD (#3526) Thanks to @MatthiasPetermann for the contribution!
• Add early build support for IBM z/OS (#4209) Thanks to @v1gnesh for the contribution!

Cluster

• Multiple routes and ability to have per-account routes to reduce head-of-line blocking in clustered setups (#4001, #4183, #4414)
• Support for S2 compression of traffic over route connections (#4115, #4137)

Config

• Reload server config by sending a message in the system account to $SYS.REQ.SERVER.{server-id}.RELOAD (#4307)

Embedded

• Add ConnectionDeadline field to User to force server disconnect after deadline (#3580, #3674)

Leafnode

• Add TLSHandshakeFirst option to perform a TLS handshake before sending connection info (#4119)
• Support S2 compression of traffic over leafnode connections where the default now is s2_auto to compress relative to the RTT of the hub (#4167, #4230)
• Allow remotes from same server binding to same hub account (#4259)

Logging

• Add logfile_max_num server config field to auto-rotate files (#4548)

JetStream

• Add stream subject transforms (#3814, #3823, #3827, #4035, #4354, #4400, #4403, #4512)
• Add freeform metadata field to stream and consumer configs (#3797)
• Add support for consumers filtering on multiple subjects (#3500, #3865, #4008, #4129, #4188)
• Add original timestamp as header to republished message (#3933) Thanks to @amorey for the contribution!
• Allow republish for mirroring/sourcing streams (#4010)
• Add optional S2 stream compression for file store-backed streams (#4004, #4072)
• Add file store ability to re-encrypt with new encryption keys (#4296)
• Add embedded option to disable JetStream ASCII art at startup (#4261) Thanks to @renevo for the contribution!
• Add ability to configure first_seq when creating streams (#4322, #4345)
• Add sync_internal option to JetStream config (#4483)

Monitoring

• Add unique_tag field in /jsz and /varz endpoints (#3617)
• Add $SYS.REQ.SERVER.PING.IDZ NATS endpoint for basic server info (#3663)
• Add $SYS.REQ.SERVER.<id>.PROFILEZ NATS endpoint for requesting debugging profiles (#3774)
• Add subscription count to /statz endpoint (#3875)
• Add Raft query parameter to /jsz to include Raft group info (#3914)
• Add slow_consumer_stats to the /varz endpoint (#4330)

MQTT

• Add support for QoS2 exactly-once delivery (#4349, #4440)

Reload

• Match --signal PIDs with globular-style expression (#4370) Thanks to @jevolk for the contribution!

Subject Mapping

• Add ability to remove wildcard tokens in subject transforms (#4152)
• Allows cluster filtering in account subject mapping (#4175)

System Services

• Add $SYS.REQ.SERVER.<id>.KICK NATS endpoint to disconnect a client by id or by name from the target server (#4298)
• Add $SYS.REQ.SERVER.<id>.LDM NATS endpoint that sends a “lame duck mode” message to a client by id or name on the target server (#4298)

Windows

• Add NATS_STARTUP_DELAY env for configurable startup time (#3743) Thanks to @Alberic-Hardis for the contribution!

Improved

Leafnodes

• Add jitter to leafnode reconnections (#4398)

Logging

• Add account, stream and consumer name to consumer alignment cleanup warning (#3666) Thanks to @ch629 for the contribution!

JetStream

• Significant optimisations and reduced memory impact for replicated streams with a large number of interior deletes (common in large KVs), considerably reducing the amount of CPU and memory required to create stream snapshots and smoothing out publish latencies (#4070, #4071, #4075, #4284, #4520, #4553)
• Improve signaling mechanism for consumers to improve performance and reduce latency (#3706)
• Allow edit of Stream RePublish (#3811)
• Add batch completed status to pull consumers (#3822)
• Improve behavior of stream source consumer creation or config updates on leadership change (#4009)
• Record the stream and consumer info timestamps (#4133)
• Allow switching between limits and interest retention policies (#4361)
• Improve performance of deleting blocks (#4371)
• Update the way meta indexing is handled for filestore, significantly reducing time to recover streams at startup (#4450, #4481)
• Add self-healing mechanism to detect and delete orphaned Raft groups (#4510)
• Improve monitoring of consumers that need to be cleaned up (#4536)

MQTT

• Optimize retained messages by using KV semantics instead of holding retained messages in memory (#4199, #4228)
• Support for topics with . character (#4243) Thanks to @petedavis and @telemac for the reports!
• Set the RETAIN flag when delivering to new subscriptions and clear the flag in all other conditions (#4443)

Profiling

• Annotate CPU and goroutine profiles with additional asset information to assist with debugging (#4204)
• Remove unused block profile rate (#4402)

Subject Mapping

• Subject transform validation and error reporting (#4202)

Fixed

Accounts

• Fix data race when updating accounts (#4435, #4550, #4561) Thanks to @hspaay for the report!

Clients

• Check if client connection name was already set when storing it (#3824)

Leafnode

• Data race during validation and setup (#4194)

JetStream

• Check for invalid stream name in sources (#4222)
• Stream config update idempotency (#4292)
• Seqset encode bug that could cause bad stream state snapshots (#4348)
• Ensure stream assignment is set when checking replica count and updating retention (#4391)
• Hold lock when enforcing message limit on startup (#4469)
• Fix filestore data race on hash during snapshots (#4470)
• Use write lock for memory store filtered state (#4498)
• Fix data race on stream’s clustered filestore sequence (#4508)
• Fix possible panic when recalculating the first sequence of a subject (#4530) Thanks to @aldiesel for the report!
• Fix leaking timers in stream sources resulting in runaway CPU usage (#4532)
• Fix possible panic when consumer is not closed (#4541)
• Fix data race when accessing consumer assignment (#4547)
• Fix data race when changing stream retention policy (#4551)
• Fix data race when loading the next message in memory-based streams (#4552)
• Prevent forward proposals in consumers after scaling down a stream (#4556)

OSCP

• Fixed local issuer determination for OCSP Staple (#4355)

Routes

• Update LastActivity on connect for routes (#4415)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.22...v2.10.0

Changelog

Go Version

• 1.20.8 (updated out-of-cycle since Go 1.19 is now EOL)

Dependencies

• github.com/nats-io/jwt/v2 v2.5.0
• golang.org/x/crypto v0.12.0
• golang.org/x/sys v0.11.0

Improved

Monitoring

• CORS Allow-Origin passthrough for monitoring server (#4423) Thanks to @mdawar for the contribution!

JetStream

• Improve consumer scaling reliability with filters and cluster restart (#4404)
• Send event on lame duck mode (LDM) to avoid placing assets on shutting down nodes (#4405)
• Skip filestore tombstones if downgrade from 2.10 occurs (#4452)
• Adjust delivered and waiting count when consumer message delivery fails (#4472)

Fixed

Config

• Allow empty configs and fix JSON compatibility (#4394, #4418)
• Remove TLS OCSP debug log on reload (#4453)

Monitoring

• Fix Content-Type header when /healthz is not 200 OK (#4437) Thanks to @mdawar for the contribution!
• Fix server /connz idle time sorting (#4463) Thanks to @mdawar for the contribution!
• Interface conversion bug which could cause a panic when calling /ipqueuesz endpoint (#4477)

Leafnode

• Fix race condition which could affect propagating interest over leafnode connections (#4464)

JetStream

• Fix possible deadlock in checking for drift in the usage reporting when storing a message (#4411)
• Durable pull consumers could get cleaned up incorrectly on leader change (#4412)
• Moving an R1 stream could sometimes lose all messages (#4413)
• Prevent peer-remove of an R1 stream which could result in the stream becoming orphaned (#4420)
• Ensure consumer ack pending is less than max ack pending on state restore (#4427)
• Ensure to reset election timer when catching up (#4428) Thanks to @yuzhou-nj for the report!
• Auto step-down Raft leader if an entry is missing on a catchup request (#4432)
• Fix PurgeEx with keep having deletes in blocks (#4431)
• Update global subject index when message blocks expire (#4439)
• Ensure max messages per subject is respected after update (#4446) Thanks to @anthonyjacques20 for the report!
• Ignore and remove empty message blocks on rebuild (#4447)
• Fix possible accounting discrepancy on message write (#4455)
• Fix potential message duplication from stream sources when downgrading from 2.10 (#4454)
• Check for checksum violations for all records before sequence processing (#4465)
• Fix message block accounting (#4473)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.21...v2.9.22

Changelog

Go Version

• 1.19.12

Dependencies

• github.com/klauspost/compress v1.16.7
• github.com/nats-io/nats.go v1.28.0
• go.uber.org/automaxprocs v1.5.3
• golang.org/x/crypto v0.11.0
• golang.org/x/sys v0.10.0

Added

OCSP

• Add fetch, cache, and verification of client CA's OCSP Response for NATS, WebSocket, and MQTT client mTLS connections (#4362, backported from 2.10)
• Add bi-directional fetch, cache, and verification of CA OCSP Response for LEAF connections (#4362, backported from 2.10)

See ADR-38 OCSP Peer Verification

General

• Add UTC log timestamp option (#4331, backported from 2.10)

Improved

JetStream

• Don't error to server logs if message was deleted for consumer (#4328)
• Improve publish performance for zero-interest subjects (#4359) Thanks to @antlad for reporting the issue!
• Sync and reset message rejected count to ensure replicas don’t incorrectly discard messages (#4365, #4366)

Fixed

General

• Leaking memory on usage of getHash() (#4329) Thanks to @VuongUranus for reporting the issue!
• Server reload with highly active accounts and service imports could cause panic or dataloss (#4327)
• Fix detection of an unusable configuration file (#4358)
  • NOTE: as a side effect of this fix, the server will no longer startup with an empty config file
• Fix a few system service imports going missing after configuration reload (#4360)

OCSP

• Fix local-determination of issuer CA at startup (#4362)
• Remove constraint that all (super)cluster node peers must be issued by the same CA (#4362)

Embedded

• Don't require TLS for in-process client connection (#4323)

JetStream

• Fix serializability guarantee for concurrent publish when using expected-last-subject-sequence (#4319)
• Report correct consumer count in paged list response (#4339)
• Fix not validating single token filtered consumer (#4338)
• Fix stream recovery of message block with sequence gaps (#4344)
• Fix panic when re-calculating first sequence of SimpleState info (#4346)
• Fix stream store accounting drift (#4357)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.20...v2.9.21

Changelog

Go Version

• 1.19.11

Added

Windows

• Backport 2.10 support for native Windows certificate store (#4268)

Improved

Accounts

• Allow advisories to be exported/imported across accounts (#4302)

JetStream

• Optimize consumer create time on streams with a large number of blocks (#4269)

Fixed

Gateways

• Protect possible data race when reloading accounts (#4274)

Leafnodes

• Prevent zombie subscriptions which could lead to silent data loss when using queue subscriptions (#4299)

WebSocket

• Prevent reporting tls_required when tls_available is not set (#4264)

JetStream

• Prevent corrupting streams actively being restored during health check (#4277) Thank you @vitush93 for the report!
• Prevent encrypted data attempting to be decrypted with an empty key (#4301)

MQTT

• Ensure republished messages from streams are received by MQTT subscriptions (#4303)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.19...v2.9.20

Changelog

Go Version

• 1.19.10

Improved

JetStream

• Improve resource utilization when creating mirrors on very high-sequence streams (#4249)

Fixed

WebSocket

• Ensure INFO properties are populated based on the WebSocket listener when enabled (#4255) Thanks to @Envek for reporting the issue!

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.18...v2.9.19

Changelog

Go Version

• 1.19.10

Dependency Updates

• golang.org/x/crypto v0.9.0 (#4236)
• golang.org/x/sys v0.8.0 (#4236)
• github.com/nats-io/nats.go v1.27.0 (#4239)

Improved

Monitoring

• Optimize /statsz locking and sending in standalone mode (#4235)

JetStream

• Apply ack floor check only for interest-based streams (#4206)
• Improved efficiency and reduced CPU usage of the consumer ack floor check, particularly when the stream first sequence is a large number (#4226)
• Improve clean-up phase of R1 consumers on server restart for name reuse (#4216)
• Optimize “last message lookups” by subject (KV get operations) for small messages (#4232) Thanks to @jjthiessen for reporting the issue!
• Only enable JetStream account updates in clustered mode (#4233) Thanks to @tpihl for reporting the issue!

Fixed

General

• Fix a variety of potential panic scenarios (#4214) Thanks to @Zamony and @artemseleznev for the contribution!

Leadnode

• Daisy chained leafnodes could have unreliable interest propagation (#4207)
• Properly distribute requests to queue groups across leafnodes (#4231)

JetStream

• Killed server on restart could render encrypted stream unrecoverable (#4210) Thanks to @BhatheyBalaji for the report!
• Fix a few data races detected internal testing (#4211)
• Process extended purge operations correctly when being replayed (#4212, #4213) Thanks to @MauriceVanVeen for the report and contribution!

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.17...v2.9.18

Changelog

Go Version

• 1.19.9

Dependency Updates

• github.com/klauspost/compress v1.16.5 (#4088)

Improved

Core

• Additional optimizations to outbound queues, reducing memory footprint (#4084, #4093, #4139)
• Use faster flate compression library for WebSocket transport compression (#4087)

Leafnodes

• Optimize subscription interest propagation for large leafnode fleet (#4117, #4135)

Monitoring

• Support sorting by RTT for /connz (#4157)

Resolver

• Improve signaling for missing account lookups (#4151)

JetStream

• Optimized determining if a stream snapshot is required (#4074)
• Run periodic check for consumer “ack floor” drift on leader (#4086)
• Optimize leadership transfer during a stream migration (#4104)
• Improve how clustered consumer state is hydrated on startup (#4107)
• Add operation type to panic messages for improved debugging (#4108)
• Improve health check to repair stalled assets periodically (#4116, #4172)
• Remove unnecessary filestore lock to improve I/O performance (#4123)
• Various Raft leadership improvements (#4126, #4142, #4143, #4145)
• Improve accuracy of account usage (#4131)
• Clean up old Raft groups when streams are reset (#4177)

Fixed

General

• Fix various names in comments (#4099) Thanks to @cuishuang for the contribution!
• Fix various typos in comments (#4169) Thanks to @savion1024 for the contribution!
• Update tests to reflect the server.Start() call no longer blocks (#4111) Thanks to @lheiskan for reporting the issue!
• Fix race condition in config reload with gateway sublist check (#4127)
• Track all remote servers in a NATS system with different domains (#4159)

Core

• Fix premature closing in WebSocket transport due to outbound queue changes (#4084)
• Fix subscription interest for config-based accounts during config reload (#4130)
• Use monotonic time for measuring durations internally (#4132, #4154, #4163)

Monitoring

• Service import reporting for /accountz when mapping to local subjects (#4158)

JetStream

• Fix formatting of Raft debug log (#4090)
• Prevent failure of /healthz in single server mode on failed snapshot restore (#4100)
• Ensure a stream Raft node has fully stopped and resources freed (#4118)
• Fix case where R1 streams are orphaned and can’t scale up (#4146)
• Protect against out of bounds access on usage updates (#4164)
• Fix state rebuild where the first block is truncated and missing index info (#4166)
• Avoid stale KV reads on server restarted for replicated stores (#4171) Thanks to @yixinin for reporting the issue!
• Prevent deadlock with usage report for accounts (#4176)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.16...v2.9.17

Changelog

Go Version

• 1.19.8

Dependency Updates

• github.com/klauspost/compress v1.16.4
• github.com/nats-io/jwt/v2 v2.4.1
• github.com/nats-io/nkeys v0.4.4
• golang.org/x/crypto v0.8.0
• golang.org/x/sys v0.7.0

Added

Build

• Nightly build of the “main” branch as a Docker image: synadia/nats-server:nightly-main (#3961, #3962, #3963, #3972, #4019, #4063)
• Version control SHA in the Goreleaser build of the server (#3993). Thanks for the report @jzhoucliqr! Monitoring
• Add server name and route remote server name to /routez (#4054)

Resolver

• Add “hard_delete” option for stored JWTs (#3783). Thanks for the contribution @JulienVdG!

Improved

JetStream

• Storage and Raft layer improvements to decrease p99 latency variance and memory pressure in high load environments (#3956, #3952, #3965, #3981, #3999, #4018, #4020, #4021, #4022, #4026, #4027, #4028, #4029, #4030, #4038, #4045, #4050, #4053)
• Don’t show Raft warning for node that is closed (#3968)
• Use pooled buffer for flushing encrypted message blocks (#3975)
• Remove snapshotting of cores and maxprocs (#3979)
• Improvements to interest-based streams to optimize when messages are deleted (#4006, #4007, #4012)
• Better handling of concurrent stream and consumer creation of the same name (#4013)
• Finer-grain locking during asset checking to reduce contention in the /healthz endpoint (#4031)
• Encrypted file stores will now limit block sizes to improve performance (#4046)
• Improve performance on storing messages with varying subjects and limits are imposed (#4048, #4057). Thanks for the report @kung-foo!

Fixed

Subjects

• Ensure subjects containing percent (%) are escaped (#4040)

Accounts

• Fix data race when setting up service import subscriptions (#4068)

Leaf

• Fix leaf client connection failing on OCSP setups (#3964)
• Fix case when allow/deny permissions on leaf connection could block legitimate interest (#4032)

Cluster

• Route disconnect not detected by ping/pong (#4016). Thanks for the contribution @sandykellagher!

JetStream

• Pull consumer not sending timeout error to clients for expired requests (#3942)
• Prevent meta leader deadlock during deletion of orphaned streams during server startup (#3945)
• Clear ack’ed messages when scaling workqueue or interest-based streams (#3960) Thanks for the report @Kaarel!
• Remove messages from interest-based stream on consumer snapshot (#3970)
• Fix potential panic in message block buffer pool (#3978)
• Fixed an issue with consumer states growing and causing instability (#3980)
• Improve handling of out-of-storage condition (#3985)
• Address memory leak of unreachable Raft groups when JetStream becomes disabled (#3986)
• Prevent Raft leader from being placed on server in lame-duck mode (#4002)
• Remove potential race condition on sysRequest (#4017)
• Fix FirstSeq not being updated with filestore when purging subject (#4041). Thanks for the contribution @MauriceVanVeen!
• Fix Raft log debug reloading (#4047)
• Ensure consumer recovers fully on restart before being eligible for leader (#4049)
• Fix incorrect check between stream source/mirror with external streams (#4052)
• Fix various conditions during Raft group recovery (#4056, #4058)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.15...v2.9.16

Changelog

Go Version

• 1.19.6: Both the release executables and Docker images are built with this Go release

Added

• Monitoring
  • Add raft query parameter to /jsz to include group info (#3915)
  • Update /leafz to include leaf node remove server name and “spoke” flag (#3923, #3925)

Changed

• Lower default value of jetstream.max_outstanding_catchup to prevent slow consumers between routes (#3922)
  • Note: The new value is now 64MB from 128MB. This is better optimized for 1 Gbit links, however if your links are 10 Gbit or higher, this value can be set back to 128MB if slow consumers were not previously observed.

Improved

• Refactor intra-process queue to reduce allocations (#3894)
• JetStream
  • Better system stability and recovery from corrupt metadata due to hard forced restarts (#3934)
  • Optimize on-disk, per-subject info update (#3867)
  • Limit concurrent blocking IO to improve stability under heavy IO loads (#3867)
  • Improve message expiry calculation for large numbers of messages (#3867)
  • Optimize when and how consumer num pending is calculated, significantly speeding up consumer info requests (#3877)
  • Improve parallel consumer creation to prevent dropped messages (#3880)
  • Properly warn on consumer state update state failures (#3892)
  • Performance of consumer creation for certain configurations (#3901)
  • Send current snapshot to followers when becoming meta-leader (#3904)
  • Ensure preferred peer during stepdown is healthy (#3905)
  • Optimized various store calls on stream state (#3910)
  • Various performance and stability under heavy IO loads (#3922) (Thank you @matkam and @davidzhao for the report and the test harness!)

Fixed

• Fix stack overflow panic in reverse entry check when inbox ends with wildcard (#3862)
• Check if client connection name was already set when storing, preventing recursive memory growth (#3886)
• Fix check for count of wildcard tokens in “partition” subject transform (#3887) (Thank you @MauriceVanVeen for the contribution!)
• Fix panic if service export is nil (#3917) (Thank you @MauriceVanVeen for the report!)
• JetStream
  • Ensure per-subject info is updated when doing stream compact (#3860)
  • Ensure account usage is updated in the filestore when extended version purge occurs (#3876)
  • Prevent consumer deletes on restart, with non-fatal errors (#3881)
  • Do not warn if consumer replicas is zero since it will be inherited from the stream (#3882)
  • Named push consumers with inactive thresholds deleted when still active (#3884)
  • Prevent spurious “Error storing entry to WAL” log messages (#3893, #3891)
  • Clean up consumer redelivery state on stream purge (#3892)
  • Clean up consumer ack pending if below stream ack floor (#3892)
  • Update ack floors on messages being expired (#3892)
  • Fix lost ack pending consumer state on partial stream purge (#3892)
  • Snapshot and compact the consumer RAFT WAL, even when state changes do not occur, to prevent excessive disk usage (#3898)
  • Fix KV accounting errors under heavy concurrent usage (#3900)
  • Ensure new replicas respect MaxAge when a stream is scaled up (#3861)
  • Snapshots would not compact after being applied (#3907)
  • Fix filtered pending state calculation (#3910)
  • Recover from a failed truncate on raft WAL (#3911)
  • Fix JWT claims update if headers are passed (#3918)
• MQTT
  • Prevent use of wildcard characters with topics beginning with $, per the MQTT spec violation 4.7.2-1 (#3926) (Thank you @dominikh for the report!)

Dependency Updates

• klauspost/compress - v1.16.0
• nats-io/nats.go - v1.24.0
• golang.org/x/crypto - v0.6.0
• golang.org/x/sys - v0.5.0

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.14...v2.9.15

Changelog

Go Version

• 1.19.5: Both the release executables and Docker images are built with this Go release

Fixed

• JetStream
  • Fix circumstance when an empty snapshot could be written (#3844)
  • Fix possible panic and deadlock during a consumer filter subject update (#3845)
  • Fix consumer snapshot logic (#3846)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.12...v2.9.14