Pulsar Versions Save

Broker

• [fix][admin] Fix force delete subscription not working (#22423)
• [fix][broker] Fix Reader can be stuck from transaction aborted messages. (#22610)
• [fix][admin] Fix can't delete tenant for v1 (#22550)
• [fix][admin] Fix namespace admin api exception response (#22587)
• [fix][broker] Avoid being stuck when closing the broker with extensible load manager (#22573)
• [fix][broker] Create new ledger after the current ledger is closed (#22034)
• [fix][broker] Disable system topic message deduplication (#22582)
• [fix][broker] Do not migrate internal topics during Blue-Green Migration when ExtensibleLoadBalancer is used (#22478)
• [fix][broker] Fix BufferOverflowException and EOFException bugs in /metrics gzip compression (#22576)
• [fix][broker] Fix NPE causing dispatching to stop when using Key_Shared mode and allowOutOfOrderDelivery=true (#22533)
• [fix][broker] Fix ProducerBusy issue due to incorrect userCreatedProducerCount on non-persistent topic (#22685)
• [fix][broker] Fix a deadlock in SystemTopicBasedTopicPoliciesService during NamespaceEventsSystemTopicFactory init (#22528)
• [fix][broker] Fix broken topic policy implementation compatibility with old pulsar version (#22535)
• [fix][broker] Fix consumer stops receiving messages when with large backlogs processing (#22454)
• [fix][broker] Fix invalid condition in logging exceptions (#22412)
• [fix][broker] Fix message drop record in producer stat (#22458)
• [fix][broker] Fix metrics pulsar_topic_load_failed_count is 0 when load non-persistent topic fails and fix the flaky test testBrokerStatsTopicLoadFailed (#22580)
• [fix][broker] Fix nothing changed after removing dynamic configs (#22673)
• [fix][broker] Fix typos in Consumer class (#22532)
• [fix][broker] One topic can be closed multiple times concurrently (#17524)
• [fix][broker] Optimize /metrics, fix unbounded request queue issue and fix race conditions in metricsBufferResponse mode (#22494)
• [fix][broker] Part-1: Replicator can not created successfully due to an orphan replicator in the previous topic owner (#21946)
• [fix][broker] Part-2: Replicator can not created successfully due to an orphan replicator in the previous topic owner (#21948)
• [fix][broker] Prevent long deduplication cursor backlog so that topic loading wouldn't timeout (#22479)
• [fix][broker] Prevent long deduplication cursor backlog so that topic loading wouldn't timeout (#22479)
• [fix][broker] Reader stuck after call hasMessageAvailable when enable replicateSubscriptionState (#22572)
• [fix][broker] Skip topic.close during unloading if the topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to use ns.checkOwnershipPresentAsync for ExtensibleLoadBalancer (#22379) (#22407)
• [fix][broker] Support OIDC providers with JWK without alg field set in keys (#22421)
• [fix][broker] Update TransferShedder underloaded broker check to consider max loaded broker's msgThroughputEMA and update IsExtensibleLoadBalancerImpl check (#22321) (#22418)
• [fix][broker] Update topic partition failed when config maxNumPartitionsPerPartitionedTopic<0 (#22397)
• [fix][broker] avoid offload system topic (#22497)
• [fix][broker] fix replicated subscriptions for transactional messages (#22452)
• [fix][broker] rename to changeMaxReadPositionCount (#22656)
• [fix][broker] usedLocallySinceLastReport should always be reset (#22672)
• [fix][broker][admin] Fix cannot update properties on NonDurable subscription. (#22411)
• [improve][broker] Make the config metricsBufferResponse description more effective (#22490)
• [improve][meta] Retry re-validating ResourceLock with backoff after errors (#22617)
• [improve][admin] Check if the topic existed before the permission operations (#22547)
• [improve][broker] Add additionalSystemCursorNames ignore list for TTL check (#22614)
• [improve][broker] Add topic name to emitted error messages. (#22506)
• [improve][broker] Avoid repeated Read-and-discard when using Key_Shared mode (#22245)
• [improve][broker] Create partitioned topics automatically when enable topic level replication (#22537)
• [improve][broker] Don't log brokerClientAuthenticationParameters and bookkeeperClientAuthenticationParameters by default (#22395)
• [improve][broker] Improve Gzip compression, allow excluding specific paths or disabling it (#22370)
• [improve][broker] Optimize gzip compression for /metrics endpoint by sharing/caching compressed result (#22521)
• [improve][broker] Propagate cause exception in TopicBusyException when applicable (#22596)
• [improve][broker] Repeat the handleMetadataChanges callback when configurationMetadataStore equals localMetadataStore (#22519)
• [improve][broker] Servlet support response compression (#21667)
• [improve][broker] Support X-Forwarded-For and HA Proxy Protocol for resolving original client IP of http/https requests (#22524)
• [improve][broker] backlog quota exceed limit log replaced with debug (#22488)
• [fix][ml] Fix NPE of getValidPositionAfterSkippedEntries when recovering a terminated managed ledger (#22552)
• [fix][ml] Mark delete stuck due to switching cursor ledger fails (#22662)
• [fix][ml] No rollover inactive ledgers when metadata service invalid (#22284)
• [fix][offload] Fix OOM in tiered storage, caused by unbounded offsets cache (#22679)
• [fix][offload] Increase file upload limit from 2048MiB to 4096MiB for GCP/GCS offloading (#22554)
• [fix][storage] ReadonlyManagedLedger initialization does not fill in the properties (#22630)
• [fix][txn] Handle exceptions in the transaction pending ack init (#21274)
• [improve][meta] Log a warning when ZK batch fails with connectionloss (#22566)
• [improve][offload] Apply autoSkipNonRecoverableData configuration to tiered storage (#22531)
• [improve][offload] Replace usage of shaded class in OffsetsCache (#22683)

Client

• [fix][client] Include swagger annotations in shaded client lib (#22570)
• [fix][client] Fix Consumer should return configured batch receive max messages (#22619)
• [fix][client] Fix ReaderBuilder doest not give illegalArgument on connection failure retry (#22639)
• [fix][client] Fix client side memory leak when call MessageImpl.create and fix imprecise client-side metrics: pendingMessagesUpDownCounter, pendingBytesUpDownCounter, latencyHistogram (#22393)

Pulsar IO and Pulsar Functions

• [fix][fn] make sure the classloader for ContextImpl is functionClassLoader in different runtimes (#22501)
• [fix][io] CompressionEnabled didn't work on elasticsearch sink (#22565)
• [fix][io] Fix es index creation (#22654)
• [fix][io] Kafka Source connector maybe stuck (#22511)
• [improve][io]: Add validation for JDBC sink not supporting primitive schema (#22376)

Others

• [improve][ws] Add memory limit configuration for Pulsar client used in Websocket proxy (#22666)
• [fix][build] Fix networkaddress.cache.negative.ttl config (#22400)
• [improve][misc] Specify valid home dir for the default user in the Ubuntu based docker image (#22446)

Library updates

• [fix][broker] upgrade jclouds 2.5.0 -> 2.6.0 (#22220)
• [improve][misc] Upgrade to Bookkeeper 4.16.5 (#22484)
• [improve][misc] Upgrade to Netty 4.1.108 and tcnative 2.0.65 (#22369)
• [fix][misc] Rename all shaded Netty native libraries (#22415)
• [fix][sec] Upgrade Bouncycastle to 1.78 (#22509)
• [fix][sec] Upgrade aws-sdk.version to avoid CVE-2024-21634 (#22633)
• [fix][sec] Upgrade elasticsearch-java version to avoid CVE-2023-4043 (#22640)
• [fix][sec] Upgrade postgresql version to avoid CVE-2024-1597 (#22635)
• [improve][build] Upgrade Lombok to 1.18.32 for Java 22 support (#22425)

Tests & CI

• [fix][test] Clear MockedPulsarServiceBaseTest fields to prevent test runtime memory leak (#22659)
• [fix][test] Clear fields in test cleanup to reduce memory consumption (#22583)
• [fix][test] Fix NPE in BookKeeperClusterTestCase tearDown (#22493)
• [fix][test] Fix flaky test ReplicatorTest (#22594)
• [fix][test] Fix the flaky tests of ManagedLedgerImplUtilsTest (#22611)
• [fix][test] Flaky-test: ManagedLedgerTest.testTimestampOnWorkingLedger (#22600)
• [fix][test] Flaky-test: testMessageExpiryWithTimestampNonRecoverableException and testIncorrectClientClock (#22489)
• [fix][test] SchemaMap in AutoConsumeSchema has been reused (#22500)
• [fix][test][branch-3.2] Fix DeduplicationDisabledBrokerLevelTest. Adjust to PR 22034 presence.
• [improve][test] Move ShadowManagedLedgerImplTest to flaky tests (#22526)
• [improve][test] Move most flaky tests to flaky group (#22433)
• [improve][test] Replace usage of curl in Java test and fix stream leaks (#22463)
• [fix][build] Fix building java-test-image without setting IMAGE_JDK_MAJOR_VERSION (#22095)
• [improve][build] Upgrade OWASP Dependency check version to 9.1.0 (#22530)
• [improve][ci][branch-3.2] Upgrade actions in pulsar-ci and pulsar-ci-flaky, port owasp cache change

For the complete list, check the full changelog.

Broker

• [fix][broker] Fix force delete subscription not working (#22423)
• [fix][broker] Avoid being stuck when closing the broker with extensible load manager (#22573)
• [fix][broker] Create new ledger after the current ledger is closed (#22034)
• [fix][broker] Disable system topic message deduplication (#22582)
• [fix][broker] Fix BufferOverflowException and EOFException bugs in /metrics gzip compression (#22576)
• [fix][broker] Fix NPE causing dispatching to stop when using Key_Shared mode and allowOutOfOrderDelivery=true (#22533)
• [fix][broker] Fix ProducerBusy issue due to incorrect userCreatedProducerCount on non-persistent topic (#22685)
• [fix][broker] Fix a deadlock in SystemTopicBasedTopicPoliciesService during NamespaceEventsSystemTopicFactory init (#22528)
• [fix][broker] Fix broken topic policy implementation compatibility with old pulsar version (#22535)
• [fix][broker] Fix configurationMetadataSyncEventTopic is marked supporting dynamic setting, but not implemented (#22684)
• [fix][broker] Fix consumer stops receiving messages when with large backlogs processing (#22454)
• [fix][broker] Fix invalid condition in logging exceptions (#22412)
• [fix][broker] Fix message drop record in producer stat (#22458)
• [fix][broker] Fix metrics pulsar_topic_load_failed_count is 0 when load non-persistent topic fails and fix the flaky test testBrokerStatsTopicLoadFailed (#22580)
• [fix][broker] Fix nothing changed after removing dynamic configs (#22673)
• [fix][broker] Fix typos in Consumer class (#22532)
• [fix][broker] Fix typos lister -> listener (#21068)
• [fix][broker] One topic can be closed multiple times concurrently (#17524)
• [fix][broker] Optimize /metrics, fix unbounded request queue issue and fix race conditions in metricsBufferResponse mode (#22494)
• [fix][broker] Part-1: Replicator can not created successfully due to an orphan replicator in the previous topic owner (#21946)
• [fix][broker] Part-2: Replicator can not created successfully due to an orphan replicator in the previous topic owner (#21948)
• [fix][broker] Prevent long deduplication cursor backlog so that topic loading wouldn't timeout (#22479)
• [fix][broker] Reader stuck after call hasMessageAvailable when enable replicateSubscriptionState (#22572)
• [fix][broker] Skip topic.close during unloading if the topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to use ns.checkOwnershipPresentAsync for ExtensibleLoadBalancer (#22379) (#22403)
• [fix][broker] Support OIDC providers with JWK without alg field set in keys (#22421)
• [fix][broker] Update TransferShedder underloaded broker check to consider max loaded broker's msgThroughputEMA and update IsExtensibleLoadBalancerImpl check (#22321) (#22416)
• [fix][broker] Update topic partition failed when config maxNumPartitionsPerPartitionedTopic<0 (#22397)
• [fix][broker] avoid offload system topic (#22497)
• [fix][broker] fix replicated subscriptions for transactional messages (#22452)
• [fix][broker] rename to changeMaxReadPositionCount (#22656)
• [fix][broker] usedLocallySinceLastReport should always be reset (#22672)
• [fix][broker][admin] Fix cannot update properties on NonDurable subscription. (#22411)
• [fix][broker] Fix Reader can be stuck from transaction aborted messages. (#22610)
• [improve][broker] Make the config metricsBufferResponse description more effective (#22490)
• [improve][broker] Retry re-validating ResourceLock with backoff after errors (#22617)
• [improve][broker] Add topic_load_failed metric (#19236)
• [improve][broker] Add topic name to emitted error messages. (#22506)
• [improve][broker] Avoid repeated Read-and-discard when using Key_Shared mode (#22245)
• [improve][broker] Create partitioned topics automatically when enable topic level replication (#22537)
• [improve][broker] Don't log brokerClientAuthenticationParameters and bookkeeperClientAuthenticationParameters by default (#22395)
• [improve][broker] Improve Gzip compression, allow excluding specific paths or disabling it (#22370)
• [improve][broker] Optimize gzip compression for /metrics endpoint by sharing/caching compressed result (#22521)
• [improve][broker] Propagate cause exception in TopicBusyException when applicable (#22596)
• [improve][broker] Repeat the handleMetadataChanges callback when configurationMetadataStore equals localMetadataStore (#22519)
• [improve][broker] Servlet support response compression (#21667)
• [improve][broker] Support X-Forwarded-For and HA Proxy Protocol for resolving original client IP of http/https requests (#22524)
• [improve][broker] backlog quota exceed limit log replaced with debug (#22488)
• [fix][ml] Fix NPE of getValidPositionAfterSkippedEntries when recovering a terminated managed ledger (#22552)
• [fix][ml] Mark delete stuck due to switching cursor ledger fails (#22662)
• [fix][ml] No rollover inactive ledgers when metadata service invalid (#22284)
• [fix][offload] Fix OOM in tiered storage, caused by unbounded offsets cache (#22679)
• [fix][offload] Increase file upload limit from 2048MiB to 4096MiB for GCP/GCS offloading (#22554)
• [fix][storage] ReadonlyManagedLedger initialization does not fill in the properties (#22630)
• [fix][admin] Fix namespace admin api exception response (#22587)
• [fix][txn]Handle exceptions in the transaction pending ack init (#21274)
• [improve][log] Print source client addr when enabled haProxyProtocolEnabled (#22686)
• [improve][meta] Log a warning when ZK batch fails with connectionloss (#22566)
• [improve][offload] Apply autoSkipNonRecoverableData configuration to tiered storage (#22531)
• [improve][offload] Replace usage of shaded class in OffsetsCache (#22683)
• [improve][sec] Align some namespace level policy authorisation check (#21640)

Client

• [fix][client] Fix Consumer should return configured batch receive max messages (#22619)
• [fix][client] Fix ReaderBuilder doest not give illegalArgument on connection failure retry (#22639)
• [fix][client] Fix client side memory leak when call MessageImpl.create and fix imprecise client-side metrics: pendingMessagesUpDownCounter, pendingBytesUpDownCounter, latencyHistogram (#22393)

Pulsar IO and Pulsar Functions

• [fix][fn] make sure the classloader for ContextImpl is functionClassLoader in different runtimes (#22501)
• [fix][io] CompressionEnabled didn't work on elasticsearch sink (#22565)
• [fix][io] Fix es index creation (#22654) (#22701)
• [fix][io] Kafka Source connector maybe stuck (#22511)
• [improve][io]: Add validation for JDBC sink not supporting primitive schema (#22376)

Others

• [improve][ws] Add memory limit configuration for Pulsar client used in Websocket proxy (#22666)
• [fix][build] Fix building docker images without setting UBUNTU_MIRROR
• [fix][build] Fix networkaddress.cache.negative.ttl config (#22400)
• [improve][misc] Specify valid home dir for the default user in the Ubuntu based docker image (#22446)

Library updates

• [fix][broker] upgrade jclouds 2.5.0 -> 2.6.0 (#22220)
• [improve][misc] Upgrade to Bookkeeper 4.16.5 (#22484)
• [improve][misc] Upgrade to Netty 4.1.108 and tcnative 2.0.65 (#22369)
• [fix][misc] Rename all shaded Netty native libraries (#22415)
• [fix][sec] Upgrade Bouncycastle to 1.78 (#22509)
• [fix][sec] Upgrade aws-sdk.version to avoid CVE-2024-21634 (#22633)
• [fix][sec] Upgrade elasticsearch-java version to avoid CVE-2023-4043 (#22640)
• [fix][sec] Upgrade postgresql version to avoid CVE-2024-1597 (#22635)
• [improve][build] Upgrade Lombok to 1.18.32 for Java 22 support (#22425)

Tests & CI

• [fix][test] Clear MockedPulsarServiceBaseTest fields to prevent test runtime memory leak (#22659)
• [fix][test] Clear fields in test cleanup to reduce memory consumption (#22583)
• [fix][test] Fix NPE in BookKeeperClusterTestCase tearDown (#22493)
• [fix][test] Fix flaky test ReplicatorTest (#22594)
• [fix][test] Fix resource leak in TransactionCoordinatorClientTest (#21380)
• [fix][test] Fix the flaky tests of ManagedLedgerImplUtilsTest (#22611)
• [fix][test] Flaky-test: ManagedLedgerTest.testTimestampOnWorkingLedger (#22600)
• [fix][test] Flaky-test: testMessageExpiryWithTimestampNonRecoverableException and testIncorrectClientClock (#22489)
• [fix][test] SchemaMap in AutoConsumeSchema has been reused (#22500)
• [fix][test][branch-3.0] Fix test PersistentTopicsTest.testUpdatePartitionedTopic
• [improve][test] Move ShadowManagedLedgerImplTest to flaky tests (#22526)
• [improve][test] Move most flaky tests to flaky group (#22433)
• [improve][test] Replace usage of curl in Java test and fix stream leaks (#22463)
• [improve][ci][branch-3.0] Upgrade actions in pulsar-ci and pulsar-ci-flaky, port owasp cache change
• [improve][build] Upgrade OWASP Dependency check version to 9.1.0 (#22530)

For the complete list, check the full changelog.

What's Changed

Broker

• [fix][broker] Avoid execute prepareInitPoliciesCacheAsync if namespace is deleted (#22268)
• [fix][broker] Avoid expired unclosed ledgers when checking expired messages by ledger closure time (#22335)
• [fix][broker] Check cursor state before adding it to the waitingCursors (#22191)
• [fix][broker] Close dispatchers stuck due to mismatch between dispatcher.consumerList and dispatcher.consumerSet (#22270)
• [fix][broker] Fix OpReadEntry.skipCondition NPE issue (#22367)
• [fix][broker] Fix ResourceGroup report local usage (#22340)
• [fix][broker] Fix ResourceGroups loading (#21781)
• [fix][broker] Fix issue of field 'topic' is not set when handle GetSchema request (#22377)
• [fix][broker] Fix wrong double-checked locking for readOnActiveConsumerTask in dispatcher (#22279)
• [fix][broker] fix mismatch between dispatcher.consumerList and dispatcher.consumerSet (#22283)
• [fix][ml]Expose ledger timestamp (#22338)
• [fix][offload] Fix Offload readHandle cannot close multi times. (#22162)
• [fix][txn]Fix TopicTransactionBuffer potential thread safety issue (#22149)
• [improve][admin] Fix the createMissingPartitions doesn't response correctly (#22311)
• [improve][broker] Add createTopicIfDoesNotExist option to RawReader constructor (#22264)
• [improve][broker] Add fine-grain authorization to ns/topic management endpoints (#22305)
• [improve][broker] Add missing configuration keys for caching catch-up reads (#22295)
• [improve][broker] Support create RawReader based on configuration (#22280)

Client

• [fix][client] Consumer lost message ack due to race condition in acknowledge with batch message (#22353)
• [fix][client] Do no retrying for error subscription not found when disabled allowAutoSubscriptionCreation (#22164)
• [fix][client] Fix wrong results of hasMessageAvailable and readNext after seeking by timestamp (#22363)
• [fix][client] GenericProtobufNativeSchema not implement getNativeSchema method. (#22204)
• [fix][client] Unclear error message when creating a consumer with two same topics (#22255)
• [fix][client] fix Reader.hasMessageAvailable might return true after seeking to latest (#22201)
• [fix][client]Fixed getting an incorrect maxMessageSize value when accessing multiple clusters in the same process (#22306)

Others

• [fix][misc] Make ConcurrentBitSet thread safe (#22361)
• [improve][misc] Remove the call to sun InetAddressCachePolicy (#22329)
• [fix][ws] Check the validity of config before start websocket service (#22231)

Library updates

• [fix][sec] Upgrade jose4j to 0.9.4 (#22273)
• [fix][sec] Go Functions security updates (#21844)
• [fix][sec] Upgrade Zookeeper to 3.9.2 to address CVE-2024-23944 (#22275)
• [improve][misc] Include native epoll library for Netty for arm64 (#22319)
• [improve][misc] Upgrade Netty version to 4.1.105.Final (#21923)

Tests & CI

• [fix][ci] Enable CI for branch-3.2 (#22287)
• [fix][ci] Tolerate mount option change failing in CI (#22241)
• [fix][test] Fix flaky ManagedLedgerErrorsTest.recoverAfterZnodeVersionError (#22368)
• [fix][test] Fix flaky RGUsageMTAggrWaitForAllMsgsTest (#22252)
• [fix][test] Fix flaky test BrokerServiceAutoSubscriptionCreationTest (#22190)
• [fix][test] Fix flaky test ManagedLedgerTest.testGetNumberOfEntriesInStorage (#22344)
• [fix][test] Fix thread leak in TopicPoliciesAuthZTest (#22257)
• [fix][test][branch-3.2] Fix broken ManagedLedgerTest.testGetNumberOfEntriesInStorage
• [fix][fn] fix broken function-go test (#22260)
• [fix][test] Fix the tests with same namespace name (#22240)
• [fix][build] Fix version in pulsar-bom/pom.xml and fix src/set-project-version.sh
• [improve][test][branch-3.2] Improve ManagedLedgerTest.testGetNumberOfEntriesInStorage
• [improve][misc] Upgrade checkstyle to 10.14.2 (#22291)

For the complete list, check the full changelog.

What's Changed

Broker

• [fix][broker] Avoid execute prepareInitPoliciesCacheAsync if namespace is deleted (#22268)
• [fix][broker] Avoid expired unclosed ledgers when checking expired messages by ledger closure time (#22335)
• [fix][broker] Check cursor state before adding it to the waitingCursors (#22191)
• [fix][broker] Close dispatchers stuck due to mismatch between dispatcher.consumerList and dispatcher.consumerSet (#22270)
• [fix][broker] Fix OpReadEntry.skipCondition NPE issue (#22367)
• [fix][broker] Fix ResourceGroup report local usage (#22340)
• [fix][broker] Fix ResourceGroups loading (#21781)
• [fix][broker] Fix issue of field 'topic' is not set when handle GetSchema request (#22377)
• [fix][broker] Fix wrong double-checked locking for readOnActiveConsumerTask in dispatcher (#22279)
• [fix][broker] fix mismatch between dispatcher.consumerList and dispatcher.consumerSet (#22283)
• [fix][ml]Expose ledger timestamp (#22338)
• [improve][admin] Fix the createMissingPartitions doesn't response correctly (#22311)
• [improve][broker] Add createTopicIfDoesNotExist option to RawReader constructor (#22264)
• [improve][broker] Add fine-grain authorization to ns/topic management endpoints (#22309)
• [improve][broker] Add missing configuration keys for caching catch-up reads (#22295)
• [improve][broker] Change log level to reduce duplicated logs (#22147)

Client

• [fix][client] Consumer lost message ack due to race condition in acknowledge with batch message (#22353)
• [fix][client] Fix wrong results of hasMessageAvailable and readNext after seeking by timestamp (#22363)
• [fix][client] GenericProtobufNativeSchema not implement getNativeSchema method. (#22204)
• [fix][client] Unclear error message when creating a consumer with two same topics (#22255)
• [fix][client] fix Reader.hasMessageAvailable might return true after seeking to latest (#22201)
• [fix][client]Fixed getting an incorrect maxMessageSize value when accessing multiple clusters in the same process (#22306)
• [improve][client] Add backoff for seek (#20963)

Pulsar IO and Pulsar Functions

• [fix][fn] enable Go function token auth and TLS (#20468)

Others

• [improve][misc] Remove the call to sun InetAddressCachePolicy (#22329)
• [fix][misc] Make ConcurrentBitSet thread safe (#22361)
• [fix][ws] Check the validity of config before start websocket service (#22231)

Library updates

• [fix][sec] Upgrade jose4j to 0.9.4 (#22273)
• [fix][sec] Bump google.golang.org/grpc from 1.38.0 to 1.56.3 in /pulsar-function-go (#21444)
• [fix][sec] Go Functions security updates (#21844)
• [fix][sec] Upgrade Zookeeper to 3.9.2 to address CVE-2024-23944 (#22275)
• [fix][sec] Upgrade prometheus client_golang to v1.12.2 to fix CVE-2022-21698 (#20579)
• [fix][build] Upgrade alluxio version to 2.9.3 to fix CVE-2023-38889 (#21715)
• [improve][misc] Include native epoll library for Netty for arm64 (#22319)
• [improve][misc] Pin Netty version in pulsar-io/alluxio (#21728)
• [improve][misc] Upgrade Netty version to 4.1.105.Final (#21923)

Tests & CI

• [fix][ci] Tolerate mount option change failing in CI (#22241)
• [fix][ci][branch-3.0] Increase Maven's heap size from 1024m to 1500m as it is in master
• [fix][test] Fix flaky ManagedLedgerErrorsTest.recoverAfterZnodeVersionError (#22368)
• [fix][test] Fix flaky RGUsageMTAggrWaitForAllMsgsTest (#22252)
• [fix][test] Fix flaky test BrokerServiceAutoSubscriptionCreationTest (#22190)
• [fix][test] Fix flaky test ManagedLedgerTest.testGetNumberOfEntriesInStorage (#22344)
• [fix][test][branch-3.0] Fix broken ManagedLedgerTest.testGetNumberOfEntriesInStorage
• [fix][test] Fix the tests with same namespace name (#22240)
• [improve][test][branch-3.0] Improve ManagedLedgerTest.testGetNumberOfEntriesInStorage
• [improve][misc] Upgrade checkstyle to 10.14.2 (#22291)

For the complete list, check the full changelog.

What's Changed

Broker

• [improve][broker] Consistently add fine-grain authorization to REST API 22202

• [improve][broker] Add fine-grain authorization to retention admin API 22163

• [improve][broker] Cache the internal writer when sent to system topic. 22099

• [improve][broker] Add an error log to troubleshoot the failure of starting broker registry. 22065

• [improve][broker] Do not retain the data in the system topic 22022

• [improve] [broker] Do not try to open ML when the topic meta does not exist and do not expect to create a new one. #21995 22004

• [improve] [broker] Do not print an Error log when responding to HTTP-404 when calling Admin API and the topic does not exist. 21995

• [improve][broker] Do not close the socket if lookup failed due to LockBusyException 21993

• [improve][broker] Don't rollover empty ledgers based on inactivity 21893

• [improve][broker] Improve NamespaceUnloadStrategy error message 21880

• [improve][broker] Skip loading the NAR packages if not configured 21867

• [improve][broker] defer the ownership checks if the owner is inactive (ExtensibleLoadManager) 21857

• [improve][broker] defer the ownership checks if the owner is inactive (ExtensibleLoadManager) 21811

• [improve][broker] cleanup the empty subscriptionAuthenticationMap in zk when revoke subscription permission 21696

• [improve] [broker] Let the producer request success at the first time if the previous one is inactive 21220

• [improve][broker] Avoid record inactiveproducers when deduplication is disable. 21193

• [improve][broker] Support not retaining null-key message during topic compaction (#21578) 21662

• [improve][broker] Print recoverBucketSnapshot log if cursorProperties are empty 21651

• [improve][broker][PIP-318] Support not retaining null-key message during topic compaction 21578

• [fix] [broker] print non log when delete partitioned topic failed 22153

• [fix][broker] Fix broker not starting when both transactions and the Extensible Load Manager are enabled 22139

• [fix][broker] Return getOwnerAsync without waiting on source broker upon Assigning and Releasing and handle role change during role init 22112

• [fix] [broker] Enabling batch causes negative unackedMessages due to ack and delivery concurrency 22090

• [fix][broker] Support running docker container with gid != 0 22081

• [fix][broker] Set ServiceUnitStateChannel topic compaction threshold explicitly, improve getOwnerAsync, and fix other bugs 22064

• [fix][broker] Fix hash collision when using a consumer name that ends with a number 22053

• [fix][broker] Sanitize values before logging in apply-config-from-env.py script 22044

• [fix] [broker] Fix can not subscribe partitioned topic with a suffix-matched regexp 22025

• [fix] [broker] Subscription stuck due to called Admin API analyzeSubscriptionBacklog 22019

• [fix][broker][branch-3.1] Avoid PublishRateLimiter use an already closed RateLimiter 22011

• [fix] [broker] [branch-3.0] Fast fix infinite HTTP call createSubscriptions caused by wrong topicName 21997

• [fix][broker]Support setting autoSkipNonRecoverableData dynamically in expiryMon… 21991

• [fix] [broker] add timeout for health check read. 21990

• [fix][broker] Fix schema deletion error when deleting a partitioned topic with many partitions and schema 21977

• [fix] [broker] Fix reader stuck when read from compacted topic with read compact mode disable 21969

• [fix] [broker] Replication stopped due to unload topic failed 21947

• [fix] [broker] Expire messages according to ledger close time to avoid client clock skew 21940

• [fix][broker] Restore the broker id to match the format used in existing Pulsar releases 21937

• [fix][broker] Fix getMessageById throws 500 21919

• [fix] [broker] fix write all compacted out entry into compacted topic 21917

• [fix][broker] Fix deadlock while skip non-recoverable ledgers. 21915

• [fix][broker] Fix leader broker cannot be determined when the advertised address and advertised listeners are configured 21894

• [fix] [broker] Fix break change: could not subscribe partitioned topic with a suffix-matched regexp due to a mistake of PIP-145 21885

• [fix][broker] Fix compaction/replication data loss when expire messages 21865

• [fix][broker]Fix NonPersistentDispatcherMultipleConsumers ArrayIndexOutOfBoundsException 21856

• [fix][broker] Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set 21842

• [fix][broker] Fix String wrong format 21829

• [fix] [broker] Update topic policies as much as possible when some ex was thrown 21810

• [fix][broker] Fix TableViewLoadDataStoreImpl NPE 21777

• [fix][broker] Fix the wrong value of BrokerSrevice.maxUnackedMsgsPerDispatcher 21765

• [fix][broker] Fixed the ExtensibleLoadManagerImpl internal system getTopic failure when the leadership changes 21764

• [fix][broker] Delete compacted ledger when topic is deleted 21745

• [fix][broker] Skip topic auto-creation for ExtensibleLoadManager internal topics 21729

• [fix][broker] Fixed ServiceUnitStateChannel monitor to tombstone only inactive bundle states 21721

• [fix][broker] Avoid compaction task stuck when the last message to compact is a marker 21718

• [fix][broker] Fix the issue of topics possibly being deleted. 21704

• [fix][broker] Fix typo in the config key 21690

• [fix] [broker] network package lost if enable haProxyProtocolEnabled 21684

• [fix][broker] Record GeoPersistentReplicator.msgOut before producer#sendAsync 21673

• [fix][broker] Fix memory leak during topic compaction 21647

• [fix][broker] Fix lookupRequestSemaphore leak when topic not found 21646

• [fix][broker] Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set 21633

• [fix][broker] Fixed getting incorrect KeyValue schema version 21632 (https://github.com/apache/pulsar/pull/21620)

• [fix] [broker] Delete topic timeout due to NPE 21595

• [fix][broker] Fix incorrect unack count when using shared subscription on non-persistent topic 21592

• [fix][broker] Duplicate LedgerOffloader creation when namespace/topic… 21591

• [fix][broker] Correct schema deletion for partitioned topic 21574

• [fix][broker] Fix setReplicatedSubscriptionStatus incorrect behavior 21510

• [fix][broker] Do not write replicated snapshot marker when the topic which is not enable replication 21495

• [fix][broker] Fix issue with consumer read uncommitted messages from compacted topic 21465

• [fix][broker] Fix resource_quota_zpath 21461

• [fix][broker] Fix MultiRoles token provider NPE when using anonymous clients 21429

• [fix][broker] Avoid consumers receiving acknowledged messages from compacted topic after reconnection 21187

• [fix] [broker] Make the new exclusive consumer instead the inactive one faster 21183

• [fix][broker] Fix PulsarService.getLookupServiceAddress returns wrong port if TLS is enabled 21015

• [fix][broker] fix Update contains no change error when use --update-auth-data flag to update function/sink/source 19450

• [improve] [proxy] Add a check for brokerServiceURL that does not support multi uri yet 21972

• [improve][proxy] Fix comment about enableProxyStatsEndpoints 21757

• [improve][proxy] When adding new brokers resolve the DNS name more quickly 21207

• [fix][txn]Fix TopicTransactionBuffer potential thread safety issue 22149

• [fix] [txn] Get previous position by managed ledger. 22024

• [fix][txn] Ack all message ids when ack chunk messages with transaction. 21268

• [fix][txn] Fix getting last message ID when there are ongoing transactions 21466

• [fix] [ml] Fix retry mechanism of deleting ledgers to invalidate 21869

• [fix][ml] Make mlOwnershipChecker asynchronous so that it doesn't block/deadlock threads 21333

• [improve][ml] Filter out deleted entries before read entries from ledger. 21739

• [fix][offload] Fix Offload readHandle cannot close multi times. 22162

• [fix][offload] Don't cleanup data when offload met MetaStore exception 21686

• [fix] [ci] [branch-3.0] Fix the build issue from cherry-pick 22042

• [fix][sec] Add a check for the input time value 22023

• [fix] [log] fix the vague response if topic not found 20932

CLI

• [fix][admin] Fix KeyValue schema compatibility check caused OOM 21645
• [improve][admin] Add clusters check when set replication clusters 21650-

Clients

• [fix] [client] fix huge permits if acked a half batched message 22091
• [fix] [client] Do no retrying for error subscription not found when disabled allowAutoSubscriptionCreation 22078
• [fix][client] Fix ConsumerBuilderImpl#subscribe silent stuck when using pulsar-client:3.0.x with jackson-annotations prior to 2.12.0 21985
• [fix][client] Fix multi-topics consumer could receive old messages after seek 21945
• [cleanup][client] Fix inconsistent API annotations of getTopicName [21620]
• [fix][client] fix negative message re-delivery twice issue 20750
• [fix][client] Fix messages in the batch container timed out unexpectedly 21889
• [improve] [client] Prevent reserve memory with a negative memory size to avoid send task stuck 21804
• [fix][client] Fix producer thread block forever on memory limit controller 21790
• [fix] [client] Messages lost due to TopicListWatcher reconnect 21853

Pulsar IO and Pulsar Functions

• [improve][fn] Add missing "exception" argument to some log.error 22140
• [improve][fn] Optimize Function Worker startup by lazy loading and direct zip/bytecode access 22122
• [fix][fn] Fix Deadlock in Functions Worker LeaderService 21711
• [fix][fn] Use unified PackageManagement service to download packages 21955
• [improve][io] Make connectors load sensitive fields from secrets 21675

Others

• [fix][test] Fix test testAsyncFunctionMaxPending 22121
• [fix][test] fix test testSyncNormalPositionWhenTBRecover 22120
• [fix][test] Fix test testTransactionBufferMetrics 22117
• [fix][test] Fix PerformanceProducer send count error 21706
• [fix][test] ProxyWithoutServiceDiscoveryTest should enable authz 20348
• [cleanup] Consolidate certs used in tests 20336
• [cleanup] Deduplicate test certificates to simplify management 20289
• [cleanup] Consolidate certs in broker (and some proxy) tests 20353
• [fix][test] Make base test class method protected so that it passes ReportUnannotatedMethods validation 21976
• [improve][build] Add a default username in the image 21695
• [fix][build] Fix Stage Docker images fail on M1 Mac 21659

Library updates

• [fix][sec] Upgrade Jetty to 9.4.54.v20240208 to address CVE-2024-22201 22144
• [fix][sec] Upgrade rabbitmq client to address CVE-2023-46120 21619
• [fix][sec] Upgrade commons-compress to 1.26.0 22086
• [fix][sec] Exclude avro from hadoop-client 21719
• [improve][ci] Upgrade pulsar-client-python to 3.4.0 to avoid CVE-2023-1428 21899
• [fix][sec] Upgrade org.bouncycastle:bc-fips to 1.0.2.4 21730
• [improve][build] Upgrade Apache ZooKeeper to 3.9.1 20933
• [fix][sec] Upgrade snappy-java to 1.1.10.5 21280
• [improve] [bk] Upgrade BookKeeper dependency to 4.16.4 21983

New Contributors

• @zifengmo made their first contribution in https://github.com/apache/pulsar/pull/22078

cherry-picked: https://github.com/apache/pulsar/pulls?q=is%3Apr+is%3Amerged+label%3Arelease%2F3.0.3+label%3Acherry-picked%2Fbranch-3.0+sort%3Acreated-asc+

Full Changelog: https://github.com/apache/pulsar/compare/v3.0.2...v3.0.3

Broker

• [improve][broker] Consistently add fine-grain authorization to REST API #22202 (@mattisonchao)
• [improve][broker] Add fine-grain authorization to retention admin API #22163 (@mattisonchao)
• [improve][broker] Cache the internal writer when sent to system topic. #22099 (@Technoboy-)
• [improve][broker] Add an error log to troubleshoot the failure of starting broker registry. #22065 (@liangyepianzhou)
• [improve][broker] Do not retain the data in the system topic #22022 (@liangyepianzhou)
• [improve][broker] Do not try to open ML when the topic meta does not exist and do not expect to create a new one. #21995 #22004 (@poorbarcode)
• [improve][broker] Do not print an Error log when responding to HTTP-404 when calling Admin API and the topic does not exist. #21995 (@poorbarcode)
• [improve][broker] Do not close the socket if lookup failed due to LockBusyException #21993 (@BewareMyPower)
• [improve][broker] Don't rollover empty ledgers based on inactivity #21893 (@lhotari)
• [improve][broker] Improve NamespaceUnloadStrategy error message #21880 (@dragosvictor)
• [improve][broker] Skip loading the NAR packages if not configured #21867 (@BewareMyPower)
• [improve][broker] defer the ownership checks if the owner is inactive (ExtensibleLoadManager) #21811 (@heesung-sn)
• [improve][broker] cleanup the empty subscriptionAuthenticationMap in zk when revoke subscription permission #21696 (@TakaHiR07)
• [improve][broker] Not close the socket if lookup failed caused by bundle unloading or metadata ex #21211 (@poorbarcode)
• [fix][broker][branch-3.1] Fix broker not starting when both transactions and the Extensible Load Manager are enabled #22194 (@dragosvictor)
• [fix][broker] print non log when delete partitioned topic failed #22153 (@poorbarcode)
• [fix][broker] Return getOwnerAsync without waiting on source broker upon Assigning and Releasing and handle role change during role init #22112 (@heesung-sn)
• [fix][broker] Enabling batch causes negative unackedMessages due to ack and delivery concurrency #22090 (@poorbarcode)
• [fix][broker] Support running docker container with gid != 0 #22081 (@lhotari)
• [fix][broker] Set ServiceUnitStateChannel topic compaction threshold explicitly, improve getOwnerAsync, and fix other bugs #22064 (@heesung-sn)
• [fix][broker] Fix hash collision when using a consumer name that ends with a number #22053 (@lhotari)
• [fix][broker] Sanitize values before logging in apply-config-from-env.py script #22044 (@lhotari)
• [fix][broker] Fix can not subscribe partitioned topic with a suffix-matched regexp #22025 (@poorbarcode)
• [fix][broker] Subscription stuck due to called Admin API analyzeSubscriptionBacklog #22019 (@poorbarcode)
• [fix][broker][branch-3.1] Avoid PublishRateLimiter use an already closed RateLimiter #22011 (@coderzc)
• [fix][broker] Support setting autoSkipNonRecoverableData dynamically in expiryMon… #21991 (@chenhongSZ)
• [fix][broker] add timeout for health check read. #21990 (@thetumbled)
• [fix][broker] Fix schema deletion error when deleting a partitioned topic with many partitions and schema #21977 (@heesung-sn)
• [fix][broker] Fix reader stuck when read from compacted topic with read compact mode disable #21969 (@thetumbled)
• [fix][broker] Replication stopped due to unload topic failed #21947 (@poorbarcode)
• [fix][broker] Expire messages according to ledger close time to avoid client clock skew #21940 (@315157973)
• [fix][broker] Restore the broker id to match the format used in existing Pulsar releases #21937 (@lhotari)
• [fix][broker] Fix getMessageById throws 500 #21919 (@nodece)
• [fix][broker] fix write all compacted out entry into compacted topic #21917 (@thetumbled)
• [fix][broker] Fix deadlock while skip non-recoverable ledgers. #21915 (@hrzzzz)
• [fix][broker] Fix leader broker cannot be determined when the advertised address and advertised listeners are configured #21894 (@lhotari)
• [fix][broker] Fix break change: could not subscribe partitioned topic with a suffix-matched regexp due to a mistake of PIP-145 #21885 (@poorbarcode)
• [fix][broker] Fix compaction/replication data loss when expire messages #21865 (@coderzc)
• [fix][broker] Fix NonPersistentDispatcherMultipleConsumers ArrayIndexOutOfBoundsException #21856 (@Technoboy-)
• [fix][broker] Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set #21842 (@Technoboy-)
• [fix][broker] Fix TableViewLoadDataStoreImpl NPE #21777 (@Demogorgon314)
• [fix][broker] Fix the wrong value of BrokerSrevice.maxUnackedMsgsPerDispatcher #21765 (@aloyszhang)
• [fix][broker] Fixed the ExtensibleLoadManagerImpl internal system getTopic failure when the leadership changes #21764 (@heesung-sn)
• [fix][broker] Delete compacted ledger when topic is deleted #21745 (@coderzc)
• [fix][broker] Skip topic auto-creation for ExtensibleLoadManager internal topics #21729 (@heesung-sn)
• [fix][broker] Avoid compaction task stuck when the last message to compact is a marker #21718 (@coderzc)
• [fix][broker] Fix the issue of topics possibly being deleted. #21704 (@crossoverJie)
• [fix][broker] network package lost if enable haProxyProtocolEnabled #21684 (@poorbarcode)
• [fix][broker] Fix memory leak during topic compaction #21647 (@coderzc)
• [fix][broker] Fix lookupRequestSemaphore leak when topic not found #21646 (@pengxiangrui127)
• [fix][broker] Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set #21633 (@coderzc)
• [fix][broker] Fixed getting incorrect KeyValue schema version #21632 (@Technoboy-)
• [fix][broker] Fix issue with GetMessageIdByTimestamp can't find match messageId from compacted ledger #21600 (@coderzc)
• [fix][broker] Fix incorrect unack count when using shared subscription on non-persistent topic #21592 (@1Jack2)
• [fix][broker] Correct schema deletion for partitioned topic #21574 (@poorbarcode)
• [fix][broker] Avoid consumers receiving acknowledged messages from compacted topic after reconnection #21187 (@coderzc)
• [fix][broker] Fix PulsarService.getLookupServiceAddress returns wrong port if TLS is enabled #21015 (@Technoboy-)
• [fix][txn] Fix TopicTransactionBuffer potential thread safety issue #22149 (@dao-jun)
• [fix][txn] Get previous position by managed ledger. #22024 (@thetumbled)
• [fix][txn] Fix getting last message ID when there are ongoing transactions #21466 (@liangyepianzhou)
• [fix][offload] Fix Offload readHandle cannot close multi times. #22162 (@dao-jun)
• [improve][ml] Filter out deleted entries before read entries from ledger. #21739 (@dao-jun)
• [fix][ml] Fix retry mechanism of deleting ledgers to invalidate #21869 (@poorbarcode)
• [fix][ml] Make mlOwnershipChecker asynchronous so that it doesn't block/deadlock threads #21333 (@lhotari)

Client

• [improve][client] Prevent reserve memory with a negative memory size to avoid send task stuck #21804 (@poorbarcode)
• [fix][client] fix huge permits if acked a half batched message #22091 (@poorbarcode)
• [fix][client] Do no retrying for error subscription not found when disabled allowAutoSubscriptionCreation #22078 (@zifengmo)
• [fix][client] Fix ConsumerBuilderImpl#subscribe silent stuck when using pulsar-client:3.0.x with jackson-annotations prior to 2.12.0 #21985 (@Shawyeok)
• [fix][client] Fix multi-topics consumer could receive old messages after seek #21945 (@BewareMyPower)
• [fix][client] Fix messages in the batch container timed out unexpectedly #21889 (@BewareMyPower)
• [fix][client] Messages lost due to TopicListWatcher reconnect #21853 (@poorbarcode)
• [fix][client] Fix producer thread block forever on memory limit controller #21790 (@wenbingshen)

Admin

• [improve][admin] Expose the offload threshold in seconds to the admin #22101 (@zymap)
• [improve][admin] Add clusters check when set replication clusters #21650 (@Technoboy-)
• [fix][admin] Fix KeyValue schema compatibility check caused OOM #21645 (@Technoboy-)

Pulsar IO and Pulsar Functions

• [improve][fn] Add configuration for connector & functions package url sources #22184 (@lhotari)
• [improve][fn] Optimize Function Worker startup by lazy loading and direct zip/bytecode access #22122 (@lhotari)
• [improve][io] Make connectors load sensitive fields from secrets #21675 (@jiangpengcheng)
• [fix][fn] Use unified PackageManagement service to download packages #21955 (@jiangpengcheng)
• [fix][fn] Fix Deadlock in Functions Worker LeaderService #21711 (@Technoboy-)

Proxy

• [improve][proxy] Add a check for brokerServiceURL that does not support multi uri yet #21972 (@poorbarcode)
• [improve][proxy] Fix comment about enableProxyStatsEndpoints #21757 (@lhotari)

Misc

• [improve][bk] Upgrade BookKeeper dependency to 4.16.4 #21983 (@hangc0276)
• [improve][ci] Upgrade pulsar-client-python to 3.4.0 to avoid CVE-2023-1428 #21899 (@BewareMyPower)
• [improve][build] Add a default username in the image #21695 (@zymap)
• [improve][build] Upgrade Apache ZooKeeper to 3.9.1 #20933 (@eolivelli)
• [fix][sec] Upgrade Jetty to 9.4.54.v20240208 to address CVE-2024-22201 #22144 (@lhotari)
• [fix][sec] Upgrade commons-compress to 1.26.0 #22086 (@massakam)
• [fix][sec] Add a check for the input time value #22023 (@liangyepianzhou)
• [fix][sec] Upgrade org.bouncycastle:bc-fips to 1.0.2.4 #21730 (@massakam)
• [fix][sec] Exclude avro from hadoop-client #21719 (@liangyepianzhou)
• [fix][test] Fix test testAsyncFunctionMaxPending #22121 (@liangyepianzhou)
• [fix][test] fix test testSyncNormalPositionWhenTBRecover #22120 (@liangyepianzhou)
• [fix][test] Fix test testTransactionBufferMetrics #22117 (@liangyepianzhou)
• [fix][test] Make base test class method protected so that it passes ReportUnannotatedMethods validation #21976 (@lhotari)
• [fix][test] Fix PerformanceProducer send count error #21706 (@Technoboy-)
• [fix][test] ProxyWithoutServiceDiscoveryTest should enable authz #20348 (@michaeljmarshall)
• [cleanup] Consolidate certs in broker (and some proxy) tests #20353 (@michaeljmarshall)
• [cleanup] Consolidate certs used in tests #20336 (@michaeljmarshall)

Broker

• Consistently add fine-grain authorization to REST API #22202
• Add fine-grain authorization to retention admin API #22163
• Print non log when delete partitioned topic failed #22153
• Fix broker not starting when both transactions and the Extensible Load Manager are enabled #22139
• Return getOwnerAsync without waiting on source broker upon Assigning and Releasing and handle role change during role init #22112
• Cache the internal writer when sent to system topic. #22099
• Enabling batch causes negative unackedMessages due to ack and delivery concurrency #22090
• Support running docker container with gid != 0 #22081
• Add an error log to troubleshoot the failure of starting broker registry #22065
• Set ServiceUnitStateChannel topic compaction threshold explicitly, improve getOwnerAsync, and fix other bugs #22064
• Fix hash collision when using a consumer name that ends with a number #22053
• Sanitize values before logging in apply-config-from-env.py script #22044
• Fix can not subscribe partitioned topic with a suffix-matched regexp #22025
• Do not retain the data in the system topic #22022
• Subscription stuck due to called Admin API analyzeSubscriptionBacklog #22019
• Do not try to open ML when the topic meta does not exist and do not expect to create a new one #22004
• Do not print an Error log when responding to HTTP-404 when calling Admin API and the topic does not exist #21995
• Do not close the socket if lookup failed due to LockBusyException #21993
• Support setting autoSkipNonRecoverableData dynamically in PersistentMessageExpiryMonitor #21991
• Add timeout for health check read #21990
• Fix schema deletion error when deleting a partitioned topic with many partitions and schema #21977
• Defer the ownership checks if the owner is inactive (ExtensibleLoadManager) #21811
• Fix compaction/replication data loss when expire messages #21865
• Skip loading the NAR packages if not configured #21867
• Improve NamespaceUnloadStrategy error message #21880
• Don't rollover empty ledgers based on inactivity #21893
• Expire messages according to ledger close time to avoid client clock skew #21940
• Fix reader stuck when read from compacted topic with read compact mode disable #21969

Client

• Fix multi-topics consumer could receive old messages after seek #21945
• Fix ConsumerBuilderImpl#subscribe silent stuck when using pulsar-client:3.0.x with jackson-annotations prior to 2.12.0 #21985

Pulsar IO and Pulsar Functions

• [FN] Add configuration for connector & functions package url sources #22184
• [FN] Add missing "exception" argument to some log.error #22140
• [FN] Prevent putstate uses empty values #22127
• [FN] Optimize Function Worker startup by lazy loading and direct zip/bytecode access #22122
• [FN] Prevent create state table from API calls for non-exists instances #22107
• [FN] Use unified PackageManagement service to download packages #21955

CLI

• [Admin] Expose the offload threshold in seconds to the admin #22101
• [CLI] Fix the bug when set-retention specified size with -T #22150
• [CLI] Error out when --destinationBroker is set when --bundle is empty #21879

Others

• [MISC] Add a check for the input time value #22023
• [Proxy] Add a check for brokerServiceURL that does not support multi uri yet #21972
• [TX] Get previous position by managed ledger #22024
• [ML] Fix retry mechanism of deleting ledgers to invalidate #21869
• [ML] Filter out deleted entries before read entries from ledger #21739

Library updates

• Upgrade Jetty to 9.4.54.v20240208 to address CVE-2024-22201 #22144
• pgrade commons-compress to 1.26.0 #22086
• Upgrade BookKeeper dependency to 4.16.4 #21983

A special thanks to the following contributors who contributed to Pulsar 3.2.1: mattisonchao, lhotari, poorbarcode, Technoboy-, jiangpengcheng, dragosvictor, freeznet, liangyepianzhou, heesung-sn, zymap, massakam, thetumbled, BewareMyPower, chenhongSZ, Shawyeok, hangc0276, 315157973, coderzc, dao-jun.

What's Changed

Broker

• [fix][broker] Fix break change: could not subscribe partitioned topic with a suffix-matched regexp due to a mistake of PIP-145 (#21885)
• [fix][broker] Fix String wrong format (#21829)
• [fix][broker] Enabling batch causes negative unackedMessages due to ack and delivery concurrency (#22090)
• [fix][broker] Fix can not subscribe partitioned topic with a suffix-matched regexp (#22025)
• [fix][broker] Fix lookupRequestSemaphore leak when topic not found (#21646)
• [fix][broker] Fix the issue of topics possibly being deleted. (#21704)
• [fix][broker] Fix write all compacted out entry into compacted topic (#21917)
• [fix][broker] Replication stopped due to unload topic failed (#21947)
• [fix][broker] Sanitize values before logging in apply-config-from-env.py script (#22044)
• [fix][broker] Support running docker container with gid != 0 (#22081)
• [fix][broker] Update topic policies as much as possible when some ex was thrown (#21810)
• [fix][broker] fix the wrong value of BrokerSrevice.maxUnackedMsgsPerDispatcher (#21765)
• [fix][broker]Delete compacted ledger when topic is deleted (#21745) (#21850)
• [fix][broker]Fix NonPersistentDispatcherMultipleConsumers ArrayIndexOutOfBoundsException (#21856)
• [fix][broker][branch-3.1] Avoid PublishRateLimiter use an already closed RateLimiter (#22011)
• [fix][ml] Fix retry mechanism of deleting ledgers to invalidate (#21869)
• [improve][admin] internalGetMessageById shouldn't be allowed on partitioned topic (#19013)
• [improve][broker] Add fine-grain authorization to retention admin API (#22163)
• [improve][broker] Avoid print redirect exception log when get list from bundle (#20846)
• [improve][broker] Consistently add fine-grain authorization to REST API (#22202)
• [improve][broker] Do not print an Error log when responding to HTTP-404 when calling Admin API and the topic does not exist. (#21995)
• [improve][broker] Make get list from bundle Admin API async (#20652)
• [refactor][broker] Suppress error logging when message expiration fails (#19778)

Misc

• [improve][build] Upgrade Apache ZooKeeper to 3.9.1 (#20933)
• [fix][misc] Bump GRPC version to 1.55.3 to fix CVE (#21057)
• [fix][sec] Bump avro version to 1.11.3 for CVE-2023-39410 (#21341)
• [fix][sec] Exclude avro from hadoop-client (#21719)
• [fix][sec] Upgrade Jetty to 9.4.53 to address CVE-2023-44487 (#21395)
• [fix][sec] Upgrade Jetty to 9.4.54.v20240208 to address CVE-2024-22201 (#22144)
• [fix][sec] Upgrade Netty to 4.1.100 to address CVE-2023-44487 (#21397)
• [fix][sec] Upgrade commons-compress to 1.26.0 (#22086)
• [fix][sec] cve: exclude ch.qos.logback in canal.protocol * resolve CVE-2023-6378 (95e1de78)
• [fix][test] Fix PerformanceProducer send count error (#21706)
• [fix][test] Fix test testTransactionBufferMetrics (#22117)
• [fix][test] Make base test class method protected so that it passes ReportUnannotatedMethods validation (#21976)
• [fix][test] testModularLoadManagerRemoveBundleAndLoad (#19710)

Proxy

• [improve][proxy] Fix comment about enableProxyStatsEndpoints (#21757)

Functions

• [fix][fn] Fix Deadlock in Functions Worker LeaderService (#21711)
• [improve][fn] Add configuration for connector & functions package url sources (#22184)
• [improve][fn] Optimize Function Worker startup by lazy loading and direct zip/bytecode access (#22122)

Clients

• [fix][client] fix huge permits if acked a half batched message (#22091)

Full Changelog: https://github.com/apache/pulsar/compare/v2.11.3...v2.11.4

Broker

Improvements

• [improve][broker] Consistently add fine-grain authorization to REST API 22202
• [improve] [broker] Do not print an Error log when responding to HTTP-404 when calling Admin API and the topic does not exist. 21995
• [improve][proxy] Fix comment about enableProxyStatsEndpoints 21757
• [improve][broker][PIP-318] Support not retaining null-key message during topic compaction 21578
• [improve] [client] Merge lookup requests for the same topic 21232
• [improve] [broker] Let the producer request success at the first time if the previous one is inactive 21220
• [improve] [proxy] Not close the socket if lookup failed caused by too many requests 21216
• [improve] [broker] Not close the socket if lookup failed caused by bundle unloading or metadata ex 21211
• [improve] [broker] Print warn log if ssl handshake error & print ledger id when switch ledger 21201
• [improve] [broker] improve read entry error log for troubleshooting 21169
• [improve][proxy] Support disabling metrics endpoint 21031
• [improve][broker] Improve cache handling for partitioned topic metadata when doing lookup 21063
• [improve][broker] Avoid print redirect exception log when get list from bundle 20846
• [improve][sql] Fix the wrong format of the logs 20907
• [improve][broker] Add consumer-id into the log when doing subscribe. 20568
• [improve] [broker] Print warn log if compaction failure 19405
• [improve][admin]internalGetMessageById shouldn't be allowed on partitioned topic 19013

Fixes

• [fix][sec] Upgrade Jetty to 9.4.54.v20240208 to address CVE-2024-22201 22144
• [fix][sec] Upgrade commons-compress to 1.26.0 22086
• [fix][broker] Support running docker container with gid != 0 22081
• [fix][broker] Sanitize values before logging in apply-config-from-env.py script 22044
• [fix][broker][branch-3.1] Avoid PublishRateLimiter use an already closed RateLimiter 22011
• [fix] [broker] Replication stopped due to unload topic failed 21947
• [fix] [broker] fix write all compacted out entry into compacted topic 21917
• [fix] [ml] Fix retry mechanism of deleting ledgers to invalidate 21869
• [fix][broker]Fix NonPersistentDispatcherMultipleConsumers ArrayIndexOutOfBoundsException 21856
• [fix][broker] Fix String wrong format 21829
• [fix] [broker] Update topic policies as much as possible when some ex was thrown 21810
• [fix][broker] Fix the wrong value of BrokerSrevice.maxUnackedMsgsPerDispatcher 21765
• [fix][sec] Exclude avro from hadoop-client 21719
• [fix][test] Fix PerformanceProducer send count error 21706
• [fix][broker] Fix the issue of topics possibly being deleted. 21704
• [fix][broker] Fix typo in the config key 21690
• [fix] [broker] network package lost if enable haProxyProtocolEnabled 21684
• [fix][broker] Fix memory leak during topic compaction 21647
• [fix][broker] Duplicate LedgerOffloader creation when namespace/topic… 21591
• [fix][broker] Correct schema deletion for partitioned topic 21574
• [fix][broker] Fix namespace bundle stuck in unloading status (#21445) 21567
• [fix][broker] Fix create topic with different auto creation strategies causes race condition 21545
• [fix] [ml] Fix orphan scheduled task for ledger create timeout check 21542
• [fix] [broker] Fix thousands orphan PersistentTopic caused OOM 21540
• [fix][ml] Fix unfinished callback when deleting managed ledger 21530
• [fix][broker] Fix setReplicatedSubscriptionStatus incorrect behavior 21510
• [fix][broker] Do not write replicated snapshot marker when the topic which is not enable replication 21495
• [branch-2.10][fix][proxy] Move status endpoint out of auth coverage 21494
• [fix][broker] Avoid pass null role in MultiRolesTokenAuthorizationProvider 21486
• [fix][broker] Fix the deadlock when using BookieRackAffinityMapping with rackaware policy 21481
• [fix][broker] Fix namespace bundle stuck in unloading status 21445
• [fix][broker] Fix MultiRoles token provider NPE when using anonymous clients 21429
• [fix][proxy] Move status endpoint out of auth coverage 21428
• [fix][sec] Upgrade Netty to 4.1.100 to address CVE-2023-44487 21397
• [fix][sec] Bump avro version to 1.11.3 for CVE-2023-39410 21341
• [fix][test] Fix flaky test NarUnpackerTest 21328
• [fix] [bk-client] Fix bk client MinNumRacksPerWriteQuorum and EnforceMinNumRacksPerWriteQuorum not work problem. 21327
• [fix][ml] Fix thread safe issue with RangeCache.put and RangeCache.clear 21302
• [fix][sec] Upgrade snappy-java to 1.1.10.5 21280
• [fix][txn] Ack all message ids when ack chunk messages with transaction. 21268
• [fix][broker][branch-2.10] Fix inconsistent topic policy 21258
• [fix] [ml] fix wrong msg backlog of non-durable cursor after trim ledgers 21250
• [fix] [ml] Reader can set read-pos to a deleted ledger 21248
• [fix][broker]Fixed produce and consume when anonymousUserRole enabled 21237
• [fix][broker] Fix inconsistent topic policy 21231
• [fix][broker] Fixed reset for AggregatedNamespaceStats 21225
• [fix][broker] Make the new exclusive consumer instead the inactive one faster 21183
• [fix][broker][branch-2.10] Backport fix UniformLoadShedder selecet wrong overloadbroker and underloadbroker 21182
• [Branch-2.10] [imporve] [bookie] Upgrade BookKeeper dependency to 4.14.8 for branch 2.10 21148
• [fix][client] Fix repeat consume when using n-ack and batched messages 21116
• [fix][client] Avoid ack hole for chunk message 21101
• [branch-2.10] [fix] [broker] Fix isolated group not work problem. 21098
• [fix] [broker] Fix isolated group not work problem. 21096
• [fix][broker] Fix write duplicate entries into the compacted ledger after RawReader reconnects 21081
• [fix][client] Fix consumer can't consume resent chunked messages 21070
• [fix][broker] Make sure all inflight writes have finished before completion of compaction 21067
• [fix][broker] Use MessageDigest.isEqual when comparing digests 21061
• [fix][misc] Bump GRPC version to 1.55.3 to fix CVE 21057
• [fix] [bk] Correctct the bookie info after ZK client is reconnected 21035
• [fix][io] Update test certs for Elasticsearch 21001
• [fix][broker] Fix can't stop phase-two of compaction even though messageId read reaches lastReadId 20988
• [fix][broker] Fix message loss during topic compaction 20980
• [fix][broker] Fix incorrect number of read compacted entries 20978
• [fix][broker]Fix chunked messages will be filtered by duplicating 20948
• [fix][broker]Check that the super user role is in the MultiRolesTokenAuthorizationProvider plugin 20939
• [fix] [log] fix the vague response if topic not found 20932
• [fix][broker] fix MessageDeduplication throw NPE when enable broker dedup and set namespace disable deduplication. 20905
• [fix] [ml] fix discontinuous ledger deletion 20898
• [fix][broker] In replication scenario, remote consumer could not be registered if there has no message was sent 20888
• [branch-2.10][fix][broker] Fix inconsensus namespace policies by getPoliciesIfCached 20873
• [branch-2.10][fix][broker] Inconsistent behaviour for topic auto_creation 20872
• [fix][broker] Fix inconsensus namespace policies by getPoliciesIfCached 20855
• [fix][broker] Avoid infinite bundle unloading 20822
• [fix][broker] Fix get topic policies as null during clean cache 20763
• [fix] [broker] do not filter system topic while shedding. 18949
• [improve] Introduce the sync() API to ensure consistency on reads during critical metadata operation paths 18518
• [fix][broker] Fix namespace not found will cause request timeout 18512
• [fix][io] exclude logback dependency for canal connector 18226

Client

Fixes

• [fix][client] Fix ConsumerBuilderImpl#subscribe silent stuck when using pulsar-client:3.0.x with jackson-annotations prior to 2.12.0 21985
• [fix] [client] fix reader.hasMessageAvailable return false when incoming queue is not empty 21259
• [fix][client] fix same producer/consumer use more than one connection per broker 21144
• [fix][client] Fix repeat consume when using n-ack and batched messages 21116
• [fix][client] Avoid ack hole for chunk message 21101
• [fix][client] Fix consumer can't consume resent chunked messages 21070
• [fix] [cli] the variable producerName of BatchMsgContainer is null 20819
• [fix][client] fix negative message re-delivery twice issue 20750

Broker

• [fix][broker] In replication scenario, remote consumer could not be registered if there has no message was sent (#20888)
• [improve] [broker] Print warn log if compaction failure (#19405)
• [branch-2.10][fix][broker] Fix inconsensus namespace policies by getPoliciesIfCached (#20873)
• [branch-2.10][fix][broker] Inconsistent behaviour for topic auto_creation (#20872)
• [fix] [broker] Can not receive any messages after switching to standby cluster (#20767)
• [improve] [broker] Add consumer-id into the log when doing subscribe. (#20568)
• [fix][broker][branch-2.10] Fix NPE when resetting Replicator's cursor by position. (#20597) (#20781)
• [fix][broker] Fix namespace deletion if __change_events topic has not been created yet (#18804)
• [fix][schema] Only handle exception when there has (#20730)
• [fix][broker] Topic policy can not work well if replay policy message has any exception. (#20613)
• [fix][broker] Fix return the earliest position when query position by timestamp. (#20457)
• [fix][broker] Return if AbstractDispatcherSingleActiveConsumer closed (#19934)
• [broker] clean inactive bundle from bundleData in loadData and bundlesCache (#13974)
• [fix][branch-2.10] Fix compilation issue introduced by fixing Repeated messages of shared dispatcher (#16812)
• Issue 16802: fix Repeated messages of shared dispatcher (#16812)
• [fix][branch-2.10]Fix compilation issue introduced by Save createIfMissing in TopicLoadingContext (#19993)
• [improve][broker] Save createIfMissing in TopicLoadingContext (#19993)
• [fix][broker] Invalidate metadata children cache after key deleted (#20363)
• [improve] [broker] Avoid PersistentSubscription.expireMessages logic check backlog twice. (#20416)
• [fix][broker]fix the publish latency spike issue with large number of producers (#20607)
• [fix][branch-2.10] Fix duplicated deleting topics (#20685)
• [improve][broker][branch-2.10] Backport Linux metrics changes from master branch (#20659)
• [cleanup][broker] Validate authz earlier in delete subscription logic (#20549)
• [fix][broker] release orphan replicator after topic closed (#20567)
• [fix][broker] REST Client Producer fails with TLS only (#20535)
• [fix][broker] Restore solution for certain topic unloading race conditions (#20527)
• [fix][ml] There are two same-named managed ledgers in the one broker (#18688)
• [fix][broker] Fix skip message API when hole messages exists (#20326)
• [fix][broker] If ledger lost, cursor mark delete position can not forward (#18620)
• [fix][broker] partitioned __change_events topic is policy topic (#20392)
• [fix][ml] Fix ledger left in OPEN state when enabling inactiveLedgerRollOverTimeMs (#20276)
• [fix][broker] Fix default bundle size used while setting bookie affinity (#20250)
• [fix][broker] Fix the behavior of delayed message in Key_Shared mode (#20233)
• [improve][broker] Get lowest PositionImpl from NavigableSet (#18278)
• [fix] [broker] error TimeUnit to record publish latency (#20074)
• [fix] [broker] In Key_Shared mode: remove unnecessary mechanisms of message skip to avoid unnecessary consumption stuck (#20335)
• [fix][broker]Fix deadlock of metadata store (#20189)
• [fix] [broker] [branch-2.10] Upgrade rocksDB version to 6.16.4 to keep sync with BookKeeper 4.14.7 (#20312)
• [improve] [broker] Skip split bundle if only one broker (#20190)
• [fix][monitor] topic with double quote breaks the prometheus format (#20230)
• [fix][broker] Fix RoaringBitmap.contains can't check value 65535 (#20176)
• [fix][broker] Fix the reason label of authentication metrics (#20030)
• [fix] [broker] Fix infinite ack of Replicator after topic is closed (#20232)
• [fix] [broker] Producer created by replicator is not displayed in topic stats (#20229)
• [fix] [broker] delete topic failed if disabled system topic (#19735)
• [fix] [ml] make the result of delete cursor is success if cursor is deleted (#19825)
• [fix] [broker] Fast fix infinite HTTP call getSubscriptions caused by wrong topicName (#20131)
• [fix][broker] Fix issue where msgRateExpired may not refresh forever (#19759)
• [fix][broker] Fix can't send ErrorCommand when message is null value (#19899)
• [fix][broker] Fix Return value of getPartitionedStats doesn't contain subscription type (#20210)
• [branch-2.10][fix][build] Upgrade swagger version to fix CVE-2022-1471 (#20172)
• [fix][broker] Fix getPartitionedStats miss subscription's messageAckRate (#19870)
• [fix] Use scheduled executor in BinaryProtoLookupService (#20043)
• [improve] [broker] Fix broker restart logic (#20113)

Transaction

• [fix] [txn] fix consumer can receive aborted txn message when readType is replay (#19815)

Pulsar IO and Pulsar Functions

• [fix][io][branch-2.10] Not restart instance when kafka source poll exception. (#20816)
• [fix][offload] Filesystem offloader class not found hadoop-hdfs-client (#20365)
• [fix][fn] Make KubernetesRuntime translate characters in function tenant, namespace, and name during function removal to avoid label errors (#19584)
• [fix][fn] Fix JavaInstanceStarter inferring type class name error (#19896)
• [fix][io] Close the kafka source connector got stuck (#20698)
• [fix][fn] Exit JVM when main thread throws exception (#20689)
• Optimize conusmer pause (#14566)
• [fix][broker] Fix the publish latency spike from the contention of MessageDeduplication (#20647)
• [fix][fn] Configure pulsar admin for TLS (#20533)
• [fix][fn] Go functions must retrieve consumers by non-particioned topic ID (#20413)
• [fix][io] Close the kafka source connector if there is uncaught exception (#20479)
• [fix][fn] Go functions need to use static grpcPort in k8s runtime (#20404)
• [fix][fn] Make pulsar-admin support update py/go with package url (#19897)
• [fix][txn] Fix transaction is not aborted when send or ACK failed (#20240)

CLI

• [fix] [cli] the variable producerName of BatchMsgContainer is null (#20819)
• [fix][ci] Update nar maven plugin version to fix excessive downloads (#20410)

Admin

• [improve][admin] Return BAD_REQUEST on cluster data is null for createCluster (#20346)

Security

• [fix][sec] Upgrade snappy-java to address multiple CVEs (#20604)
• [fix][sec] Upgrade Guava to 32.0.0 to address CVE-2023-2976 (#20459)
• [improve][misc] Upgrade Netty to 4.1.93.Final (#20423)
• [improve][misc] Upgrade Netty to 4.1.89.Final (#19649)
• [fix][sec] Upgrade sqlite-jdbc to resolve CVE-2023-32697 (#20411)

CI & Test

• [fix][test] Replace test call to Auth0 with call to WireMock (#20465)
• [fix][test] Fix flaky testCreateTopicWithZombieReplicatorCursor (#20037) *[fix][flaky-test]NamespaceServiceTest.flaky/testModularLoadManagerRemoveBundleAndLoad (#17487)
• [improve][test][branch-2.10] Backport disabling disk usage threshold for Elastic Testcontainers (#20676)

Others

• [fix][build]Fix compatibility issue caused by #20819 (#20834)
• [fix][build] Mongo is fixed for 2.10.5 (#20810)
• [fix][meta] Bookie Info lost by notification race condition. (#20642)
• [fix] [meta]Switch to the metadata store thread after zk operation (#20303)
• [fix][ws] Remove unnecessary ping/pong implementation (#20733)
• fix: bundle-data metadata leak because of bundlestats was not clean (#17095)
• [fix] [Perf] PerformanceProducer do not produce expected number of messages. (#19775)
• [fix][meta] Adding the missed bookie id in the registration manager. (#20641)
• [fix][build] Don't publish docker image with "latest" tag to docker repository
• [fix][build][branch-2.10] Fix ci-license check (#20505)
• [improve][monitor] Add JVM start time metric (#20381)
• [fix][build] update the zookeeper version to 3.6.4 (#20333)
• [fix][monitor] Fix the partitioned publisher topic stat aggregation bug (#18807)
• [fix][meta] deadlock of zkSessionWatcher when zkConnection loss (#20122)
• [improve][build] Upgrade dependencies to reduce CVE. (#20162)
• [branch-2.10][improve][build] Upgrade snakeyaml version to 2.0 (#20118)

Client

• [fix] [client] Messages lost when consumer reconnect (#20695)
• [fix][client] Make the whole grabCnx() progress atomic (#20595)
• [fix][client] Fix race condition that leads to caching failed CompletableFutures in ConnectionPool (#19661)
• [fix][client] Fix deadlock issue of consumer while using multiple IO threads (#20669)
• [fix][client] Cache empty schema version in ProducerImpl schemaCache. (#19929)
• [fix][fn] Reset idle timer correctly (#20450)
• [fix][client] Release the orphan producers after the primary consumer is closed (#19858)
• [fix][client] Fix DeadLetterProducer creation callback blocking client io thread. (#19930)