The Apache Kafka C/C++ library
v1.4.4 is a maintenance release with the following fixes and enhancements:
rd_kafka_send_offsets_to_transaction() CONCURRENT_TRANSACTION (et.al)
errors as retriable.socket.timeout.ms was ignored when transactional.id was set.Note: there was no v1.4.3 librdkafka release
v1.4.2 is a maintenance release with the following fixes and enhancements:
ssl.ca.location accordingly
if OpenSSL is statically linked or ssl.ca.location is set to probe.enable.random.seed=false (#2795)Note: there was no v1.4.1 librdkafka release
Release asset checksums:
ac50da08be69365988bad3d0c46cd87eced9381509d80d3d0b4b50b2fe9b9fa9
3b99a36c082a67ef6295eabd4fb3e32ab0bff7c6b0d397d6352697335f4e57eb
v1.4.0 is a feature release:
librdkafka now has complete Exactly-Once-Semantics (EOS) functionality, supporting the idempotent producer (since v1.0.0), a transaction-aware consumer (since v1.2.0) and full producer transaction support (in this release). This enables developers to create Exactly-Once applications with Apache Kafka.
See the Transactions in Apache Kafka page for an introduction and check the librdkafka transactions example for a complete transactional application example.
Two security issues have been identified in the SASL SCRAM protocol handler:
sasl.username and sasl.password contained characters that needed escaping, a buffer overflow and heap corruption would occur. This was protected, but too late, by an assertion.Both of these issues are fixed in this release.
fnv1a_random partitioner is compatible with Sarama's NewHashPartitioner partition, easing transition from Sarama to librdkafka-based clients such as confluent-kafka-go.rd_kafka_error_t / RdKafka::Error complex error type which provides error attributes such as indicating if an error is retriable.make install
General:
ERR__TRANSPORT rather than ERR__NODE_UPDATE to avoid an extra error code for the application to handle.Producer:
flush() now works with RD_KAFKA_EVENT_DR
Consumer:
session.timeout.ms in the consumer itself (#2631)max.poll.interval.ms is now only enforced when using subscribe()
consumer_lag calculation for transactional topicstopic debuggingmax.poll.interval.ms is exceeded.rd_kafka_event_message_array() to avoid overflow (#2773)Release asset checksums:
eaf954e3b8a2ed98360b2c76f55048ee911964de8aefd8a9e1133418ec9f48dd
ae27ea3f3d0d32d29004e7f709efbba2666c5383a107cc45b3a1949486b2eb84
This is a feature release adding support for KIP-392 Fetch from follower, allowing a consumer to fetch messages from the closest replica to increase throughput and reduce cost.
consumer_lag in stats when consuming from broker versions <0.11.0.0 (regression in librdkafka v1.2.0).Release asset checksums:
bd3373c462c250ecebea9043fb94597a11bd6e0871d3cde19019433d3f74a99e
465cab533ebc5b9ca8d97c90ab69e0093460665ebaf38623209cf343653c76d2
v1.2.2 fixes the producer performance regression introduced in v1.2.1 which may affect high-throughput producer applications.
debug=msg
Release asset checksums:
7557b37e5133ed4c9b0cbbc3fd721c51be8e934d350d298bd050fcfbc738e551
c5d6eb6ce080431f2996ee7e8e1f4b8f6c61455a1011b922e325e28e88d01b53
Warning: v1.2.1 has a producer performance regression which may affect high-throughput producer applications. We recommend such users to upgrade to v1.3.0
v1.2.1 is a maintenance release:
--disable-c11threads to avoid using libc-provided C11 threads.Release asset checksums:
8b5e95318b190f40cbcd4a86d6a59dbe57b54a920d8fdf64d9c850bdf05002ca
f6be27772babfdacbbf2e4c5432ea46c57ef5b7d82e52a81b885e7b804781fd6
WARNING: There is an issue with SASL GSSAPI authentication on Windows with this release. Upgrade directly to v1.2.1 which fixes the issue.
v1.2.0 is a feature release making the consumer transaction aware.
isolation.level=read_committed) implemented by @mhowlett.linger.ms) on the producer.This release adds consumer-side support for transactions.
In previous releases, the consumer always delivered all messages to the application, even those in aborted or not yet committed transactions. In this release, the consumer will by default skip messages in aborted transactions.
This is controlled through the new isolation.level configuration property which
defaults to read_committed (only read committed messages, filter out aborted and not-yet committed transactions), to consume all messages, including for aborted transactions, you may set this property to read_uncommitted to get the behaviour of previous releases.
For consumers in read_committed mode, the end of a partition is now defined to be the offset of the last message of a successfully committed transaction (referred to as the 'Last Stable Offset').
For non-transactional messages there is no change from previous releases, they will always be read, but a consumer will not advance into a not yet committed transaction on the partition.
linger.ms default was changed from 0 to 0.5 ms to promote some level of batching even with default settings.isolation.level=read_committed ensures the consumer will only read messages from successfully committed producer transactions. Default is read_committed. To get the previous behaviour, set the property to read_uncommitted, which will read all messages produced to a topic, regardless if the message was part of an aborted or not yet committed transaction.rd_kafka_committed() and rd_kafka_offsets_for_times() (@damour, #2393)Conf::c_ptr*() to retrieve the underlying C config object.on_thread_start() and on_thread_exit() interceptors.queue.buffering.max.kbytes max to INT_MAX.General:
linger.ms, this reduces CPU load and lock contention for high throughput producer applications. (#2509)rd_kafka_topic_partition_list_new().enable.ssl.certificate.verification=false (@salisbury-espinosa, #2433)Headers (by Vladimir Sakharuk)|| inside markdown table (@mhowlett)Consumer:
rd_kafka_pause|resume_partitions() synchronous, making sure that a subsequent consumer_poll() will not return messages for the paused partitions (#2455).Producer:
message.timeout.ms=0 is now accepted even if linger.ms > 0 (by Jeff Snyder)max.poll.interval.ms unless in Consumer mode, this saves quite a few memory barries for high-performance Producers.Release asset checksums:
6e57f09c28e9a65abb886b84ff638b2562b8ad71572de15cf58578f3f9bc45ec
eedde1c96104e4ac2d22a4230e34f35dd60d53976ae2563e3dd7c27190a96859
v1.1.0 is a security-focused feature release:
ssl.endpoint.identification.algorithm=https (off by default) to validate the broker hostname matches the certificate. Requires OpenSSL >= 1.0.2.ssl.ca.location), librdkafka will load the CA certs by default from the Windows Root Certificate Store.enable.ssl.certificate.verification=false)%{broker.name} is no longer supported in sasl.kerberos.kinit.cmd since kinit refresh is no longer executed per broker, but per client instance.New configuration properties:
ssl.key.pem - client's private key as a string in PEM formatssl.certificate.pem - client's public key as a string in PEM formatenable.ssl.certificate.verification - enable(default)/disable OpenSSL's builtin broker certificate verification.enable.ssl.endpoint.identification.algorithm - to verify the broker's hostname with its certificate (disabled by default).rd_kafka_conf_set_ssl_cert() to pass PKCS#12, DER or PEM certs in (binary) memory form to the configuration object.rd_kafka_conf() to retrieve the client's configuration objectmessage.timeout.ms max value from 15 minutes to 24 days (@sarkanyi, workaround for #2015)sasl.kerberos.kinit.cmd to first attempt ticket refresh, then acquire.max.poll.interval.ms now correctly handles blocking poll calls, allowing a longer poll timeout than the max poll interval.Release asset checksums:
70279676ed863c984f9e088db124ac84a080e644c38d4d239f9ebd3e3c405e84
123b47404c16bcde194b4bd1221c21fdce832ad12912bd8074f88f64b2b86f2b
v1.0.1 is a maintenance release with the following fixes:
v1.0.0 is a major feature release:
max.poll.interval.ms for high-level consumersThis release also changes configuration defaults and deprecates a set of configuration properties, make sure to read the Upgrade considerations section below.
librdkafka v1.0.0 is API (C & C++) and ABI (C) compatible with older versions of librdkafka, but there are changes to configuration properties that may require changes to existing applications.
The following configuration properties have changed default values, which may require application changes:
acks (alias request.required.acks) default is now all (wait for ack from all in-sync replica brokers), the previous default was 1 (only wait for ack from partition leader) which could cause data loss if the leader broker goes down.enable.partition.eof is now false by default. Applications that rely on ERR__PARTITION_EOF to be emitted must now explicitly set this property to true. This change was made to simplify the common case consumer application consume loop.broker.version.fallback was changed from 0.9 to 0.10 and broker.version.fallback.ms was changed to 0.
Users on Apache Kafka <0.10 must set api.version.request=false and broker.version.fallback=.. to their broker version. For users >=0.10 there is no longer any need to specify any of these properties. See https://github.com/edenhill/librdkafka/wiki/Broker-version-compatibility for more information.topic.metadata.refresh.fast.cnt is no longer used.socket.blocking.max.ms is no longer used.reconnect.backoff.jitter.ms is no longer used, see reconnect.backoff.ms and reconnect.backoff.max.ms.offset.store.method=file is deprecated.offset.store.path is deprecated.offset.store.sync.interval.ms is deprecated.queuing.strategy was an experimental property that is now deprecated.msg_order_cmp was an experimental property that is now deprecated.produce.offset.report is no longer used. Offsets are always reported.auto.commit.enable (topic level) for the simple (legacy) consumer is
now deprecated.Use of any deprecated configuration property will result in a warning when the client instance is created. The deprecated configuration properties will be removed in a future version of librdkafka. See issue #2020 for more information.
The checks for incompatible configuration has been improved, the client
instantiation (rd_kafka_new()) will now fail if incompatible configuration
is detected.
max.poll.interval.ms is enforcedThis release adds support for max.poll.interval.ms (KIP-62), which requires
the application to call rd_kafka_consumer_poll()/rd_kafka_poll() at least every max.poll.interval.ms.
Failure to do so will make the consumer automatically leave the group, causing a group rebalance,
and not rejoin the group until the application has called ..poll() again, triggering yet another group rebalance.
max.poll.interval.ms is set to 5 minutes by default.
This release adds support for Idempotent Producer, providing exactly-once producing and guaranteed ordering of messages.
Enabling idempotence is as simple as setting the enable.idempotence
configuration property to true.
There are no required application changes, but it is recommended to add support for the newly introduced fatal errors that will be triggered when the idempotent producer encounters an unrecoverable error that would break the ordering or duplication guarantees.
See Idempotent Producer in the manual and the Exactly once semantics blog post for more information.
In previous releases librdkafka would maintain open connections to all brokers in the cluster and the bootstrap servers.
With this release librdkafka now connects to a single bootstrap server to retrieve the full broker list, and then connects to the brokers it needs to communicate with: partition leaders, group coordinators, etc.
For large scale deployments this greatly reduces the number of connections between clients and brokers, and avoids the repeated idle connection closes for unused connections.
See Sparse connections in the manual for more information.
Original issue #825.
compression.type=zstd, instead use compression.type=producer, see #2183)max.poll.interval.ms (KIP-62, #1039) to allow long processing times.ERR__FATAL and rd_kafka_fatal_error()) to
raise unrecoverable errors to the application. Currently only triggered
by the Idempotent Producer.rd_kafka_message_status() producer API that may be used from
the delivery report callback to know if the message was persisted to brokers
or not. This is useful for applications that want to perform manual retries
of messages, to know if a retry could lead to duplication.reconnect.backoff.ms and reconnect.backoff.max.ms)../configure --install-deps (and also --source-deps-only) will automatically install dependencies through the native package manager and/or from source.rd_kafka_version() was not thread safedelivery.timeout.ms -> message.timeout.ms aliascnd_timedwait*() could leave the cond signalled, resulting in high CPU usage.