Karafka Versions Save

• [Enhancement] Allow for usage of producer variants and alternative producers with ActiveJob Jobs (Pro).
• [Enhancement] Support :earliest and :latest in Karafka::Admin#seek_consumer_group.
• [Enhancement] Align configuration attributes mapper with exact librdkafka version used and not master.
• [Maintenance] Use base64 from RubyGems as it will no longer be part of standard library in Ruby 3.4.
• [Fix] Support migrating via aliases and plan with aliases usage.
• [Fix] Active with default set to false cannot be overwritten
• [Fix] Fix inheritance of ActiveJob adapter karafka_options partitioner and dispatch method.

This release contains BREAKING changes. Make sure to read and apply upgrade notes.

• [Breaking] Drop Ruby 2.7 support.
• [Breaking] Drop the concept of consumer group mapping.
• [Breaking] karafka topics migrate will now perform declarative topics configuration alignment.
• [Breaking] Replace deserializer config with #deserializers in routing to support key and lazy header deserializers.
• [Breaking] Rename Karafka::Serializers::JSON::Deserializer to Karafka::Deserializers::Payload to reflect its role.
• [Feature] Support custom OAuth providers (with a lot of help from bruce-szalwinski-he and hotelengine.com).
• [Feature] Provide karafka topics alter for declarative topics alignment.
• [Feature] Introduce ability to use direct assignments (Pro).
• [Feature] Provide consumer piping API (Pro).
• [Feature] Introduce karafka topics plan to describe changes that will be applied when migrating.
• [Feature] Introduce ability to use custom message key deserializers.
• [Feature] Introduce ability to use custom message headers deserializers.
• [Feature] Provide Karafka::Admin::Configs API for cluster and topics configuration management.
• [Enhancement] Protect critical rdkafka thread executable code sections.
• [Enhancement] Assign names to internal threads for better debuggability when on TTIN.
• [Enhancement] Provide log_polling setting to the Karafka::Instrumentation::LoggerListener to silence polling in any non-debug mode.
• [Enhancement] Provide metadata#message to be able to retrieve message from metadata.
• [Enhancement] Include number of attempts prior to DLQ message being dispatched including the dispatch one (Pro).
• [Enhancement] Provide ability to decide how to dispatch from DLQ (sync / async).
• [Enhancement] Provide ability to decide how to mark as consumed from DLQ (sync / async).
• [Enhancement] Allow for usage of a custom Appsignal namespace when logging.
• [Enhancement] Do not run periodic jobs when LRJ job is running despite polling (LRJ can still start when Periodic runs).
• [Enhancement] Improve accuracy of periodic jobs and make sure they do not run too early after saturated work.
• [Enhancement] Introduce ability to async lock other subscription groups polling.
• [Enhancement] Improve shutdown when using long polling setup (high max_wait_time).
• [Enhancement] Provide Karafka::Admin#read_lags_with_offsets for ability to query lags and offsets of a given CG.
• [Enhancement] Allow direct assignments granular distribution in the Swarm (Pro).
• [Enhancement] Add a buffer to the supervisor supervision on shutdown to prevent a potential race condition when signal pass lags.
• [Enhancement] Provide ability to automatically generate and validate fingerprints of encrypted payload.
• [Enhancement] Support enable.partition.eof fast yielding.
• [Enhancement] Provide #mark_as_consumed and #mark_as_consumed! to the iterator.
• [Enhancement] Introduce graceful #stop to the iterator instead of recommending of usage of break.
• [Enhancement] Do not run jobs schedulers and other interval based operations on each job queue unlock.
• [Enhancement] Publish listeners status lifecycle events.
• [Enhancement] Use proxy wrapper for Admin metadata requests.
• [Enhancement] Use limited scope topic info data when operating on direct topics instead of full cluster queries.
• [Enhancement] No longer raise Karafka::UnsupportedCaseError for not recognized error types to support dynamic errors reporting.
• [Change] Do not create new proxy object to Rdkafka with certain low-level operations and re-use existing.
• [Change] Update karafka.erb template with a placeholder for waterdrop and karafka error instrumentation.
• [Change] Replace statistics.emitted.error error type with callbacks.statistics.error to align naming conventions.
• [Fix] Pro Swarm liveness listener can report incorrect failure when dynamic multiplexing scales down.
• [Fix] K8s liveness listener can report incorrect failure when dynamic multiplexing scales down.
• [Fix] Fix a case where connection conductor would not be released during manager state changes.
• [Fix] Make sure, that all Admin operations go through stabilization proxy.
• [Fix] Fix an issue where coordinator running jobs would not count periodic jobs and revocations.
• [Fix] Fix a case where critically crashed supervisor would raise incorrect error.
• [Fix] Re-raise critical supervisor errors before shutdown.
• [Fix] Fix a case when right-open (infinite) swarm matching would not pass validations.
• [Fix] Make #enqueue_all output compatible with ActiveJob.perform_all_later (oozzal)
• [Fix] Seek consumer group on a topic level is updating only recent partition.

Upgrade Notes

PLEASE MAKE SURE TO READ AND APPLY THEM!

Available here.

This release contains BREAKING changes. Make sure to read and apply upgrade notes.

• [Breaking] Drop Ruby 2.7 support.
• [Breaking] Drop the concept of consumer group mapping.
• [Breaking] karafka topics migrate will now perform declarative topics configuration alignment.
• [Breaking] Replace deserializer config with #deserializers in routing to support key and lazy header deserializers.
• [Breaking] Rename Karafka::Serializers::JSON::Deserializer to Karafka::Deserializers::Payload to reflect its role.
• [Feature] Support custom OAuth providers (with a lot of help from bruce-szalwinski-he and hotelengine.com).
• [Feature] Provide karafka topics alter for declarative topics alignment.
• [Feature] Introduce ability to use direct assignments (Pro).
• [Feature] Provide consumer piping API (Pro).
• [Feature] Introduce karafka topics plan to describe changes that will be applied when migrating.
• [Feature] Introduce ability to use custom message key deserializers.
• [Feature] Introduce ability to use custom message headers deserializers.
• [Feature] Provide Karafka::Admin::Configs API for cluster and topics configuration management.
• [Enhancement] Assign names to internal threads for better debuggability when on TTIN.
• [Enhancement] Provide log_polling setting to the Karafka::Instrumentation::LoggerListener to silence polling in any non-debug mode.
• [Enhancement] Provide metadata#message to be able to retrieve message from metadata.
• [Enhancement] Include number of attempts prior to DLQ message being dispatched including the dispatch one (Pro).
• [Enhancement] Provide ability to decide how to dispatch from DLQ (sync / async).
• [Enhancement] Provide ability to decide how to mark as consumed from DLQ (sync / async).
• [Enhancement] Allow for usage of a custom Appsignal namespace when logging.
• [Enhancement] Do not run periodic jobs when LRJ job is running despite polling (LRJ can still start when Periodic runs).
• [Enhancement] Improve accuracy of periodic jobs and make sure they do not run too early after saturated work.
• [Enhancement] Introduce ability to async lock other subscription groups polling.
• [Enhancement] Improve shutdown when using long polling setup (high max_wait_time).
• [Enhancement] Provide Karafka::Admin#read_lags_with_offsets for ability to query lags and offsets of a given CG.
• [Enhancement] Allow direct assignments granular distribution in the Swarm (Pro).
• [Enhancement] Add a buffer to the supervisor supervision on shutdown to prevent a potential race condition when signal pass lags.
• [Enhancement] Provide ability to automatically generate and validate fingerprints of encrypted payload.
• [Enhancement] Support enable.partition.eof fast yielding.
• [Enhancement] Provide #mark_as_consumed and #mark_as_consumed! to the iterator.
• [Enhancement] Introduce graceful #stop to the iterator instead of recommending of usage of break.
• [Enhancement] Do not run jobs schedulers and other interval based operations on each job queue unlock.
• [Enhancement] Publish listeners status lifecycle events.
• [Enhancement] Use proxy wrapper for Admin metadata requests.
• [Enhancement] Use limited scope topic info data when operating on direct topics instead of full cluster queries.
• [Enhancement] No longer raise Karafka::UnsupportedCaseError for not recognized error types to support dynamic errors reporting.
• [Change] Do not create new proxy object to Rdkafka with certain low-level operations and re-use existing.
• [Change] Update karafka.erb template with a placeholder for waterdrop and karafka error instrumentation.
• [Fix] Pro Swarm liveness listener can report incorrect failure when dynamic multiplexing scales down.
• [Fix] K8s liveness listener can report incorrect failure when dynamic multiplexing scales down.
• [Fix] Fix a case where connection conductor would not be released during manager state changes.
• [Fix] Make sure, that all Admin operations go through stabilization proxy.
• [Fix] Fix an issue where coordinator running jobs would not count periodic jobs and revocations.
• [Fix] Fix a case where critically crashed supervisor would raise incorrect error.
• [Fix] Re-raise critical supervisor errors before shutdown.
• [Fix] Fix a case when right-open (infinite) swarm matching would not pass validations.
• [Fix] Make #enqueue_all output compatible with ActiveJob.perform_all_later (oozzal)
• [Fix] Seek consumer group on a topic level is updating only recent partition.

Upgrade Notes

PLEASE MAKE SURE TO READ AND APPLY THEM!

Available here.

• [Fix] Seek consumer group on a topic level is updating only recent partition.

This release contains BREAKING changes. Make sure to read and apply upgrade notes.

• [Breaking] Drop Ruby 2.7 support.
• [Breaking] Drop the concept of consumer group mapping.
• [Breaking] karafka topics migrate will now perform declarative topics configuration alignment.
• [Breaking] Replace deserializer config with #deserializers in routing to support key and lazy header deserializers.
• [Breaking] Rename Karafka::Serializers::JSON::Deserializer to Karafka::Deserializers::Payload to reflect its role.
• [Feature] Support custom OAuth providers (with a lot of help from bruce-szalwinski-he and hotelengine.com).
• [Feature] Provide karafka topics alter for declarative topics alignment.
• [Feature] Introduce ability to use direct assignments (Pro).
• [Feature] Provide consumer piping API (Pro).
• [Feature] Introduce karafka topics plan to describe changes that will be applied when migrating.
• [Feature] Introduce ability to use custom message key deserializers.
• [Feature] Introduce ability to use custom message headers deserializers.
• [Feature] Provide Karafka::Admin::Configs API for cluster and topics configuration management.
• [Enhancement] Assign names to internal threads for better debuggability when on TTIN.
• [Enhancement] Provide log_polling setting to the Karafka::Instrumentation::LoggerListener to silence polling in any non-debug mode.
• [Enhancement] Provide metadata#message to be able to retrieve message from metadata.
• [Enhancement] Include number of attempts prior to DLQ message being dispatched including the dispatch one (Pro).
• [Enhancement] Provide ability to decide how to dispatch from DLQ (sync / async).
• [Enhancement] Provide ability to decide how to mark as consumed from DLQ (sync / async).
• [Enhancement] Allow for usage of a custom Appsignal namespace when logging.
• [Enhancement] Do not run periodic jobs when LRJ job is running despite polling (LRJ can still start when Periodic runs).
• [Enhancement] Improve accuracy of periodic jobs and make sure they do not run too early after saturated work.
• [Enhancement] Introduce ability to async lock other subscription groups polling.
• [Enhancement] Improve shutdown when using long polling setup (high max_wait_time).
• [Enhancement] Provide Karafka::Admin#read_lags_with_offsets for ability to query lags and offsets of a given CG.
• [Enhancement] Allow direct assignments granular distribution in the Swarm (Pro).
• [Enhancement] Add a buffer to the supervisor supervision on shutdown to prevent a potential race condition when signal pass lags.
• [Enhancement] Provide ability to automatically generate and validate fingerprints of encrypted payload.
• [Enhancement] Support enable.partition.eof fast yielding.
• [Enhancement] Provide #mark_as_consumed and #mark_as_consumed! to the iterator.
• [Enhancement] Introduce graceful #stop to the iterator instead of recommending of usage of break.
• [Enhancement] Do not run jobs schedulers and other interval based operations on each job queue unlock.
• [Enhancement] Publish listeners status lifecycle events.
• [Enhancement] Use proxy wrapper for Admin metadata requests.
• [Enhancement] Use limited scope topic info data when operating on direct topics instead of full cluster queries.
• [Enhancement] No longer raise Karafka::UnsupportedCaseError for not recognized error types to support dynamic errors reporting.
• [Change] Do not create new proxy object to Rdkafka with certain low-level operations and re-use existing.
• [Change] Update karafka.erb template with a placeholder for waterdrop and karafka error instrumentation.
• [Fix] Pro Swarm liveness listener can report incorrect failure when dynamic multiplexing scales down.
• [Fix] K8s liveness listener can report incorrect failure when dynamic multiplexing scales down.
• [Fix] Fix a case where connection conductor would not be released during manager state changes.
• [Fix] Make sure, that all Admin operations go through stabilization proxy.
• [Fix] Fix an issue where coordinator running jobs would not count periodic jobs and revocations.
• [Fix] Fix a case where critically crashed supervisor would raise incorrect error.
• [Fix] Re-raise critical supervisor errors before shutdown.
• [Fix] Fix a case when right-open (infinite) swarm matching would not pass validations.
• [Fix] Make #enqueue_all output compatible with ActiveJob.perform_all_later (oozzal)
• [Fix] Seek consumer group on a topic level is updating only recent partition.

Upgrade Notes

PLEASE MAKE SURE TO READ AND APPLY THEM!

Available here.

This release contains BREAKING changes. Make sure to read and apply upgrade notes.

• [Breaking] Drop Ruby 2.7 support.
• [Breaking] Drop the concept of consumer group mapping.
• [Breaking] karafka topics migrate will now perform declarative topics configuration alignment.
• [Breaking] Replace deserializer config with #deserializers in routing to support key and lazy header deserializers.
• [Breaking] Rename Karafka::Serializers::JSON::Deserializer to Karafka::Deserializers::Payload to reflect its role.
• [Feature] Support custom OAuth providers (with a lot of help from bruce-szalwinski-he and hotelengine.com).
• [Feature] Provide karafka topics alter for declarative topics alignment.
• [Feature] Introduce ability to use direct assignments (Pro).
• [Feature] Provide consumer piping API (Pro).
• [Feature] Introduce karafka topics plan to describe changes that will be applied when migrating.
• [Feature] Introduce ability to use custom message key deserializers.
• [Feature] Introduce ability to use custom message headers deserializers.
• [Enhancement] Assign names to internal threads for better debuggability when on TTIN.
• [Enhancement] Provide log_polling setting to the Karafka::Instrumentation::LoggerListener to silence polling in any non-debug mode.
• [Enhancement] Provide metadata#message to be able to retrieve message from metadata.
• [Enhancement] Include number of attempts prior to DLQ message being dispatched including the dispatch one (Pro).
• [Enhancement] Provide ability to decide how to dispatch from DLQ (sync / async).
• [Enhancement] Provide ability to decide how to mark as consumed from DLQ (sync / async).
• [Enhancement] Allow for usage of a custom Appsignal namespace when logging.
• [Enhancement] Do not run periodic jobs when LRJ job is running despite polling (LRJ can still start when Periodic runs).
• [Enhancement] Improve accuracy of periodic jobs and make sure they do not run too early after saturated work.
• [Enhancement] Introduce ability to async lock other subscription groups polling.
• [Enhancement] Improve shutdown when using long polling setup (high max_wait_time).
• [Enhancement] Provide Karafka::Admin#read_lags_with_offsets for ability to query lags and offsets of a given CG.
• [Enhancement] Allow direct assignments granular distribution in the Swarm (Pro).
• [Enhancement] Add a buffer to the supervisor supervision on shutdown to prevent a potential race condition when signal pass lags.
• [Enhancement] Provide ability to automatically generate and validate fingerprints of encrypted payload.
• [Enhancement] Support enable.partition.eof fast yielding.
• [Enhancement] Provide #mark_as_consumed and #mark_as_consumed! to the iterator.
• [Enhancement] Introduce graceful #stop to the iterator instead of recommending of usage of break.
• [Enhancement] Do not run jobs schedulers and other interval based operations on each job queue unlock.
• [Enhancement] Publish listeners status lifecycle events.
• [Enhancement] Use proxy wrapper for Admin metadata requests.
• [Enhancement] Use limited scope topic info data when operating on direct topics instead of full cluster queries.
• [Enhancement] No longer raise Karafka::UnsupportedCaseError for not recognized error types to support dynamic errors reporting.
• [Change] Do not create new proxy object to Rdkafka with certain low-level operations and re-use existing.
• [Change] Update karafka.erb template with a placeholder for waterdrop and karafka error instrumentation.
• [Fix] Pro Swarm liveness listener can report incorrect failure when dynamic multiplexing scales down.
• [Fix] K8s liveness listener can report incorrect failure when dynamic multiplexing scales down.
• [Fix] Fix a case where connection conductor would not be released during manager state changes.
• [Fix] Make sure, that all Admin operations go through stabilization proxy.
• [Fix] Fix an issue where coordinator running jobs would not count periodic jobs and revocations.
• [Fix] Fix a case where critically crashed supervisor would raise incorrect error.
• [Fix] Re-raise critical supervisor errors before shutdown.
• [Fix] Fix a case when right-open (infinite) swarm matching would not pass validations.
• [Fix] Make #enqueue_all output compatible with ActiveJob.perform_all_later (oozzal)

Upgrade Notes

PLEASE MAKE SURE TO READ AND APPLY THEM!

Available here.

• [Enhancement] Routing based topics allocation for swarm (Pro)
• [Enhancement] Publish the -1 shutdown reason status for a non-responding node in swarm.
• [Enhancement] Allow for using the distribution mode for DataDog listener histogram reporting (Aerdayne).
• [Change] Change internal.swarm.node_report_timeout to 60 seconds from 30 seconds to compensate for long pollings.
• [Fix] Static membership routing evaluation happens too early in swarm.
• [Fix] Close producer in supervisor prior to forking and warmup to prevent invalid memory states.

• [Feature] Provide swarm capabilities to OSS and Pro.
• [Feature] Provide ability to use complex strategies in DLQ (Pro).
• [Enhancement] Support using :partition as the partition key for ActiveJob assignments.
• [Enhancement] Expand Logger listener with swarm notifications.
• [Enhancement] Introduce K8s swarm liveness listener.
• [Enhancement] Use Process.warmup in Ruby 3.3+ prior to forks (in swarm) and prior to app start.
• [Enhancement] Provide app.before_warmup event to allow hooking code loading tools prior to final warmup.
• [Enhancement] Provide Consumer#errors_tracker to be able to get errors that occurred while doing complex recovery.
• [Fix] Infinite consecutive error flow with VPs and without DLQ can cause endless offsets accumulation.
• [Fix] Quieting mode causes too early unsubscribe.

• [Refactor] Ensure that Karafka::Helpers::Async#async_call can run from multiple threads.

• [Feature] Introduce Exactly-Once Semantics within consumers #transaction block (Pro)
• [Feature] Provide ability to multiplex subscription groups (Pro)
• [Feature] Provide Karafka::Admin::Acl for Kafka ACL management via the Admin APIs.
• [Feature] Periodic Jobs (Pro)
• [Feature] Offset Metadata storage (Pro)
• [Feature] Provide low-level listeners management API for dynamic resources scaling (Pro)
• [Enhancement] Improve shutdown process by allowing for parallel connections shutdown.
• [Enhancement] Introduce non_blocking routing API that aliases LRJ to indicate a different use-case for LRJ flow approach.
• [Enhancement] Allow to reset offset when seeking backwards by using the reset_offset keyword attribute set to true.
• [Enhancement] Alias producer operations in consumer to skip #producer reference.
• [Enhancement] Provide an :independent configuration to DLQ allowing to reset pause count track on each marking as consumed when retrying.
• [Enhancement] Remove no longer needed shutdown patches for librdkafka improving multi-sg shutdown times for cooperative-sticky.
• [Enhancement] Allow for parallel closing of connections from independent consumer groups.
• [Enhancement] Provide recovery flow for cases where DLQ dispatch would fail.
• [Change] Make Kubernetes::LivenessListener not start until Karafka app starts running.
• [Change] Remove the legacy "inside of topics" way of defining subscription groups names
• [Change] Update supported instrumentation to report on #tick.
• [Refactor] Replace define_method with class_eval in some locations.
• [Fix] Fix a case where internal Idle job scheduling would go via the consumption flow.
• [Fix] Make the Iterator #stop_partition work with karafka-rdkafka 0.14.6.
• [Fix] Ensure Pro components are not loaded during OSS specs execution (not affecting usage).
• [Fix] Fix invalid action label for consumers in DataDog logger instrumentation.
• [Fix] Fix a scenario where Karafka::Admin#seek_consumer_group would fail because reaching not the coordinator.
• [Ignore] option --include-consumer-groups not working as intended after removal of "thor"