Functional Kafka Streams for Scala
This is the first stable fs2-kafka release to be based on the 3.x series of the Apache Kafka client library. It is available for Scala 2.12, 2.13, and 3.2; all users are encouraged to upgrade. There are some backwards-incompatible API changes, set out below; for further details, see release notes for 3.0.0-RC1 and previous milestone versions.
Development of the 2.x series will now start to wind down, and the 1.x series will be discontinued. New pull requests that make substantial changes should target the 3.x branch.
Apologies to those whose outstanding pull requests weren't merged in time for this release - it was already very overdue. The intention is to start a 4.x branch in the near future to make further backwards-incompatible improvements.
Some changes have been made to how fs2-kafka represents Serializers and Deserializers. The following references Serializers but the same changes have been made to Deserializers.
KeySerializer and ValueSerializerIn some cases, such as when using a schema registry with the fs2-kafka-vulcan module, a given serializer will only be valid either for keys or for values, but not both. This is now represented by the KeySerializer type, representing the ability to serialize a key, and the ValueSerializer type, representing the ability to serialize a value. These are supertypes of the plain Serializer type, which can serialize either a key or a value.
Implementation note: KeySerializer, ValueSerializer and Serializer are aliased subtypes of a new GenericSerializer class, which is contravariantly parameterized with a marker trait KeyOrValue with subtypes Key and Value. User code usually won't need to be concerned with these details.
RecordSerializer (and RecordDeserializer) with ResourceRecordSerializer served two purposes: to distinguish between key and value serializers when summoning them implicitly, and to wrap effectful instantiation of serializers. The former purpose is now served by the KeySerializer and ValueSerializer types; the latter, by cats-effect's Resource type. Usages of RecordSerializer[F, A] are thus now replaced by Resource[F, KeySerializer[F, A]] or Resource[F, ValueSerializer[F, A]].
The AvroSerializer class no longer has a using method that returns RecordSerializer; instead it has two methods, forKey and forValue, that respectively return KeySerializer and ValueSerializer inside Resource. (Equivalent changes have been made to AvroDeserializer.)
The ProducerRecords, TransactionalProducerRecords and ProducerResult classes have been replaced with type aliases for different kinds of fs2.Chunk, which they previously wrapped. This was enabled by removal of "passthrough" functionality, which can instead be implemented in user code if needed.
Full Changelog since v2.6.0: https://github.com/fd4s/fs2-kafka/compare/v2.6.0...v3.0.0
This release previously failed to publish but has now been published successfully.
transform and option methods on RecordSerializer and RecordDeserializer by @geirolz in https://github.com/fd4s/fs2-kafka/pull/1160
Full Changelog: https://github.com/fd4s/fs2-kafka/compare/v2.5.0...v2.6.0
This release candidate builds on previous milestone releases of fs2-kafka 3.0. It includes all changes from v2.6.0 and also introduces the breaking change of removing the RecordDeserializer and RecordSerializer types, in favour of simply using Resource for allocating KeySerializer, ValueSerializer etc. It also reintroduces support for Scala 2.12, to encourage all users to migrate quickly to the new version.
RecordSerializer/RecordDeserializer with Resource by @bplommer in https://github.com/fd4s/fs2-kafka/pull/1175
Full Changelog: https://github.com/fd4s/fs2-kafka/compare/v3.0.0-M9...v3.0.0-RC1
This milestone release replaces v3.0.0-M8, which was released from the wrong branch and should not be used.
apply methods by @bastewart in https://github.com/fd4s/fs2-kafka/pull/978
chunk constructors to {Committable,}ProducerRecords by @bastewart in https://github.com/fd4s/fs2-kafka/pull/979
commitBatch and commitBatchOption in Documentation by @ASRagab in https://github.com/fd4s/fs2-kafka/pull/1022
Full Changelog: https://github.com/fd4s/fs2-kafka/compare/v3.0.0-M7...v3.0.0-M9
This is a milestone for the forthcoming v3.0.0, based on the v3.x series of the Java kafka-clients library. It contains the same changes as v2.5.0; for differences from the v2.x series, see release notes for previous milestones.
apply methods by @bastewart in https://github.com/fd4s/fs2-kafka/pull/978
chunk constructors to {Committable,}ProducerRecords by @bastewart in https://github.com/fd4s/fs2-kafka/pull/979
commitBatch and commitBatchOption in Documentation by @ASRagab in https://github.com/fd4s/fs2-kafka/pull/1022
Full Changelog: https://github.com/fd4s/fs2-kafka/compare/v3.0.0-M7...v3.0.0-M8
This release is identical to v2.5.0-RC1. It is backwards binary compatible with previous stable releases in the 2.x series, but not with milestones for v2.5.0. It includes several enhancements and bug fixes - for full details see notes for v2.5.0-M2, v2.5.0-M3 and v2.5.0-RC1.
produce method to KafkaProducerConnection by @bplommer in https://github.com/fd4s/fs2-kafka/pull/900
apply methods by @bastewart in https://github.com/fd4s/fs2-kafka/pull/978
chunk constructors to {Committable,}ProducerRecords by @bastewart in https://github.com/fd4s/fs2-kafka/pull/979
commitBatch and commitBatchOption in Documentation by @ASRagab in https://github.com/fd4s/fs2-kafka/pull/1022
Full Changelog: https://github.com/fd4s/fs2-kafka/compare/v2.4.0...v2.5.0
This release contains several enhancements and bug fixes. It also reverts two changes from v2.5.0-M2: see below.
apply methods by @bastewart in https://github.com/fd4s/fs2-kafka/pull/978
chunk constructors to {Committable,}ProducerRecords by @bastewart in https://github.com/fd4s/fs2-kafka/pull/979
withSerializers/withDeserializers method, but the same purpose can be served by using Array[Byte] for key and value types.KafkaConsumerActor - see #962 for rationale.Full Changelog: https://github.com/fd4s/fs2-kafka/compare/v2.5.0-M3...v2.5.0-RC1
This milestone release for fs2-kafka 3.x contains the same improvements as v2.5.0-M3.
This milestone release exposes some additional consumer functionality and fixes a long-standing bug/shortcoming in the Vulcan integration module, inherited from the Confluent Kafka Avro serializer, whereby auto-registration of Union and some other schemas would not work as expected.
Full Changelog: https://github.com/fd4s/fs2-kafka/compare/2.5.0-M1...v2.5.0-M3
This release is identical to v2.5.0-M1, which failed to publish.
This release adds some new features and significantly simplifies the internals of the consumer. Although we don't expect these changes to cause any problems, we're releasing this as a milestone to allow for some extra battle testing before the next stable release.
Note: there is a small change to the behaviour of consumers when using partitionedStream or partitionsMapStream. Previously, failure to deserialize a record would cause the entire consumer to fail; now, it will only cause a single partition stream to fail.
produce method to KafkaProducerConnection by @bplommer in https://github.com/fd4s/fs2-kafka/pull/900
KafkaProducerConnection allocates a producer without being tied to particular serializers, allowing creation of multiple KafkaProducer instances that share a single set of connections to the broker. With this change, it is possible to produce records directly from KafkaProducerConnection by providing serializers when calling produce.ProducerSettings and ConsumerSettings to be decoupled from provision of serializers and deserializers, which is particularly useful when creating multiple producers or consumers with different types.Full Changelog: https://github.com/fd4s/fs2-kafka/compare/v2.4.0...2.5.0-M1