A collection of open source Apache 2.0 Kafka Connector maintained by Lenses.io.
We've rolled out enhancements to tackle a common challenge faced by users of the S3 source functionality. Previously, when an external producer abruptly terminated a file without marking the end message, data loss occurred.
To address this, we've introduced a new feature: a property entry for KCQL to signal the handling of unterminated messages. Meet the latest addition, read.text.last.end.line.missing. When set to true, this property ensures that in-flight data is still recognized as a message even when EOF is reached but the end line marker is missing.
#Note
Upgrading from any version prior to 7.0.0, please see the release and upgrade notes for 7.0.0.
This release brings changes which are not compatible with the previous version for S3, GCS and Azure sinks.A migration is required. For migration details please follow the link here.
This release brings substantial enhancements to the data-lakes sink connectors, elevating their functionality and flexibility. The focal point of these changes is the adoption of the new KCQL syntax, designed to improve usability and resolve limitations inherent in the previous syntax.
New KCQL Syntax The data-lakes sink connectors now embrace the new KCQL syntax, offering users enhanced capabilities while addressing previous syntax constraints. Data Lakes Sink Partition Name This update ensures accurate preservation of partition names by avoiding the scraping of characters like \ and /. Consequently, SMTs can provide partition names as expected, leading to reduced configuration overhead and increased conciseness.
Several keywords have been replaced with entries in the PROPERTIES section for improved clarity and consistency:
WITHPARTITIONER: Replaced by PROPERTIES ('partition.include.keys'=true/false). When WITHPARTITIONER KeysAndValue is set to true, the partition keys are included in the partition path. Otherwise, only the partition values are included. WITH_FLUSH_SIZE: Replaced by PROPERTIES ('flush.size'=$VALUE). WITH_FLUSH_COUNT: Replaced by PROPERTIES ('flush.count'=$VALUE). WITH_FLUSH_INTERVAL: Replaced by PROPERTIES ('flush.interval'=$VALUE).
The adoption of the new KCQL syntax enhances the flexibility of the data-lakes sink connectors, empowering users to tailor configurations more precisely to their requirements. By transitioning keywords to entries in the PROPERTIES section, potential misconfigurations stemming from keyword order discrepancies are mitigated, ensuring configurations are applied as intended
This update specifically affects datalake sinks employing the JSON storage format. It serves as a remedy for users who have resorted to a less-than-ideal workaround: employing a Single Message Transform (SMT) to return a Plain Old Java Object (POJO) to the sink. In such cases, instead of utilizing the Connect JsonConverter to seamlessly translate the payload to JSON, reliance is placed solely on Jackson.
However, it's crucial to note that this adjustment is not indicative of a broader direction for future expansions. This is because relying on such SMT practices does not ensure an agnostic solution for storage formats (such as Avro, Parquet, or JSON).
Full Changelog: https://github.com/lensesio/stream-reactor/compare/6.3.1...6.3.1
New Connector The HTTP Sink is offered as beta. Please report any issues via GitHub issues.
New Connector The GCP Storage source is offered as beta. Please report any issues via GitHub issues.
Important AWS S3 Source Partition search properties have changed. See the release notes for detailed information.
All Connectors: In this release, all connectors have been updated to address an issue related to conflicting Antlr jars that may arise in specific environments.
AWS S3 Source: Byte Array Support: Resolved an issue where storing the Key/Value as an array of bytes caused compatibility problems due to the connector returning java.nio.ByteBuffer while the Connect framework's ByteArrayConverter only works with byte[]. This update ensures seamless conversion to byte[] if the key/value is a ByteBuffer.
JMS Sink: Fix for NullPointerException: Addressed an issue where the JMS sink connector encountered a NullPointerException when processing a message with a null JMSReplyTo header value.
JMS Source: Fix for DataException: Resolved an issue where the JMS source connector encountered a DataException when processing a message with a JMSReplyTo header set to a queue.
AWS S3 Sink/GCP Storage Sink (beta)/Azure Datalake Sink (beta): GZIP Support for JSON Writing: Added support for GZIP compression when writing JSON data to AWS S3, GCP Storage, and Azure Datalake sinks.
Three connectors were updated in this release:
The following enhancements were made:
Please Remember The Azure Data Lake and GCP Storage sinks are offered as beta. Please report any issues via GitHub issues.
For the latest version of all other connectors please see Version 6.0.0
Important Package names for several connectors have changed. This may impact your configuration if upgrading. More information about package changes.
New Connectors The Azure Data Lake and GCP Storage sinks are offered as beta. Please report any issues via GitHub issues.