🔗 A multipurpose Kafka Connect connector that makes it easy to parse, transform and stream any file, in any format, into Apache Kafka
042cca4e feat(expression): add new built-in function from_bytes 0623e612 feat(expression): add new built-in function extract_struct_field 4bb15d39 feat(expression): add new built-in function parse_url 8484877d feat(filters): add new built-in CSVFilter (#249) 1bb3c9b0 feat(filesystem): add support for AmazonS3MoveCleanupPolicy (#238) a6abb9dc feat(filesystem): allow to configure S3 client RetryPolicy with max.retries and back.off delay (#247)
b37628fd fix(filters): fix wrong arg type for DelimitedRowFilterConfig 5b5b5017 fix(filters): enhance DateFilter to support parsing simple date (#277) c4998d59 fix(api): fix wrong argurments passed to TypedStruct e62dc049 fix(api): fix TypedValue should default to string on non standard type-value (#wq108) 689c9035 fix(filters): fix DelimitedRowFilter to not discarded trailing empty strings (#257) 2d37f89e refactor(filters): refactor and normalize config names for DelimitedRowFilterConfig 82c3597b fix(api): fix Arrayschema to support merging given empty array 3bef574f Fixed Add Support for AWS AssumeRole #251 ab74f57e fix(api): remove dot from connector-name when storing file states (#231) 7a9a3334 last maven version ( > 3.8.1 ) require https repository endpoint
06a18327 docs: update installation guide d8b176bc docs: update AWS config properties 11450aea docs: fix FileSystem Listing section (#246) 934056ca docs: fix getting-started 919b4304 Fix documentation for AppendError
6786156e build(deps): bump maven-surefire-plugin from 3.0.0-M3 to 3.0.0-M7 ba06a87e build(deps): bump maven-source-plugin from 3.2.0 to 3.2.1 ec2e1e85 build(deps): bump libraries-bom from 19.2.1 to 25.4.0 fd3d2ce6 build(deps): bump maven-gpg-plugin from 1.5 to 3.0.1 c416183f build(deps): bump findsecbugs-plugin from 1.10.1 to 1.12.0 4b658557 build(deps-dev): bump json-path from 3.3.0 to 5.1.0 5908d62e build(deps): bump spotbugs from 4.5.3 to 4.7.0 ad72fd00 build(deps): bump jackson-core from 2.13.2 to 2.13. d3a084aa build(deps): bump antlr4.version from 4.7.1 to 4.10.1 d85ce863 test: add junit5 and fix integration-test b50e247a build: add spotbugs maven plugin 529a44fd build(deps-dev): bump testcontainers from 1.11.1 to 1.17.2 41fca730 build(deps): bump json from 20210307 to 20220320 276e0ba9 build(deps): bump github/codeql-action from 1 to 2 38855e84 build(deps): bump actions/setup-node from 1 to 3 3dc560e3 build(deps): bump jackson-annotations from 2.13.2 to 2.13.3 4069ffd1 build(deps): bump maven-jar-plugin from 3.0.2 to 3.2.2 fd07ee39 build(deps): bump avro from 1.10.1 to 1.11.0 90613b5e build(deps): bump actions/checkout from 2 to 3 4a8da8dc build(deps): bump org.apache.kafka.version from 2.8.0 to 3.2.0 1e5d48f4 build(deps): bump kafka-connect-maven-plugin from 0.11.3 to 0.12.0 9453b5dd build(deps): bump docker/login-action from 1 to 2 350f49e2 build(deps): bump actions/setup-java from 2 to 3 04155ac2 build(deps): bump docker/build-push-action from 2 to 3 265e03eb build(deps): bump jackson-databind from 2.12.6.1 to 2.13.3 27f0bfdf build(deps): bump Saxon-HE from 9.9.1-5 to 11.3 45952f9c ci: update git actions workflows 364d6b73 build: bump version for next iteration c836ded7 build: fix jackson version adfec263 gh: add issue template 329536db build(deps): bump jackson-databind from 2.12.2 to 2.12.6.1
$ docker pull streamthoughts/kafka-connect-file-pulse:2.7.0
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.7.0.zip https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.7.0/streamthoughts-kafka-connect-file-pulse-2.7.0.zip
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.7.0.zip.asc https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.7.0/streamthoughts-kafka-connect-file-pulse-2.7.0.zip.asc
$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.7.0.zip.asc streamthoughts-kafka-connect-file-pulse-2.7.0.zip
Full Changelog: https://github.com/streamthoughts/kafka-connect-file-pulse/compare/v2.6.0...v2.7.0
10259c57 feat(plugin): add built-in SizeFileListFilter
32d55e44 fix(api): fix typed value parsing may result in a loss of precision d17298fe fix(plugin): add generation-id prop to task-config to force reconfiguraton
a8ddbf1d script: update debug.sh
$ docker pull streamthoughts/kafka-connect-file-pulse:2.6.0
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.6.0.zip https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.6.0/streamthoughts-kafka-connect-file-pulse-2.6.0.zip
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.6.0.zip.asc https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.6.0/streamthoughts-kafka-connect-file-pulse-2.6.0.zip.asc
$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.6.0.zip.asc streamthoughts-kafka-connect-file-pulse-2.6.0.zip
e8cbbf25 (origin/master, origin/HEAD) fix(api): fix filter-pipeline to always copy record context between filters (#235) f639209f fix(filesystem): update LocalFileStorage to fallback to a simple move implements as copy+delete (#234)
$ docker pull streamthoughts/kafka-connect-file-pulse:2.5.2
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.5.2.zip https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.5.2/streamthoughts-kafka-connect-file-pulse-2.5.2.zip
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.5.2.zip.asc https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.5.2/streamthoughts-kafka-connect-file-pulse-2.5.2.zip.asc
$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.5.2.zip.asc streamthoughts-kafka-connect-file-pulse-2.5.2.zip
8b1491b9 feat(plugin): allow to keep leading underscores on field names
d60e5746 fix(api): connector should throw an error when merging struct with string 6f6deccf fix(plugin): files should be cleanup when an error occurred while converting data to connect record b308e705 fix(api): fix connect.value.schema is not used when sink topic is computed from record 6aa81239 fix(dataformat): ignore empty XML elements that contain only whitespaces 79deb60c fix(docker): fix docker-compose version
$ docker pull streamthoughts/kafka-connect-file-pulse:2.5.1
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.5.1.zip https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.5.1/streamthoughts-kafka-connect-file-pulse-2.5.1.zip
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.5.1.zip.asc https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.5.1/streamthoughts-kafka-connect-file-pulse-2.5.1.zip.asc
$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.5.1.zip.asc streamthoughts-kafka-connect-file-pulse-2.5.1.zip
b8dd10ba feat(plugin): allow to configure cache.max.size.capacity for InMemoryFileObjectStateBackingStore 4f892a87 feat: add new config props to define file ordering f964de07 feat(dataformat): add new XML config to force content field 2a782834 feat: externalize configs for XML filter and reader b1b8de96 feat: add new config xml.text.node.value.field.name
9163b740 fix(dataformat): fix invalig config and NPE in XMLNodeToStructConverter a3fe2743 fix(api): fix wrong schema for double in TypedStruct (#230) 2884fe0e fix(filters): ignore blank values during DelimitedRowFilter type conversion (#228) d7a47381 fix(filesystem-s3): fix AmazonS3FileSystemListing get stuck when bucket contains more than max response items (#226)
5f517f97 docs: fix config props for XML filter and reader 37adaa94 docs: fix typo and config error on AzureBlobStorageFileSystemListing (#229) a4e28d58 docs: fix yaml c5cb5a3e docs: fix typos and reformat
$ docker pull streamthoughts/kafka-connect-file-pulse:2.5.0
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.5.0.zip https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.5.0/streamthoughts-kafka-connect-file-pulse-2.5.0.zip
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.5.0.zip.asc https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.5.0/streamthoughts-kafka-connect-file-pulse-2.5.0.zip.asc
$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.5.0.zip.asc streamthoughts-kafka-connect-file-pulse-2.5.0.zip
fbe8b6f7 feat(scel-api): add is_empty expression for string/array fields
22325790 fix(plugins): fix task resources should be closed correctly (#224) 8d266f15 fix(amazons3-fs): fix NPE hen configuring credential provider (#219)
$ docker pull streamthoughts/kafka-connect-file-pulse:2.4.3
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.4.3.zip https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.4.3/streamthoughts-kafka-connect-file-pulse-2.4.3.zip
$ curl -L -o streamthoughts-kafka-connect-file-pulse-2.4.3.zip.asc https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.4.3/streamthoughts-kafka-connect-file-pulse-2.4.3.zip.asc
$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.4.3.zip.asc streamthoughts-kafka-connect-file-pulse-2.4.3.zip
948687d5 fix(api): fix stackoverflow while merging incompatible schemas (#208) d7106c5b build(deps): include slf4j-log4j12 only for test (#213) b89a9132 fix(dataformat): ignore XML document node type for converting data (#207)
0fd88e8a fix filename typo
01e4b4c2 Bump to actions/setup-java@v2, use Zulu Java 11 4d36de6d test(docker): add akhq to docker-compose-debug.yml 302889e1 fix(build): fix makefile
$ docker pull streamthoughts/kafka-connect-file-pulse:2.4.2
For verifying distirbution zip file
$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.4.2.zip.asc streamthoughts-kafka-connect-file-pulse-2.4.2.zip
c6c9d3b fix(api): schema value for collection should be inferred from all items 680685e bump azure blob version to 12.14.2 3883456 refactor(filesystems): improve logs and add check for file is readable 917ef7b fix(plugin): ensure that completed files are cleaned before starting task(#194) 05a0ba8 fix(plugins): fix invalid object-file should be reported as invalid (#193) 85368d8 builder(deps): bump kafka-connect base-image version (#192) 333e1ee fix(filesystems): fix GcsStorage should support blob URI with underscore (#190) 1818f00 fix(expression): fix MapAdaptable/Reflective accessors should support dotted-notation (#189)
9c6ea7e docs(site): add details about configuring connector with SSL/SASL ae05829 docs(site): add release 2.4 blog post
0967479 build(docker): refactor and cleanup docker build a9fd0b4 build(scripts): remove build-deploy-site.sh ff5f8c6 build(github-action): add workflow git docker 413f8f6 build(github): migrate github-pages build to github actions 68aadda build(deps): update kafka and confluent version 372e27e -build(maven): update java version for ossh
$ docker pull streamthoughts/kafka-connect-file-pulse:2.4.1
For verifying distirbution zip file
$ gpg --keyserver keyserver.ubuntu.com --recv-keys D12696955654914F
$ gpg --verify streamthoughts-kafka-connect-file-pulse-2.4.1.zip.asc streamthoughts-kafka-connect-file-pulse-2.4.1.zip
67683d5 feat(expression): add built-in SCeL expression function NOT 7fea775 feat(dataformat): add config to specify a prefix used to prepend XML attributes (#176) 4fc2cb9 feat(expression): add expression function TimestampDiff 9d72e47 feat(expression): add expression function ToTimestamp 0644cb9 feat(expressions): add built-in function 'gt' and 'lt' to ScEL e4375c8 feat(expressions): add built-in function 'or' and 'and' to ScEL 28a6126 feat(expressions): add built-in function 'if' to ScEL 4fe77fd feat(api): add access to error stacktrace in filter chain b9c0a40 feat(dataformat): add new config prop to exclude node attributes in namespaces (#175) 8f648c8 feat(dataformat): add new config props to exclude all XML attributes (#174) 355b6e4 feat(expression): add UnixTimestamp expression function 5a62f03 feat(filters): add new XmlToStructFilter 9cad2fa feat(filters): add new simple XmlToJsonFilter 0e29ce2 feat(plugin): add capability to merge schemas deriving from records
165a908 refactor(expressions): allow functions to not evaluate all expression args 4e9f84d fix(expressions): fix equals SCeL expression should support null argument (#187) e9cd483 fix(build): normalize artefact-ids 2b8d260 refactor(filters): rellocate json packages 4d13731 refactor(filters): cleanup classes 7bdc787 fix(filesystems): fix regression on AmazonS3Client configuration (#184) d76bac0 fix(plugin): refactor InMemoryFileObjectStateBackingStore to use an LRU cache (#183) 7edf4d7 fix(expression): fix unit tests f3179a7 refactor(expression): refactor expression function api bf3fc31 refactor(expression): reorganize packages for built-in functions 50200f7 fix(plugin): fix resources must not be closed while files are not committed 17e9efb fix(plugin): fix regression cleanup object files should not be rescheduled (#178) e2f74b2 fix(api): fix schemas should be merged per target topic 03bab9a fix(api): enchance mapping to connect schema to handle duplicate schema 760d98b fix(filters): XmlToJson should support bytes input
99c374f fix(api): fix schema behavior on array merge
643469f site(docs): update documentations be29aae docs(site): add new function descriptions 2a9a119 docs(site): fix missing config property 7533d2f docs(site): improve installation guide 71a9ebe docs(site): add doc for defining schema
7003141 update debug.sh 71f7198 improve release.sh
$ docker pull streamthoughts/kafka-connect-file-pulse:2.4.0
3da6a5b feat(filesystems): add the capability to configure alternative AWS S3 endpoint (#172) 608d1c2 feat(plugin): add new prop to cleanup on offset commit f6b443a feat(api): allow to configure a record-value schema befbc6f feat(filters): add new NullValueFilter (#169) d86804b feat(plugin): add new config tasks.empty.poll.wait.ms 39c37d9 feat(plugin): add new prop to configure if task should halt on error (#164) 7c219b8 feat(filesystems/api): enhance XMLFileInputReader to support data type inference (#163)
5314e20 fix(filters): DelimtedRowFileInputFilter should compute schema for each record (#171) 816f48a fix(filters): fix AppendFilter to set record-value to null (#167) d2e776b fix(api): fix connector should accept nullable record-cord (#170) 2266fc1 fix(expression): fix SCEL expression null
20983dd docs(site): fix documentation typos on metadata access (#165) 4eef25f docs(site): add missing config props fdd4124 docs(site): add release note for 2.2.0
Released on Docker hub
docker pull streamthoughts/kafka-connect-file-pulse:2.3.0