Ververica Flink Cdc Connectors Versions Save

Flink CDC is a streaming data integration tool

release-3.0.1

3 months ago

Release Notes

Documentation: https://ververica.github.io/flink-cdc-connectors/release-3.0

Downloads

Flink CDC 3.0.1 (tar, asc, sha512)

The binary release can be verified by public key with fingerprint F8822354BF424A5CFF9AFD57BAF388953EF5327A.

All connectors are release in JAR and available in Maven central repository.

  • Apache Doris pipeline connector 3.0.1 (jar, asc, sha1)
  • MySQL pipeline connector 3.0.1 (jar, asc, sha1)
  • StarRocks pipeline connector 3.0.1 (jar, asc, sha1)
  • Values pipeline connector 3.0.1 (for testing purpose, jar, asc, sha1)

All connectors are release in JAR and available in Maven central repository.

Improvements and Bug fixes

1.[cdc-base][hotfix] Fix SourceSplitSerializer to get totalFinishedSplitSize from recovery in 4th version 2.[hotfix][cdc-common] Fix NoSuchMethodError of CollectionUtil.newHashMapWithExpectedSize 3.[cdc][e2e] Unify logic to clean docker image when connector e2e test finished (#2978) 4.[pipeline-connector][mysql] fix timestamp with timezone format (#2952) 5.[doc] fix invalid page in website and update yaml in tutorial. 6.[cdc-connector][oracledb] Fix oracle read exceed high_watermark in backfill phase 7.[pipeline-connector][starrocks] Optimize char type mapping for starrocks pipeline connector (#2895) 8.[pipeline-connector][doris][starrocks] Remove SimpleDateFormat for thread safe (#2954) 9.[build] Add github labeler for pipeline connector. (#2851) 10.[docs][cdc-connector][oceanbase] update docker image and os compatibility in quickstart (#2897) 11.[docs] Add Supported Connectors and Supported Flink Versions to Overview. (#2920) 12.[cdc-common][pipeline-doris] Fix typo and wrong value for 'precision' of time type. (#2886) 13.[cdc-connector][mongodb] Avoid mongodb source to read data after high_watermark in backfill phase (#2893) 14.[cdc-connector][oceanbase] Fallback to current timestamp when query gts failure (#2868) 15.[cdc-connector][sqlserver][tests] Fix UT errors by correcting right output (#2864) 16.[cdc] Add Deprecated annotation to source function which connector implemented incremental 17.[hotfix] Remove mysqlToDoris.yml 18.[hotfix] Fix BinaryRecordData deserialize 19.[postgres] Add commits of PostgresConnection back to distinguish with debezium's origin class (#2839) 20.[hotfix][cdc-rumtime][minor] Clean up useless code in BinaryRecordDataSerializer (#2845) 21.[cdc-connector][sqlserver] Fix SqlServerTableSource params order error (#2847) 22.[docs] Fix overview docs to remove prefix of pipeline options (#2833) 23.[3.0][docs] fix mysql pipeline connector option in readme (#2834)

Contributor List (sorted alphabetically)

  1. GOODBOY008
  2. He Wang
  3. Hongshun Wang
  4. joyCurry30
  5. Kunni
  6. Shawn Huang
  7. TorinJie
  8. Xin Gong
  9. gongzhongqiang
  10. xuzifu666
  11. 张田

release-3.0.0

5 months ago

Release Notes

Documentation: https://ververica.github.io/flink-cdc-connectors/release-3.0

Downloads

Flink CDC 3.0.0 (tar, asc, sha512)

The binary release can be verified by public key with fingerprint C460A307EAF2701FC9D774EF96719B746D3D0C8A.

All connectors are release in JAR and available in Maven central repository.

  • Apache Doris pipeline connector 3.0.0 (jar, asc, sha1)
  • MySQL pipeline connector 3.0.0 (jar, asc, sha1)
  • Starrocks pipeline connector 3.0.0 (jar, asc, sha1)
  • Values pipeline connector 3.0.0 (for testing purpose, jar, asc, sha1)

All connectors are release in JAR and available in Maven central repository.

Features and Improvements

  1. [pipeline-connector][starrocks] Fix char/varchar length inconsistency between cdc and starrocks (#2830)
  2. [cdc-cli][cdc-composer] Use "source-table" and "sink-table" as keywords of route (#2825)
  3. [build][hotfix] Remove duplicate dependency (#2826)
  4. [cdc-cli][cdc-dist] Support loading global config from FLINK_CDC_HOME (#2822)
  5. [cdc-common][hotfix] Remove redundant prefix of pipeline options (#2821)
  6. [cdc-source-connector][oceanbase][e2e] Add OceanBase e2e test case (#2521)
  7. [build] Use flink-shaded-force-shading to force all submodules to generate dependency-reduced-pom.xml
  8. [build] Use ${project.version} instead of ${revision} for dependency version
  9. [build] Skip maven-deploy-plugin for test modules and flink-cdc-dist
  10. [cdc-pipeline-connector][mysql] Fix precision problem of BIT type conversion (#2820)
  11. [cdc-source-connector][mysql] fix deserialization issue on table metadata binlog event (#2682)
  12. [pipeline-connector][mysql] Remove unnecessary serverTimeZone in DebeziumEventDeserializationSchema (#2816)
  13. [build][hotfix] Remove useless suppressions. (#2817)
  14. [cdc-source-connector][mysql] Fix attempted load of isSuspended in MySqlSplitSerializer::deserializeSplit when cdc version <= 2.2.0 (#2550)
  15. [cdc-pipeline-connector][mysql] Ensure the inference of MEDIUMINT_UNSIGNED type matches INT type in table schema (#2811)
  16. [cdc-connector][oceanbase] Add option for obcdc extra configs (#2543)
  17. [pipeline-connector][mysql] Enable send schema change by default (#2815)
  18. [3.0][cdc-runtime] Remove waitForFlushSuccess field in request handler (#2812)
  19. [pipeline-connector][starrocks] Add starrocks pipeline connector (#2765)
  20. [pipeline-connector][doris] add doris pipeline connector. (#2810)
  21. [cdc-pipeline-connector][mysql] Ensure the precision inference of DECIMAL type matches table schema
  22. [3.0][cdc-common & runtime] Remove useless flink imports
  23. [cdc-pipeline-connector][mysql] Add tests for mysql pipeline connector
  24. [cdc-pipeline-connector][mysql] Support more mysql types
  25. [cdc-pipeleine-connecotr][mysql] Send CreateTableEvent in the mysql pipeline connector
  26. [cdc-pipeline-connector][mysql] Parse the alter statement to generate SchemaChangeEvent
  27. [cdc-pipeline-connector][mysql] Introduce mysql cdc pipeline DataSource
  28. [cdc-cli][cdc-composer][cdc-dist] Support submitting job to general Flink environments (#2807)
  29. [cdc-runtime][hotfix] Setup waitFlushSuccess before responding operator to flush. (#2805)
  30. [cdc-common] Introduce "pipeline.local-time-zone" config option which help handle time zone well (#2797)
  31. [cdc-common][hotfix] Remove useless flink imports (#2803)
  32. [build] Let surefire plugin to keep the full stacktrace on test failure (#2804)
  33. [pipeline-common] Use column name to check whether the schema contains specific column or not. (#2801)
  34. [cdc-composer][route][sink] Add name for route and improve name for sink
  35. [cdc-composer][hotfix] Always chain operator by default
  36. [cdc-composer][sink] Use correct name for sink operator
  37. [cdc-runtime] SchemaRegistry should complete the future after making the checkpoint (#2800)
  38. [cdc-connector][sqlserver][tests] Fix UT errors by increasing the wait time after committing SQL (#2799)
  39. [cdc-composer] Add common and runtime JAR when submit (#2795)
  40. [cdc-connector][debezium] Supports conversion from binary bytes to base64 encoded string (#2435)
  41. [cdc-connector][mysql] MysqlTableSource return metadata in a fixed order (#2578)
  42. [cdc-connector][mysql] Fix snapshot fetch size conf does not take effect (#2766)
  43. [cdc-connector][oracle] Expose ability to skip backfill in snapshot period for Oracle CDC (#2793)
  44. [cdc-connector][oracle] Fix wrong table path during backfill phase which leads to potential data loss (#2707)
  45. [cdc-composer][tests] Add integration tests for FlinkPipelineComposer (#2776)
  46. [cdc-dist] Introduce flink-cdc.sh as the entrypoint of the Flink CDC (#2791)
  47. [3.0] [flink-cdc-dist] Collect all distro resources into a TAR (#2636)
  48. [cdc-connector][sqlserver] Expose ability to skip backfill in snapshot period for SqlServer CDC.(#2783)
  49. [cdc-connector][mysql] Expose ability to skip backfill in snapshot period for MySQL CDC.
  50. [cdc-connector][mongodb] Expose ability to skip backfill in snapshot period for Mongodb CDC.
  51. [cdc-connector][postgres] Expose ability to skip backfill in snapshot period for Postgres CDC.
  52. [cdc-connector][base] Support skip snapshot backfill for incremental snapshot connector
  53. [cdc-connector][base][tests] Introduce snapshot phase hooks for better snapshot test
  54. [cdc-composer] Add routing logic into the Flink composer
  55. [cdc-cli][cdc-composer] Use sourceTable and sinkTable as keyword in route definition
  56. [cdc-runtime] Add RouteFunction for routing change events
  57. [cdc-common] Introduce AssertJ-style assertion system for Flink CDC data structures
  58. [hotfix][mysql][tests] Avoid canceling a finished job in tests of SpecificStartingOffsetITCase (#2687)
  59. [docs][zh][hotfix] Remove redundant sections from mysql-cdc documentation (#2777)
  60. [mongodb][hotfix] Fix MongoDB's rewriteOutputBuffer to emit +I rather than +U (#2760) (#2760)
  61. [cdc-common] Introduce createFieldGetters method in SchemaUtils to build FieldGetters of given Schema. (#2762)
  62. [cdc-composer] use DataSinkWriterOperatorFactory to replace all SinkWriterOperatorFactory. (#2773)
  63. [cdc-pipeline-connector][values] Avoid using default as name of namespace and schema. (#2774)
  64. [cdc-runtime] Improve DataSinkWriterOperator logic that only emit latest schema for events that are not CreateTableEvent (#2771)
  65. [cdc-common] add PublicEvolving annotation to SchemaUtils (#2756)
  66. [3.0][cdc-composer] Introduce pipeline.name config to support custom flink job name (#2768)
  67. [cdc-base] Extract the common parts of each ScanFetchTask into AbstractScanFetcherTask.
  68. [docs][hotfix] Configuration 'slot.name' is required in mysql-postgres-tutorial-zh.md (#2763)
  69. [cdc-runtime] Improve partitioning logics and add tests for PrePartitionOperator(#2761)
  70. [cdc-runtime][tests] Introduce Harness test utilities for CDC customized operators
  71. [cdc-runtime] Make sure CreateTableEvent is always sent before DataChangeEvent even if during restoration (#2767)
  72. [docs][hotfix] Configuration 'slot.name' is required in mysql-postgres-tutorial.md (#2764)
  73. [oceanbase] Use global timestamp to start log client (#2565)
  74. [cdc-pipeline-connector][values] Add event set MULTI_SPLITS_SINGLE_TABLE to mock source with multiple splits (#2737)
  75. [hotfix][cdc-composer] Use correct factory to discovery DataSink (#2757)
  76. [3.0][cdc-common] Introduce BinaryRecordDataGenerator to help to create BinaryRecordData
  77. [3.0][cdc-common] Use BinaryRecordData in DataChangeEvent instead of GenericRecordData
  78. [3.0][cdc-common] Introduce binaryRecordData to avoid type serialization and improve performance as well
  79. [build][hotfix] Use the correct format for labeler.yml (#2758)
  80. [build] Add github labeler for pull request
  81. [docs] Add DataStream application package guidance (#2583)
  82. [cdc-common] Introduce TableFilter to filter table objects from given TableId pattern (#2673)
  83. [cdc-composer] Initialize context with existing config of SourceDef and SinkDef (#2749)
  84. [hotfix][docs] Adjust the correct doc versions (#2679)
  85. [docs][oracle] Fix oracle cdc quick start (#2727)
  86. [cdc-runtime] Use CollectCoordinationResponse as a wrapper of CoordinationResponse
  87. [cdc-common] MetadataApplier should extends Serializable
  88. [cdc-runtime] Rewrite DataSinkWriterOperator to avoid classloading issues using reflection
  89. [cdc-composer] Implementation of Flink pipeline composer for chaining everything together
  90. [cdc-composer] Introduce partitioning related runtime functions and translator
  91. [cdc-composer] Introduce data sink and schema operator translators
  92. [cdc-composer] Introduce DataSourceTranslator for composing source
  93. [cdc-runtime] Introduce schema manager for managing schema version and handling schema changes
  94. [cdc-composer] Add FlinkEnvironmentUtils for adding JAR to StreamExecutionEnvironment
  95. [cdc-common] Define pipeline options and schema change behaviors
  96. [cdc-common] Support adding Column directly in schema builder
  97. [hotfix][vitess] Fix invalidate package name
  98. [cdc-pipeline-connector][values] Introduce values cdc pipeline connector
  99. [3.0][cdc-common] Introduce TableSchemaState and its Serializer for DataSink to hold the table schemas.
  100. [3.0][cdc-common][minor] Add toString method for some classes.
  101. [3.0][cdc-common] Add SchemaUtil to perform schema evolution (#2732)
  102. [3.0][cdc-runtime] Provide SchemaOperator and SchemaRegistry to handle schema changes (#2685)
  103. [sqlserver] Fix timestamp type parse of sqlserver (#2708)
  104. [3.0][cdc-runtime] Add TypeInformation for Event class (#2728)
  105. [3.0][cdc-runtime][tests] Add tests for type serializers
  106. [3.0][cdc-runtime][hotfix] Move type serializers to runtime module
  107. [3.0][cdc-common] Introduce data type for record field in DataChangeEvent
  108. [3.0][cdc-common][minor] Improve public methods for schema event
  109. [3.0][cdc-common][minor] Move FlushEvent to common module
  110. [mongodb] Incremental snapshot source supports comma-separated format parameter for databases and collections (#2724)
  111. [3.0][cdc-common] Use built-in ConfigOption for cdc-common module (#2716)
  112. [3.0][cdc-common] Add ConText for Source/Sink initialization. (#2672)
  113. [3.0][cdc-common] Add TypeSerializer implementations (#2683)
  114. [build] Add header copyright check for java files
  115. [common] Introduce Configuration and related utilities (#2681)
  116. [oracle] Skip unsupported nested table for legacy debezium source function
  117. [oracle] Skip unsupported nested table for Incremental OracleSource
  118. [3.0][common] Use built-in API annotations to make common module free with flink
  119. [3.0][common] Introduce built-in API annotations for project
  120. [build] Bump Flink version to 1.18.0 (#2463)
  121. [build][tests] Enable CI tests for CDC 3.0 modules (#2678)
  122. [3.0][cdc-common] Introduce internal data structures and generic implementations (#2688)
  123. [composer] Connector factory and JAR discovery utilities (#2662)
  124. [postgres-cdc] Add tests for latest-offset startup strategy (#2527)
  125. [3.0][cdc-common] Change DataChangeEvent from interface to class, and add implementation for DataChangeEvent. (#2677)
  126. [build][tests] Enable CI tests for CDC 3.0 modules
  127. [3.0][cdc-runtime] add DataSinkWriterOperator to process Event (#2649)
  128. [3.0][cdc-common] Add SchemaChangeEvent implementations (#2664)
  129. [cli] Implementation of CLI frontend (#2660)
  130. [docs] Remove flink 1.13 from supported flink versions
  131. [3.0][cdc-common] Add GenericRecordData implementation for RecordData
  132. [3.0][cdc-common] Keep columns in Schema in fixed order
  133. [3.0][cdc-common] Use RecordData to represent before and after record.
  134. [build] Introduce -Dfast property for skipping code style checks
  135. [test] Bump maven-surefire-plugin version to 3.0.0-M5
  136. [test] Introduce JUnit 5 and AssertJ dependencies
  137. [flink-cdc-composer] Base interfaces and models for composer and pipeline definition (#2656)
  138. [3.0][cdc-common] Introduce Connector Factory APIs (#2661)
  139. [3.0][cdc-common] Add getTableSchema method for MetadataAccessor (#2650)
  140. [3.0][cdc-common] Introduce Schema API for CDC 3.0
  141. [3.0][cdc-common] Introduce Event DataSink APIs for Flink CDC 3.0
  142. [3.0][cdc-common] Introduce DataSource related APIs for Flink CDC 3.0
  143. [pom] Use ${revision} replace specific version. (#2623)
  144. [3.0][cdc-runtime] introduce schema evolution framework API
  145. [3.0][cdc-common] Introduce data types for cdc 3.0
  146. [3.0][cdc-common] Introduce Event related APIs for Flink CDC 3.0
  147. [3.0][kickoff] Introduce basic modules for Flink CDC 3.0
  148. [mongodb] Use SampleBucketSplitStrategy for shard collection with hashed keys (#2595)
  149. [vitess][hotfix] Use stable image tag for vitess docker image (#2594)
  150. [oracle] Bump oracle docker image version to 19c
  151. [oracle] Refactor oracle unit test
  152. [build] Bump testcontainers version and polish dependency
  153. [mysql] Use automatic resource management to replace 'try - finally' code block. (#2552)
  154. [sqlserver] Fix get latest lsn (#2551)
  155. [flink-connector-debezium] Support overriding some debezium properties in DebeziumSourceFunction (#2530)
  156. [e2e] Remove db docker image after test cases are executed. (#2508)
  157. [postgres] Reuse pool connection based on database in postgres cdc
  158. [mysql] remove redundant source metrics processTime and emitDelay in mysql source
  159. [cdc-base]remove redundant source metrics register(#2403)
  160. [cdc-base] JdbcConnectionPools supports multiple JDBC sources
  161. [tests][e2e] add log config to e2e tests (#2464)
  162. [postgres] Not drop replication slot for stream split (#2436)
  163. [postgres] Close jdbc connection after creating replication slot for stream split (#2425)
  164. [test] Close resources after each test (#2438)
  165. [hotfix][docs] Fix the abnormal display of pictures (#2513)
  166. [cdc-base] Fix TM hangs caused by uncaught exception (#2511)
  167. [postgres] Close idle readers when snapshot finished (#2400)
  168. [oceanbase][ci] Move the OceanBase CI to free azure pipeline (#2506)
  169. [mongodb][hotfix] Fix Mongo collection regex match logic (#2503)
  170. [hotfix][mysql] Keep assigned splits in order to fix wrong meta group calculation
  171. [mysql] Add finished unack splits to state for the MysqlSourceReader (#2399)
  172. [mongodb][hotfix] Mongo CDC fails to capture collections with . in names (#2488)
  173. [e2e][tidb] Split e2e tests into two groups to decrease the disk usage for per azure job to avoid tidb disk full error
  174. [hotfix][e2e][vitess] Fix module name typo
  175. [e2e] Remove flink 1.13 tests from e2e test matrix
  176. [cdc-base] Fix parsing error when serializing and deserializing the table name with dot (#2443)
  177. [oracle] Correct the naming error (#2405)
  178. [docs] Remove unreachable images in README (#2426)
  179. [docs] Add contribution guidance section in document (#2411)
  180. [sqlserver] Add support bigint type as table split column (#2416)
  181. [hotfix][sqlserver] Fix backfill stream task hang (#2374)
  182. [hotfix][mysql] Trim default value for BIGINT and SMALLINT types when parsing MySQL DDL (#2418)
  183. [sqlserver] Add table filter to speed up SqlServerSchema read (#2369)
  184. [Doc] Use redo log instead of binlog in Oracle document (#2408)
  185. [mysql] Filter databases that do not need to be read when discovering tables (#2160)
  186. [test] Fix TiDBE2eITCase by updating the container version to avoid tikv disk full failure (#2402)
  187. [test][postgres] Close postgres containers after tests and fix container reuse bug (#2394)
  188. [mysql] quota the field names when executing mysql query (#2381) (#2388)
  189. [test][mysql] Provide the server-time-zone setting in MySqlSourceExampleTest#testConsumingAllEvents (#2364)
  190. [hotfix][mysql] Catch underlying throwable in snapshot reading.
  191. [hotfix][mysql] Fix failed test LegacyMySqlSourceTest#testSnapshotOnFailedSource (#2384)
  192. [db2] [tests] Fix unstable Db2ConnectorITCase due to cleanup failure
  193. [vitess] Improve build changelog mode (#2355)
  194. [mongodb][tests] Improve test coverage & add class visibility annotations
  195. [mongodb] Add pendingRecords and numRecordsInErrors metrics for source
  196. [mongodb][hotfix] Fix SourceRecord range check (#2343)
  197. [polish] Add type parameter for MySqlRecords(#2300)
  198. [doc] fix errors in PostgresParallelSourceExample (#2324)
  199. [docs][oracle] Add an example of Incremental Snapshot based DataStream for Oracle CDC Connector
  200. [mongodb] Add disableCursorTimeout option (#2332)
  201. [mongodb] [tests] Loosen E2E timeout limitations
  202. [vitess] Migrate connector properties to debezium 1.9.7 (#2260)
  203. [hotfix][sqlserver] Fix sqlserver monitor same table in other database (#2336)
  204. [debezium] Fix DebeizumSourceFunction can not do savepoint after close (#2259)
  205. [Mysql] Fix assigning duplicate snapshot splits when enable scan newly added tables (#2096)
  206. [mysql] skip closing reader when the reader received the binlog split (#2261)
  207. [MySQL] Optimize how to construct config table.include.list (#2274)
  208. [oracle] Fix class conflict when using multi cdc sql connector
  209. [hotfix][doc] Fix doc site with 404 (#2322)
  210. [docs] Add connector features table to help users quickly understand the supported features of each connector
  211. [mysql] Remove the finished snapshot splits for the binlog split when restoring from the checkpoint (#2292)
  212. [hotfix][sqlserver] Fix sqlserver close idle and chunk key column missing (#2305)
  213. [mongodb] [docs] Adds scanFullChangelog docs (#2294)
  214. [mongodb] Supports scanFullChangelog feature & deprecate copyExisting option
  215. [mongodb] Support MongoDB 6.0+ & update testcases
  216. [postgres] Fix the slot name conflict bug (#2251) (#2281)
  217. [docs] Update Postgres CDC documents for required slot.name option (#2270)
  218. [build] Add vcs.xml for IDEA (#2064)
  219. [docs] add docs for the mysql tables without primary keys (#2227)
  220. [MySQL] Fix the startup mode exception message (#2264)
  221. [oracle] Use oracle connection in context for each reader subtask (#2254) (#2258)
  222. [hotfix] Update support version (#2255)
  223. [hotfix] update dbz option name 'xx. whitelist' to 'xx.include.list'(xx = database, schema or table) (#2212)

Contributor List (sorted alphabetically)

  1. BIN
  2. Dian Qi
  3. EchoLee5
  4. FlechazoW
  5. FocusComputing
  6. Hang Ruan
  7. He Wang
  8. Hongshun Wang
  9. Jiabao Sun
  10. Josh Mahonin
  11. Kunni
  12. Leonard Xu
  13. Maciej Bryński
  14. Malcolmjian
  15. North.Lin
  16. Paddy Gu
  17. PengFei Li
  18. Qingsheng Ren
  19. Shawn Huang
  20. Simonas Gelazevicius
  21. Sting
  22. Tyrantlucifer
  23. TJX2014
  24. Xin Gong
  25. baxinyu
  26. chenlei677
  27. e-mhui
  28. empcl
  29. gongzhongqiang
  30. gaotingkai
  31. ice
  32. joyCurry30
  33. l568288g
  34. lvyanquan
  35. pgm-rookie
  36. rookiegao
  37. skylines
  38. syyfffy
  39. van
  40. wudi
  41. wuzhenhua
  42. yunqingmo
  43. yuxiqian
  44. zhaomin
  45. zyi728
  46. zhangjian

release-2.4.2

6 months ago

Release Notes

Documentation: https://ververica.github.io/flink-cdc-connectors/release-2.4

Improvements and Bug fixes

  1. [sqlserver] Fix get latest lsn (#2551)
  2. [sqlserver] Add support bigint type as table split column (#2416)
  3. [hotfix] Fix vitess docker image tag
  4. [e2e] Remove db docker image after test cases are executed. (#2508)
  5. [postgres] Not drop replication slot for stream split (#2436)
  6. [postgres] Close jdbc connection after creating replication slot for stream split (#2425)
  7. [cdc-base] Fix TM hangs caused by uncaught exception (#2511)
  8. [postgres] Close idle readers when snapshot finished (#2400)
  9. [hotfix][mysql] Keep assigned splits in order to fix wrong meta group calculation
  10. [mysql] Add finished unack splits to state for the MysqlSourceReader (#2399)
  11. [cdc-base] Fix parsing error when serializing and deserializing the table name with dot (#2443)
  12. [oracle] Correct the naming error (#2405)
  13. [sqlserver] Add table filter to speed up SqlServerSchema read (#2369)
  14. [hotfix][sqlserver] Fix backfill stream task hang (#2374)
  15. [hotfix][mysql] Trim default value for BIGINT and SMALLINT types when parsing MySQL DDL (#2418)
  16. [Doc] Use redo log instead of binlog in Oracle document (#2408)
  17. [mysql] Filter databases that do not need to be read when discovering tables (#2160)
  18. [test] Fix TiDBE2eITCase by updating the container version to avoid tikv disk full failure (#2402)
  19. [test][postgres] Close postgres containers after tests and fix container reuse bug (#2394)
  20. [mysql] quota the field names when executing mysql query (#2381) (#2388)
  21. [test][mysql] Provide the server-time-zone setting in MySqlSourceExampleTest#testConsumingAllEvents (#2364)
  22. [hotfix][mysql] Catch underlying throwable in snapshot reading.
  23. [hotfix][mysql] Fix failed test LegacyMySqlSourceTest#testSnapshotOnFailedSource (#2384)
  24. [db2] [tests] Fix unstable Db2ConnectorITCase due to cleanup failure
  25. [mongodb][hotfix] Fix SourceRecord range check (#2343)
  26. [polish] Add type parameter for MySqlRecords(#2300)
  27. [doc] fix errors in PostgresParallelSourceExample (#2324)
  28. [docs][oracle] Add an example of Incremental Snapshot based DataStream for Oracle CDC Connector
  29. [mongodb] [tests] Loosen E2E timeout limitations
  30. [build] Add vcs.xml for IDEA (#2064)
  31. [hotfix] update dbz option name 'xx. whitelist' to 'xx.include.list'(xx = database, schema or table) (#2212)

Contributor List (sorted alphabetically)

  1. BIN
  2. FocusComputing
  3. GOODBOY008
  4. Hang Ruan
  5. Hongshun Wang
  6. Jiabao Sun
  7. Kunni
  8. Leonard Xu
  9. Malcolmjian
  10. North.Lin
  11. Paddy Gu
  12. Tyrantlucifer
  13. e-mhui
  14. gongzhongqiang
  15. wuzhenhua
  16. yuxiqian
  17. zhaomin

release-2.4.1

9 months ago

Release Notes

Documentation: https://ververica.github.io/flink-cdc-connectors/release-2.4

Features and Improvements

  1. [hotfix] Fix sqlserver monitor same table in other database (#2335)
  2. [debezium] Fix DebeizumSourceFunction can not do savepoint after close (#2259)
  3. [mysql] skip closing reader when the reader received the binlog split (#2261)
  4. [MySQL] Optimize how to construct config table.include.list (#2274)
  5. [oracle] Fix class conflict when using multi cdc sql connector
  6. [MySQL] Fix the startup mode exception message (#2264)
  7. [hotfix][mysql] remove unused code (#2231)
  8. [mysql] Fix assigning duplicate snapshot splits when enable scan newly added tables (#2326)
  9. [docs] Add connector features table to help users quickly understand the supported features of each connector
  10. [mysql] Remove the finished snapshot splits for the binlog split when restoring from the checkpoint (#2292) (#2318)
  11. [hotfix][sqlserver] Fix sqlserver close idle and chunk key column missing (#2310)
  12. [docs] Update Postgres CDC document for slot.name option (#2280)
  13. [postgres] Fix the slot name conflict bug (#2251)
  14. [docs] add docs for the mysql tables without primary keys (#2238)
  15. [oracle] Use oracle connection in context for each reader subtask (#2254)
  16. [hotfix] Update support version (#2256)

Contributor List (sorted alphabetically)

  1. Hang Ruan
  2. Leonard Xu
  3. SileiWu
  4. empcl
  5. gongzhongqiang
  6. skylines
  7. syyfffy

release-2.4.0

10 months ago

Release Notes

Documentation: https://ververica.github.io/flink-cdc-connectors/release-2.4

Features and Improvements

  1. [hotfix] Add vitess connector to the release profile
  2. [docs][hotfix] Update debezium reference links to 1.9 version
  3. [build] Update the copyright year to 2023 (#2205)
  4. [postgres] Fix postgres e2e test
  5. [postgres] scan.incremental.snapshot.enabled is closed by default
  6. [postgres] Backfill task will be able to end when there is not new change data but read the ending lsn
  7. [postgres] Create slot for backfill task before snapshot reading
  8. [postgres] Prepare a slot for the unique global stream split
  9. [mysql] Fix GTID issues to recover from checkpoint normally in specifying startup mode (#2220)
  10. [doc][hotfix] Fix driver version and correct vitess doc path (#2224)
  11. [vitess] Add e2e case for Vitess connector
  12. [vitess] Add Vitess tests in Azure pipeline
  13. [vitess] Add Vitess CDC connector (#456)
  14. [debezium] Bump debezium version to 1.9.7.Final (#2156)
  15. [oracle] Fix the backfill task not running bug in oracle cdc connector (#2218)
  16. [hotfix][sqlserver] Merge high and low watermark data during snapshot reading process (#2179)
  17. [Postgres] Use Incremental Snapshot Framework for Postgres CDC Connector (#2216)
  18. [oracle] Fix the incremental phase not skip the data that has been read in the snapshot full phase (#2215)
  19. [mongodb] Support specific timestamp startup mode. (#2207)
  20. [tests][mysql] Shutdown mysql container timely for legacy mysql tests
  21. [mongodb][hotfix] Fix regex inference of database or collection only contains dash. (#2209)
  22. [hotfix][build] Use different name for each job
  23. [hotfix][Mysql] MySqlBinlogSplitAssigner should support method isStreamSplitAssigned
  24. [MySQL] Remove deleted tables when using a new table filter and restarting from checkpoint
  25. [build][mysql][e2e] Increase the azure test job timeout to 90 minutes
  26. [mysql] Fix MySql CDC Connector outputs duplicated binlog when enable newly-added-table
  27. [mysql][minor] Print clear log information for key path
  28. [mongodb][hotfix] Fix drop or rename record cause documentKey being empty. (#2210)
  29. [ci][oceanbase] Only compile necessary modules for self hosted pipeline
  30. [doc] Add docs for close idle readers feature. (#2208)
  31. [cdc-base] Close idle readers when snapshot finished (#2202)
  32. [mongodb] Add support for mongodb+srv connection protocol. (#2203)
  33. [oceanbase] Always cleanup resources after reading before exit (#2189)
  34. [mongodb][hotfix] Fix NullPointerException when executing ddl operation in MongoDB (#2107)
  35. [mysql-cdc] Supports MYSQL_TYPE_TYPED_ARRAY column type when parsing the table map event
  36. [mysql-cdc][hotfix] Optimize the error msg when binlog expire in source
  37. [mysql-cdc] Optimize the error msg when binlog expire or server id conflict This closes #2010.
  38. [oceanbase] add jdbc options and support oracle mode (#1854)
  39. [mysql-cdc] Add handler for catching async exceptions in snapshot reading executor This closes #2016.
  40. [hotfix][mysql-cdc] Use session timezone instead of local time zone as the default server time zone
  41. [mysql] Fix NullPointerException when database name or table name contains dot
  42. [mysql] Fix NullPointerException caused by mysql ignores the capitalization when splitting chunks
  43. [sqlserver] Fix old change data that will be captured when the latest mode starts (#2176)
  44. [core] support print configuration options for connectors (#2099)
  45. [oceanbase] support libobcdc 4.x and fix restore timestamp config (#2161)
  46. [sqlserver] Sqlserver incremental source. (#1732)
  47. [OceanBase][test] Update oceanbase and oblogproxy docker image (#2060)
  48. [oceanbase] use real charset to convert change record (#2159)
  49. [mysql] Support tables which do not contain a primary key (#2150)
  50. [common] Bump flink to 1.17.0 (#2057)
  51. [doc] add mongodb cdc chinese doc (#2042)
  52. [hotfix][test] ignore OracleConnectorITCase.testConsumingAllEvents util issue 1875 fix (#2111)
  53. [docs][db2] Correct CHAR(n) to CHARACTER(n) (#2052)
  54. [postgres] Change config option slot.name to be required (#1996)
  55. [docs] Update the outdated DingTalk QR code
  56. [mongodb][hotfix] Fix pollAwaitTimeMillis does not take effect (#1995)
  57. [tests][mysql] Skip unnecessary IT Cases which uses legacy debezium source implementation
  58. [mysql] Add note for MySQL TINYINT(1) type to Flink SQL Boolean type
  59. [github] Minor typo in issue bug-report (#2020)
  60. [mysql-cdc] Fix race condition between split and coordinator thread that async exception might not be caught
  61. [mysql/oracle] Fix the wrong comments in ChunkSplitter
  62. [hotfix][docs] Correct the flink file path in MySQL docs
  63. [hotfix][mongodb] Fix resume token not found #1879 (#1938)
  64. [mongodb][hotfix] Fix authentication failed when using a non admin auth source and duplicated usernames #1934 (#1935)
  65. [mysql] Add various test cases to cover newly added table process from read binlog blocking to read binlog un-blocking
  66. [hotfix][base] Fix typo in java docs
  67. [mysql] Optimize the newly added table process from read binlog blocking to read binlog un-blocking
  68. [minor][mysql] Rename SplitsSize to SplitsNumber
  69. [hotfix][mysql] Correct meta group id as it is 0 based
  70. [hotfix][base][tidb][mysql] Fix wrong condition for execution timeout
  71. [mongodb] Improve code structure (#1900)
  72. [build] Use issue form templates (#1830)
  73. [ci] Fix for dubious repo ownership issue (#1843)
  74. [docs] Update mysql-cdc.md to recommend latest version (#1842)
  75. [ci] Fix for dubious repo ownership issue (#1840)
  76. [docs][sqlserver] Update data type mapping (#1802)
  77. [ci][tidb] Fix dns cache works incorrect in latest jdk 8. (#1809)
  78. [postgresql] Upgrade postgresql driver from 42.2.26 to 42.2.27 to avoid CVE-2022-41946 (#1773)
  79. [minor][common] Improve the changelogMode expression (#1793)
  80. [hotfix][debezium] Forward root cause in DebeziumSourceFunction (#1791)
  81. [docs][oracle] Add new introduced config options from incremental snapshot framework (#1719)
  82. [mysql] Fix backward compatibility on deserializing binlog offset from old versions (#1758)
  83. [hotfix][docs] Correct the startup option's data type in Mysql docs and duplicate batch.size in MongoDB docs (#1751)
  84. [hotfix][docs] Correct the mysql driver version in MySQL docs
  85. [docs][site] update the supported cdc connectors picture
  86. [oracle] Add missing supported config options for oracle table source (#1727)

Contributor List (sorted alphabetically)

  1. Hang Ruan
  2. He Wang
  3. JasonLee
  4. Jiabao Sun
  5. Jingsong Lee
  6. Xiao Meng
  7. Leonard Xu
  8. Qingsheng Ren
  9. Robert Metzger
  10. Sergey Nuyanzin
  11. Simonas Gelazevicius
  12. Yaroslav Tkachenko
  13. Tigran Manasyan
  14. Tyrantlucifer
  15. ZhongLinLeo
  16. bb chen
  17. ehui
  18. emhui
  19. gintarasm
  20. gongzhongqiang
  21. liangyaohui97
  22. lin.zl
  23. molsionmo
  24. skylines
  25. tison
  26. wallkop
  27. wangxiaojing
  28. wuzhenhua
  29. xixingya
  30. zhaomin
  31. zhuyuan03
  32. 含风

release-2.3.0

1 year ago

Release Notes

Documentation: https://ververica.github.io/flink-cdc-connectors/release-2.3

Features and Improvements

  1. [mysql] Improve error message under the case that start reading from earliest but schema change happened before (#1724)
  2. [mysql] Use event filter to drop events earlier than the specified timestamp in timestamp startup mode (#1724)
  3. [docs][oracle] Add new introduced config options from incremental snapshot framework (#1718)
  4. [docs][oceanbase][zh] Update outdated sections and translate them to Chinese
  5. [hotfix][docs] Add link for Oceanbase ZH docs
  6. [docs][oceanbase] Add Chinese documents for oceanbase-cdc connector(#1611)
  7. [db2] Fix stream api example in db2 doc. (#1725)
  8. [oracle] Add missing supported config options for oracle table source (#1720)
  9. [mysql] Fix MySqlBinlogSplitAssigner may reassign split after exception restart (#1713)
  10. [docs] Repair Connector Data Type links
  11. [docs] Update the connector download link in the document
  12. [docs] Update the flink cdc picture with supported database vendors.
  13. [tidb] Fix unstable TiDB region changed test. (#1702)
  14. [docs][mongodb] Add docs for MongoDB incremental source
  15. [oracle][mysql] Improve the Oracle all data types test and clean up debug logs
  16. [oracle] Properly support TIMESTAMP_LTZ type for oracle cdc connector
  17. [oracle]support use user given column as split column
  18. [oracle][minor] Clean up useless methods and classes
  19. [oracle] Parallel read supports startup of startupOptions
  20. [oracle] Add full type field test
  21. [oracle]OracleTableSource support splitSize, fetchSize ,connectMaxRetries, connectionPoolSize and add test
  22. [hotfix][oracle][db2] Update the license header
  23. [cdc-base][oralce] Remove useless method of JdbcSourceFetchTaskContext
  24. [cdc-base][oracle] Improve the base methods base on Oracle specific requirements
  25. [oracle] Support custom url for incremental snapshot source
  26. [oracle] Introduce SCAN_INCREMENTAL_SNAPSHOT_ENABLED option for OracleTableSourceFactory
  27. [minor][oracle] Improve tests and code format
  28. [oracle] Ignore testCheckpointAndRestore util DBZ-5245 and DBZ-4936 fix
  29. [oracle][test] Add tests for Oracle incremental snapshot source
  30. [oracle] Use ROWID as split column and replace limit syntax with ROWNUM
  31. [oracle] Let LogMinerStreamingChangeEventSource do not rely on oracle table LOG_MINING_FLUSH
  32. [oracle] Use Incremental Snapshot Framework for Oracle CDC Connector (#1079)
  33. [docs] Bump Flink version to 1.16.0
  34. [common] Bump Flink version to 1.16.0
  35. [docs][db2] Add db2 to README.md (#1699)
  36. [tidb] Checkpoint is not updated long after a task has been running (#1686)
  37. [hotfix] Add method getMaxResolvedTs back to class CDCClient. (#1695)
  38. [docs] Bump connector version to flink 1.15.2 in docs (#1684)
  39. [tidb] Fix data lost when region changed (#1632)
  40. [hotfix][docs] Correct reference link for DB2 docs (#1683)
  41. [mysql] Update docs of specifying starting offset feature of MySQL CDC source
  42. [hotfix][mysql] Remove unused constructor in MySqlTableSource
  43. [mysql] Support starting reading binlog from earliest offset / specific offset / timestamp (#1661)
  44. [mongodb] Support incremental snapshot for MongoDB CDC Connector
  45. [cdc-base] Extract a higher level IncrementalSource independent of JDBC dialects
  46. [hotfix][docs] Fix db2 doc download link (#1681)
  47. [docs][db2] Add Db2 CDC connector doc link
  48. [docs][db2] Add Db2 cdc connector docs
  49. [build][db2] Add Db2 pipeline script
  50. [e2e][db2] Add Db2 e2e test
  51. [db2] Add sql connector for Db2 CDC (#1626)
  52. [db2] Introduce Db2 cdc connector (#450)
  53. [tidb] Fix data loss in tidb-cdc when switch from snapshot read to change event read (#1207)
  54. [hotfix][cdc-base] Fix OffsetFactory is not initialized when restored from checkpoint (#1657)
  55. [common] Bump Flink version to 1.15.2 (#1504)
  56. [cdc-base] Abstract naming for variables of incremental framework (#1646)
  57. [mysql] ChunkSplitter supports to snapshot when splitting chunks for a table (#1463)
  58. [cdc-base] Fix StreamSplit's endingOffset of StreamSplitAssigner (#1659)
  59. [cdc-base] JdbcSourceConfig can be template type of DataSourceDialect (#1639)
  60. [mysql] Use the latest received meta of binlog split when request the next meta (#1479)
  61. [minor] Use Collections.singletonList instead of Arrays.asList if possible (#1569)
  62. [oceanbase] Add new deserialization schema with runtime converter (#1356)
  63. [mysql] Avoid duplicate split requests when add new table (#1156)
  64. [postgres] Upgrade org.postgresql:postgresql from 42.2.14 to 42.2.26 for CVE-2022-31197 (#1616)
  65. [test][e2e] Upgrade org.postgresql:postgresql from 42.3.1 to 42.2.26 for CVE-2022-31197(#1616)
  66. [hotfix][typo] Fix typo from schemaLess to schemaless
  67. [cdc-base] Reduce the memory usage of JM by sharing table schemas between splits
  68. [cdc-base] Release the scan fetcher thread resource after reading finished (#1619)
  69. [cdc-base] Optimize pure binlog phase check logic to improve performance (#1620)
  70. [cdc-base] Let all records of one snapshot split don't cross checkpoints (#1622)
  71. [postgres] Introduce changelog mode configuration with upsert/all modes
  72. [common] Introduce changelog mode configuration with upsert/all modes
  73. [hotfix][docs] Fix docs for scan.incremental.snapshot.chunk.size (#1617)
  74. [cdc-base] Adopt streaming merge during snapshotSplit scan to avoid OOM
  75. [docs][oceanbase][zh] Add video tutorials that synchronizes data from OceanBase to Elasticsearch (#1604)
  76. [docs][oceanbase] Add video tutorials that synchronizes data from OceanBase to Elasticsearch (#1605)
  77. [mysql] Fix parse error when ALTER SQL statement contains default string value and collate. (#1544)
  78. [docs][mysql] Add regex examples for multiple tables (#1564)
  79. [oceanbase] Shade apache commons dependency to avoid class conflict (#1532)
  80. [postgres] add geometry type support. (#1554)
  81. [build] Polish CI scripts and remove useless code (#1540)
  82. [mysql] fix parsing ddl error when use string default values for numeric types (#1522)
  83. [mysql] Enumerate captured tables lazily (#1507)
  84. [mysql] Reduce the memory usage of JM by sharing table schemas between splits (#1523)
  85. [hotfix][docs] Fix comment typo in MysqlSnapshotSplit (#1513)
  86. [mysql] Let all records of one snapshot split don't cross checkpoints (#1505)
  87. [tests] Remove usage of deprecated EnvironmentSettings.Builder#useBlinkPlanner (#1493)
  88. [cdc-base][mysql] Fix split key comparison when may contain different types (#1499)
  89. [docs][mysql] Update the description of mysql server-time-zone option (#1497)
  90. [mysql] Use local timezone as the default value of 'server-time-zone' option (#1407)
  91. [docs][postgres] Add two frequently debezium options in Postgres connector document (#1142)
  92. [mongodb] Allow mongo ARRAY to be converted to string type in Flink (#1475)
  93. [hotfix][docs] Fix the page links in MySQL Chinese document (#1466)
  94. [oceanbase][test] Use memory mode of obcdc and double startup timeout duration (#1474)
  95. [mysql][polardbx] Stop container timely in tests (#1468)
  96. [mysql][ploardbx] Support all charsets in column level (#1166)
  97. [mongodb] Expose updateLookup option for stream source. (#1461)
  98. [oceanbase] Use utc offset as 'server-time-zone' option's default value (#1443)
  99. [mysql][minor] Skip capture newly added tables according snapshotassigner's status (#1449)
  100. [polardbx][test] Bump the polardb-x image version from 2.0.1 to 2.10 (#1423)
  101. [build][test] Specify surefire provider to surefire-junit47 (#1423)
  102. [hotfix][docs] Fix typo in bug report doc (#1460)
  103. [minor][oracle] Rename JdbcConfigurationUtil to OracleJdbcUrlUtils
  104. [oracle] Support custom JDBC URL for Oracle (#1258)
  105. [docs] Correct the data type of 'server-id' option to String (#1444)
  106. [hotfix][test] Fix MySqlSourceITCase failure due to StreamExecutionEnvironment#getConfiguration returned unmodifiable object (#1446)
  107. [test] Fix tests missed 'update_before' records if the primary keys of source and sink are different (#1451)
  108. [hotfix][oracle] Fix the shell script interpreter assignment should be at first line (#1452)
  109. [hotfix][tidb-cdc] TiDB CDC dealUnsignedColumnValue throw NullPointerException. (#1448)
  110. [polardbx][test] Rename PolardbxSourceTCase to PolardbxSourceITCase (#1447)
  111. [mysql] Enable pass custom jdbc properties for debezium mysql connection (#1217)
  112. [mysql] Release the debezium reader thread resources after reading finished (#1358)
  113. [oceanbase] Introduce 'table-list' option to support capture list of tables (#1369)
  114. [cdc-base] Flink CDC base registers the identical history engine on multiple tasks (#1340)
  115. [hotfix][mysql] Fix compile error due to merge conflict
  116. [mysql] Generates multiple chunks when approximate row count is bigger than chunk size (#1193)
  117. [cdc-base] Fix NPE during snpashot scan phase (#1339)
  118. [docs] Improve MySQL ZH document and update the outdated links (#1435)
  119. [docs] Translate mysql cdc doc to Chinese (#1391)
  120. [mysql] Introduce 'scan.incremental.snapshot.chunk.key-column' option to support splitting table by custom column (#1415)
  121. [mysql] Optimize pure binlog phase check logic to improve performance (#1392)
  122. [oracle] Fix mining session stopped due to 'No more data to read from socket'
  123. [common] Ignore retriableException to make the debezium engine retry works
  124. [test][oracle] Fix unstable tests OracleE2eITCase due to reuse container
  125. [tidb] Fix NPE when the specified table does not exist (#1411)
  126. [docs] Add step up SQLServer Database cdc document in sqlserver-cdc.md.
  127. [mongodb] Expose batch.size config option for table source. (#1387)
  128. [mongodb] Example tests for MongoDBSource (#1388)
  129. [build] Use correct year and copyright owner in license files (#1390)
  130. [mongodb] Allow document field defined as string type in flink sql (#1385)
  131. [mysql] Fix dead loop when splitting chunks. (#1107)
  132. [docs] Fix a typo in sqlserver-cdc.md (#1382)
  133. [docs] Fix typo in mysql-cdc.md (#1364)
  134. [docs] Fix a typo in tidb-cdc.md (#1215)
  135. [docs] Add all supported type mapping for mysql cdc connector (#1373)
  136. [build] Build document site only if current PR modified files under docs (#1230)
  137. [oracle] Support XMLType for Oracle CDC Connector (#1342)
  138. [build] Bump Flink version to 1.14.4 (#1284)
  139. [mysql] Adopt streaming merge during snapshotSplit scan to avoid OOM (#1219)
  140. [oceanbase] Introduce more config options of log proxy and improve the oceanbase connector(#1234)
  141. [docs][cdc-base] Optimize annotations (#1320)
  142. [mysql] Connection pool id should contains username (#1328)
  143. [tidb-cdc] Monitor tidb table lose data, which table has one more regions. (#1196)
  144. [dbz] Patch for DBZ-5126 to fix memory leak problem. (#939)
  145. [hotfix][docs] Fix typo in java docs. (#1305)
  146. [mongodb] Permission optimization of single database is specified (#1240)
  147. [hotfix][cdc-base] Fix typo 'currenFetcher' -> 'currentFetcher' (#1299)
  148. [hotfix][oceanbase] Fix the default string encoding(#1294)
  149. [mongodb][doc] Fix meta field collection_name of metadata sql example (#1290)
  150. [mongodb] Fix UnsupportedTemporalTypeException when define Timestamp as String (#1268)
  151. [mysql] Support optional numeric unsigned zerofill types (#1197)
  152. [mysql] Add heartbeatInterval field to equals() and hashCode() (#1116)
  153. [mysql] Catch and hold exception in chunk splitter thread and rethrow in getNext() (#1109)
  154. [mysql] Use the gtid set firstly when compare the binlog offset (#761)
  155. [hotfix][tidb] Fix RowKind.INSERT judgement condition (#1073)
  156. [oracle] Fix duplicated records when restoration (#1100)
  157. [mysql] Throw RuntimeException timely in snapshot scan phase (#1098)
  158. [docs][mongodb] Update document about MongoDB database and collection regex filter (#1089)
  159. [common] Apply filter on ResolvedSchema to keep physical columns only (#1092)
  160. [oracle][test] Use prebuilt docker image in tests
  161. [common] Bump debezium version to 1.6.4.Final
  162. [cdc-base] Improve Incremental snapshot framework code style. (#1054)
  163. [tidb] Test TiDB 6.0.0 and add it to supported list (#1057)
  164. [mysql] Serialize isSuspended flag for MySqlBinlogSplit (#996)
  165. [docs][mysql] Fix connector document link error (#1043)
  166. [sqlserver] Add more supported(tested) SqlServer versions (#1050)
  167. [docs] Update the supported sink vendors
  168. [oceanbase] Remove useless dependency and code of test. (#1018)
  169. [docs][mysql] Add note to make the tableList usage and dependencies clearer (#1019)
  170. [hotfix][docs] Fix misspell of docs title
  171. [docs] Add example for Scan Newly Added tables feature (#1016)
  172. [hotfix][docs] Fix jar link for PolarDB-X doc. (#1012)

Download

Contributor List (sorted alphabetically)

  1. 01410172
  2. Amber Moe
  3. Dezhi Cai
  4. Enoch
  5. Hang Ruan
  6. He Wang
  7. JiaJia
  8. Jiabao Sun
  9. Junwang Zhao
  10. Kyle Dong
  11. Leonard Xu
  12. Matrix42
  13. Paul Lin
  14. Qingsheng Ren
  15. Qishang Zhong
  16. Rinka
  17. Sergey Nuyanzin
  18. Tigran Manasyan
  19. camelus
  20. dujie
  21. ehui
  22. empcl
  23. fbad
  24. gongzhongqiang
  25. hehuiyuan
  26. hele.kc
  27. hsldymq
  28. jiabao.sun
  29. legendtkl
  30. leixin
  31. leozlliang
  32. lidoudou1993
  33. lincoln lee
  34. lxxawfl
  35. lzshlzsh
  36. molsion
  37. molsionmo
  38. pacino
  39. rookiegao
  40. skylines
  41. sunny
  42. vanliu
  43. wangminchao
  44. wangxiaojing
  45. xieyi888
  46. yurunchuan
  47. zhmin
  48. 阿洋
  49. 莫贤彬

release-2.2.1

2 years ago

Release Notes

Improvements and Bug fixes

  1. [docs] Update the dependencies version used in docs to 2.2.0
  2. [hotfix][docs] Fix jar link for PolarDB-X docs (#1013)
  3. [docs] Add example for Scan Newly Added tables feature (#1016)
  4. [hotfix][docs] Fix misspell of docs title
  5. [docs][mysql] Add note to make the tableList usage and dependencies clearer (#1037)
  6. [docs] Update the supported sink vendors
  7. [sqlserver] Add more supported(tested) SqlServer versions (#1049)
  8. [oceanbase] Remove useless dependency and code of test (#1046)
  9. [docs][mysql] Fix connector document link error (#1043)
  10. [mysql] Serialize isSuspended flag for MySqlBinlogSplit (#1055)
  11. [common] Apply filter on ResolvedSchema to keep physical columns only (#1097)
  12. [docs][mongodb] Update document about MongoDB database and collection regex filter (#1090)
  13. [mysql] Throw RuntimeException timely in snapshot scan phase (#1099)
  14. [hotfix][tidb] Fix RowKind.INSERT judgement condition (#1104)
  15. [mysql] Catch and hold exception in chunk splitter thread and rethrow in getNext() (#1110)
  16. [mysql] Use the gtid set firstly when compare the binlog offset (#1111)
  17. [mysql] Add heartbeatInterval field to equals() and hashCode() (#1118)

Download

Documentation

release-2.2.0

2 years ago

Release Notes

Documentation: https://ververica.github.io/flink-cdc-connectors/release-2.2/

Features and Improvements

  1. [docs] Update the supported database vendors and improve the notes.
  2. [docs][mysql][polardbx] Add polardbx tutorials (#1003)
  3. [docs][mysql] Remove redundant docker environment variable in tutorials (#1005)
  4. [cdc-base] Add MySQL Incremental Source to validate the Incremental Snapshot Framework(#776)
  5. [cdc-base] Improve the Incremental Snapshot Interfaces
  6. [cdc-base] The first version of Incremental Snapshot Framework abstraction
  7. [hotfix][docs] Fix markdown title due to typo in readme
  8. [oceanbase] Bump the oblogclient version to 1.0.3 to fix the NPE in oblogclient connection (#998)
  9. [docs] Clarify ownership to comply with ASF trademark policies (#1000)
  10. [build] Decrease the azure job timeout to reasonable 40 minutes (#991)
  11. [hotfix][site] Use Flink with its trademark
  12. [docs] Adopt the full name 'Apache Flink' with its trademark notice (#988)
  13. [hotfix][docs] Update the copyright
  14. [docs][mysql] Correct the jdbc driver version in documentation
  15. [docs] Organize the connector docs in alphabetical order
  16. [docs][tidb] Add tidb cdc connector document
  17. [hotfix][docs] Fix typo in tidb-tutorial.md and tidb-tutorial-zh.md
  18. [mysql] Suspend SourceReader eagerly to reduce restoration time
  19. [docs][oceanbase] Add quickstart tutorials for OceanBase CDC connector
  20. [docs][oceanbase] Add docs for OceanBase CDC connector
  21. [build][oceanbase] Use self-hosted agent for oceanbase cdc tests
  22. [oceanbase] Add OceanBase CDC connector
  23. [tidb] remove useless code.
  24. [test][e2e] Use single fork in tests
  25. [test][e2e] Clean up database resources for every e2e tests.
  26. [e2e] bump flink e2e test version.
  27. [tidb] Add tidb cdc connector e2e test.
  28. [tidb] Add flink1.14 support.
  29. [hotfix][mysql] Avoid schema isn't known error when scanNewlyAddedTable enabled (#965)
  30. [mysql][polardbx] Remove debug print code in PolardbxSourceTCase (#968)
  31. [tidb] Introduce runtime converter to improve the deserialization
  32. [tidb][test] Add timezone tests for tidb
  33. [tidb] Remove useless config options
  34. [docs][site] Add favicon fot website index html
  35. [docs][site] Introduce static index html for the project website (#961)
  36. [mysql] Split snapshot chunks asynchronously (#931)
  37. [docs][tidb] Add index for tidb tutorial docs
  38. [tidb][docs] Add tidb connector tutorial (#938)
  39. [tidb] Add metadata column ITCase and improve the code style
  40. [tidb][test] Fix tikv port conflicts
  41. [tidb] Support metadata column
  42. [tidb][test] Add tidb pipeline in azure CI
  43. [tidb][test] Add TiDBTableSourceFactoryTest and TiDBConnectorITCase
  44. [tidb] Fix tikv.pd.addresses not work and use ResolvedSchema instead of TableSchema.
  45. [tidb] Bump tikv client version to v3.2.0
  46. [tidb] The first draft of tidb cdc connector
  47. [docs] List supported databases in docs (#954)
  48. [mysql] Fallback to build schema by 'desc table' case when parsing DDL failure (#949)
  49. [docs] Fix download url in mongodb tutorial (#951)
  50. [docs] Update DingTalk QR code to English version (#950)
  51. [mongodb] Match multiple database and collection names using a regular expression in MongoDB. (#940)
  52. [mysql] Allow custom JDBC URL parameters in MySQL CDC source (#921)
  53. [mongodb][test] Fix unstable tests of MongoDBConnectorITCase (#944)
  54. [mysql] Improve mysql heartbeat feature and tests
  55. [mysql] Original implementation of mysql heartbeat feature
  56. [common] Update issue templates
  57. [hotfix][mysql] Fix MySqlSourceReader#snapshotState should return unfinishedSplits (#926)
  58. [docs] Fix the outdated link of flink download link (#922)
  59. [hotfix][mysql] Fix the binlog position didn't update when emit DDL (#925)
  60. [mongodb] Correct the TypeInformation argument of DebeziumDeserializationSchema in SourceFuntion (#780)
  61. [common] Allow user to pass custom JsonConverter configs in JsonDebeziumDeserializationSchema (#615)
  62. [mysql] Improve the test to cover table with varbinary primary key (#879)
  63. [mysql] Support primary key with varbinary type (#879)
  64. [format][changelog-json] Remove the support of changelog-json format (#912)
  65. [hotfix][minor] Fix the test code typo
  66. [mysql] Fix ArrayIndexOutOfBoundsException in mysql binlog read phase (#911)
  67. [mongodb][tests] Track MongoDBITCase hang in azure (#881)
  68. [docs] Update the supported flink version mapping (#885)
  69. [common] Fix emitDelay and fetchDelay metrics to be 0L during snapshot phase (#878)
  70. [hotfix][docs][mongodb] Update supported mongodb versions in README.md
  71. [docs][sql-server] Add sqlserver-cdc connector tutorial (#687) This closes #687.
  72. [common] Use ResolvedSchema instead of deprecated TableSchema API (#782)
  73. [docs][sqlserver] Add SqlServer CDC connector document (#705)
  74. [doc][sqlserver] Add SqlServer English tutorial document. (#693)
  75. [common] Make Flink CDC Compatible with Flink 1.14
  76. [build] Bump flink version to 1.13.5
  77. [docs][oracle] Add guidance about oracle CDB database (#792)
  78. [docs] Add connection.pool.size and connect.max-retries option in mysql document (#857)
  79. [sqlserver] Add e2e tests for sqlserver cdc connector (#837)
  80. [hotfix][mysql] Rename method from dispatchHighWatermark to dispatchBinlogEndEvent (#810) This closes #810.
  81. [mysql] Use READ type for all records in snapshot reading (#842)
  82. [mysql] Rewrite the newly added table tests to cover more cases (#777) This closes #777.
  83. [mysql] Improve newly added tables process
  84. [mysql] Support capture newly added tables for existed pipeline
  85. [docs] Fix typo in mongodb-cdc.md (#836)
  86. [mysql] add test to cover the datatype GeomCollection that's synonymous with GeometryCollection for mysql8 (#822)
  87. [sqlserver] Add SqlServer cdc connector (#662)
  88. [build] Remove redundant maven enforce plugin (#794)
  89. [hotfix] Fix MySqlSource comment typo (#793)
  90. [mysql] Support MySQL 5.6 (#396)
  91. [build] Switch to Log4j 2 by default (#791)
  92. [oracle] Fix Oracle numeric types conversion (#560)
  93. [hotfix][mysql] Fix message timestamp of snapshot record to 0L (#752)
  94. [docs] Translate the Chinese FAQ document to English (#737)
  95. [docs] Remove the fixed version of website (#730)
  96. [hotfix][mysql] Set minimum connection pool size to 1 (#692)
  97. [build] Bump log4j2 version to 2.16.0 (#714)
  98. [build] Remove override definition of maven-surefire-plugin in connectors pom (#673)
  99. [docs] Add FAQ for tableList option usage in DataStream API (#634)
  100. [mongodb] Support to connect MongoDB without authentication (#642)
  101. [hotfix] Fix the parameter typo in java doc (#651)
  102. [mysql] Add a method to get MySqlSourceConfigFactory from MySqlSource (#648)
  103. [mysql] Set default driver class name for PooledDataSource (#659)
  104. [mysql] Fix there is duplicate records outputted when keep updating table during snapshot read (#592)
  105. [docs] Improve column name note in oracle cdc document. (#625)
  106. [docs] Add Flink cdc eco-system picture (#623)
  107. [hotfix][docs] Fix typo in oracle-cdc.md (#613)
  108. [docs] Add supported Flink versions for Flink CDC 2.1(#611)
  109. [build] Bump version to 2.2-SNAPSHOT

Download

Contributor List (sorted alphabetically)

  1. Dian Qi
  2. He Wang
  3. Jark Wu
  4. Jiabao Sun
  5. Jingsong Lee
  6. Junwang Zhao
  7. Konstantin Knauf
  8. Leonard Xu
  9. Mustang Kong
  10. Paul Zhang
  11. Qingsheng Ren
  12. Sergey Nuyanzin
  13. Shengkai
  14. Tuple
  15. Xianxun Ye
  16. Xuebing Li
  17. Zongwen Li
  18. camelusluo
  19. eastfisher
  20. ehui
  21. fangpc
  22. gongzhongqiang
  23. hele.kc
  24. hiscat
  25. luoyuxia
  26. paul8263
  27. skylines
  28. szgeqin
  29. taox
  30. tsreaper
  31. wangxiaojing
  32. wysstartgo
  33. yexianxun
  34. 元组

release-2.1.1

2 years ago

Release Notes

Improvements and Bug fixes

  1. [docs] Remove the fixed version of website (#730)
  2. [hotfix][mysql] Set minimum connection pool size to 1 (#692)
  3. [build] Bump log4j2 version to 2.16.0 (#714) Note: This project only uses log4j2 in test code and won't be influenced by log4shell vulnerability
  4. [build] Remove override definition of maven-surefire-plugin in connectors pom (#673)
  5. [docs] Add FAQ for tableList option usage in DataStream API (#634)
  6. [mongodb] Support to connect MongoDB without authentication (#642)
  7. [hotfix] Fix the parameter typo in java doc (#651)
  8. [mysql] Set default driver class name for PooledDataSource (#659) (#661)
  9. [mysql] Fix there is duplicate records outputted when keep updating table during snapshot read (#592)
  10. [docs] Improve column name note in oracle cdc document (#633)
  11. [docs] Add Flink cdc eco-system picture (#624)
  12. [hotfix][docs] Fix typo in oracle-cdc.md(#616)
  13. [docs] Add supported Flink versions for Flink CDC 2.1(#608)

Download

Documentation

release-2.1.0

2 years ago

Release Notes

Documentation: https://ververica.github.io/flink-cdc-connectors/release-2.1/

Features and Improvements

  1. [tests] Add end-to-end tests for cdc connector uber jars (#594)
  2. [tests] Upgrade testcontainers dependency to 1.16.2 (#591)
  3. [mysql] Reading a null column with default value defined should return null instead of default value (#555)
  4. [mysql] Optimize the dynamic chunk size calculation for table with dense distribution (#599)
  5. [mongodb] Fix MongoDBConnectorITCase is timeout on Azure (#600)
  6. [hotfix][mysql] Fix number overflow in ObjectUtils.minus (#598)
  7. [mongodb] Fix problem of cannot look up fields on non-struct type (#588)
  8. [hotfix][mongodb] Fix NullPointException when heartbeat option is enabled (#586)
  9. [postgres] Support 'schema-name' metadata column for postgres-cdc connector
  10. [oracle] Support metadata columns for oracle connector
  11. [tests] Split the CI pipeline into multiple stages for better performance (#572)
  12. [hotfix][mongodb] Correct the connection uri building of MongoDB (#568)
  13. [mysql] Skip ddl events in snapshot split normalization (#577)
  14. [mysql] Support dynamic chunk size for evenly distribution table
  15. [mysql][minor] Use 'table.include.list' key instead of legacy key 'table.whitelist' in SnapshotSplitReader
  16. [mysql] Set server time zone information for connection pool factory (#566)
  17. [mysql] Polish binlog offset compare semantic (#543)
  18. [build] Update NOTICE file for sql connector jar (#565)
  19. [postgres] Support metadata columns for postgres-cdc connector (#500)
  20. [debezium] Bump up debezium version to 1.5.4.Final (#554)
  21. [oracle] Shade dependencies for oracle-cdc connector (#519)
  22. [hotfix][docs] Fix the display of github star button
  23. [common] Fix NPE of JsonConverter when job run on cluster(#470)
  24. [mongodb] Support metadata columns for mongodb-cdc connector (#476)
  25. [mysql] Reduce the connection usage in table discovery process
  26. [build] bump maven shade plugin version to 3.2.4
  27. [mysql] Introduce Jdbc connection pools
  28. [mysql] support all mysql data types (#438)
  29. [hotfix][docs] Correct the Kibana link in Oracle connector documentation (#539)
  30. [mysql] Support binlog split with huge snapshot split meta
  31. [mysql] Support tables with different split key
  32. [mysql] Validate the regex pattern of database-name and table-name in table factory (#526)
  33. [build] Add necessary shaded dependencies (#512)
  34. [oracle][test] Fix oracle tests failure in azure due to ORA-01882
  35. [oracle][minor] Fix checkstyle and typo
  36. [oracle] Add docker build 'on-the-fly' and refactor test image
  37. [oracle] Added validator for oracle
  38. [oracle] Updated package naming to com.ververica in oracle connector
  39. [oracle] Fixing unit tests, cleaning up after adding oracle cdc
  40. [oracle] Add oracle cdc connector
  41. [build] Upgrade azure environment from ubuntu-16.04 to ubuntu-18.04 (#517)
  42. [hotfix][mysql] Correct the usage of includeSchemaChanges parameter (#501)
  43. [build] Follow Flink builtin connector shade style (#49)
  44. [mysql] Expose MySqlSource API that supports incremental snapshot (#495)
  45. [mysql] Support metadata columns for mysql-cdc connector (#496)
  46. [common] Support to read tables of inconsistent schema (#494)
  47. [build] Fix maven connection timeout issue in azure CI
  48. [mysql] Support wide table with huge number of columns
  49. [tests] Correct the argument order of assertArrayEqualInOrder/assertArrayEqualAnyOrder
  50. [tests] Introduce assertEqualsInOrder/assertEqualsInAnyOrder methods for test
  51. [tests] Improve the unstable test testStartupFromLatestOffset in MySqlConnectorITCase
  52. [mysql] Fix the restoration failure when checkpoint happened in one MySQL transaction
  53. [mysql] Optimization that skips the binlog read if the low watermark equals high watermark
  54. [mysql] Add source metrics for MySqlParallelSource
  55. [build] Use tools with stable version in building docs
  56. [mysql] Use min/max of split key and row count to describe the distribution of table
  57. [mysql] Fix the ClassCastException in RecordUtils
  58. [mongodb][docs] Add documentation for mongodb-cdc connector (#395)
  59. [mongodb] Remove no need config option prefix 'debezium.' (#420)
  60. [mongodb] Support MongoDB CDC connector (#225)
  61. [common] Shade kafka-related dependency (#386)
  62. [common] Support json deserialization schema
  63. [mysql] Improve the tests code
  64. [mysql] Fix the MySqlSnapshotSplitAssigner may assign duplicate splits
  65. [mysql] Add tests to check mysql-cdc works well under various timezones (#367)
  66. [mysql] Optimize the checkpoint be optional under single parallelism (#365)
  67. [postgres] Fix postgres-cdc connector cannot recognize the optional option 'slot.name'
  68. [mysql] Correct the initial binlog offset for MySqlHybridSplitAssigner
  69. [mysql] Add default value for 'database.history.instance.name' in MySqlParallelSource
  70. [common] Remove useless parameter 'converter.schemas.enable'
  71. [build] Run integration tests for each building
  72. [mysql] Improve the code format in SignalEventDispatcher
  73. [postgres] Fix Validator didn't implement Serializable
  74. [changelog] fix changelog-deserialize exception message typo.
  75. [build] Update issue templates
  76. [docs] Add tutorial about building real-time data lake for MySQL sharding table with Flink CDC (#551)
  77. [docs] Add metadata column example for mysql-cdc and mongodb-cdc connector
  78. [docs] Add checkpoint configuration limitation for Oracle and Postgres connector (#580)
  79. [docs] Fix the link of 'edit on github' on site (#578)
  80. [docs] Use SNAPSHOT version dependency in master documentation (#497)
  81. [docs] Use more generic term 'transaction logs' in README
  82. [docs] Add MySQL HA cluster support explanation
  83. [docs] Update the dependency version to 2.1.0
  84. [docs] Add checkpoint configuration limitation for Oracle and Postgres connector
  85. [docs] Fix the link of 'edit on github' on site (#578)
  86. [docs][oracle] Fix oracle DATE type mapping in documentation (#569)
  87. [docs] Translate "Streaming ETL for MySQL and Postgres with Flink CDC" into Chinese (#550)
  88. [docs] Add GitHub star button to documentation (#547)
  89. [docs] Add Oracle support in README
  90. [docs] Restructure the "tutorials" to "quickstart"
  91. [docs] Add documentation for oracle-cdc (#515)
  92. [docs] Rewrite tutorial doc for mysql-postgres-tutorial (#514)
  93. [docs] Add favicon for docs website (#525)
  94. [docs] Use new MySqlSource API in documentation
  95. [docs] Correct DataStream API usage in document examples
  96. [docs] Use JsonDeserializationSchema as output format in examples
  97. [docs] Update documentation of tutorial
  98. [docs] Add meaningful title for tutorial
  99. [docs] Support build document website locally
  100. [docs] Fix the broken link in readme
  101. [docs] Update the debezium document link to version 1.5
  102. [docs][mysql] Fix LowerCase Class name typo
  103. [docs] Add checkpoint config in document example
  104. [docs] Add FAQ for MySQL 8.0 Public Key Retrieval setting
  105. [docs] Add checkpoint and primary key setting for example in tutorials
  106. [docs] Fix the imports in document demo
  107. [docs] Set the content width of the website
  108. [docs] Update DingTalk group picture size in README.md
  109. [docs] Update the supported connector version and link

Download

Contributor list

Gunnar Morling, Jark Wu, Jiabao Sun, Leonard Xu, MartijnVisser, Marton Balassi, Shengkai, Tamas Kiss, Tamas Zseder, Zongwen Li, dongdongking008, frey66, gongzhongqiang, ili zh, jpatel, lsy, luoyuxia, manmao, mincwang, taox, tuple, wangminchao, yangbin09