2.3.5 Release Note

Bug fix

Core

• [fix] fix get seatunnel row size npe issue (#6681)
• [Hotfix] Fix DEFAULT TABLE problem (#6352)
• [Chore] Fix file spell errors (#6606)
• [BugFix][Spark-translation] map type cast error (#6552)
• [Hotfix] Fix spark example (#6486)
• [Hotfix] Fix compile error (#6463)

Transformer

• [Fix][SQLTransform] fix the scale loss for the sql transform (#6553)
• [Bug] Fix minus constant error in SQLTransform (#6533)

Connectors

• [Fix][Kafka-Sink] fix kafka sink factory option rule (#6657)
• [Hotfix] fix http source can not read yyyy-MM-dd HH:mm:ss format bug & Improve DateTime Utils (#6601)
• [Bug] Fix OrcWriteStrategy/ParquetWriteStrategy doesn't login with kerberos (#6472)
• [Fix][Doc] Fix FTP sink config key username to user (#6627)
• [E2E] Fix AmazondynamodbIT unstable (#6640)
• [Fix][Connector-V2] Fix add hive partition error when partition already existed (#6577)
• [Fix][Connector-V2] Fixed doris/starrocks create table sql parse error (#6580)
• [Fix][Connector-V2] Fix doris sink can not be closed when stream load not read any data (#6570)
• [Fix][Connector-V2] Fix connector support SPI but without no args constructor (#6551)
• [Fix][Connector-V2] Fix doris source select fields loss primary key information (#6339)
• [Fix][FakeSource] fix random from template not include the latest value issue (#6438)
• [Fix][Connector-V2] Fix mongodb cdc start up mode option values not right (#6338)
• [BugFix][Connector-file-sftp] Fix SFTPInputStream.close does not correctly trigger the closing of the file stream (#6323) (#6329)
• [Fix] Fix doris stream load failed not reported error (#6315)
• [fix][connector-rocketmq]Fix a NPE problem when checkpoint.interval is set too small(#6624)
• [Bugfix][TDengine] Fix the issue of losing the driver due to multiple calls to the submit job REST API #6581 (#6596)
• [Fix][StarRocks] Fix NPE when upstream catalogtable table path only have table name part (#6540)

Formats

• [Bug] [formats] Fix fail to parse line when content contains the file delimiter (#6589)

Zeta(ST-Engine)

• [Hotfix] fix http source can not read yyyy-MM-dd HH:mm:ss format bug & Improve DateTime Utils (#6601)
• [Fix][Zeta] Fix the thread stuck problem caused by savepoint checking mechanism (#6568)
• [Fix][Zeta] improve the local mode hazelcast connection (#6521)
• [Fix][Zeta] Fix thread classloader be set to null when use cache mode (#6509)
• [Bug] [zeta] Fix null pointer exception when submitting jobs (#6492)
• [bugfix] [Zeta] Fix the problem of class loader not releasing when using REST API to submit jobs
• [BUG][Zeta]job name Display error #6470
• [Hotfix][Zeta] Fix job deadlock when schema change (#6389)

E2E

• [E2E] Enable StarRocksCDCSinkIT (#6626)

Improve

• [Doc][Improve] Add Support Chinese for seatunnel-engine (#6656)
• [Doc][Improve]Add Support Chinese for start-v2/locally/quick-start-flink.md and start-v2/locally/quick-start-spark.md (#6412)
• [Improve] add icon for idea (#6394)
• [Improve] Add deprecated annotation for ReadonlyConfig::toConfig (#6353)

Core

• [Improve][RestAPI] always return jobId when call getJobInfoById API (#6422)
• [Improve][RestAPI] return finished job info when job is finished (#6576)
• [Improve] Improve MultiTableSinkWriter prepare commit performance (#6495)
• [Improve] Add SaveMode log of process detail (#6375)
• [Improve][API] Unify type system api(data & type) (#5872)

Formats

• [Improve] Improve read with parquet type convert error (#6683)

Connector-V2

• [Improve][Connector-V2]Support multi-table sink feature for redis (#6314)
• [Improve][Connector-V2] oracle cdc e2e optimization (#6232)
• [Improve][Connector-V2]Support multi-table sink feature for httpsink (#6316)
• [Improve][Connector-V2] Support INFINI Easysearch (#5933)
• [Improve][Connector-V2] Support hadoop ha and kerberos for paimon sink (#6585)
• [Improve][CDC-Connector]Fix CDC option rule. (#6454)
• [Improve][CDC] Optimize memory allocation for snapshot split reading (#6281)
• [Improve][Connector-V2] Support TableSourceFactory on StarRocks (#6498)
• [Improve][Jdbc] Using varchar2 datatype store string in oracle (#6392)
• [Improve] StarRocksSourceReader use the existing client (#6480)
• [Improve][JDBC] Optimized code style for getting jdbc field types (#6583)
• [Improve][Connector-V2] Add ElasticSearch type converter (#6546)
• [Improve][Connector-V2] Support read orc with schema config to cast type (#6531)
• [Improve][Jdbc] Support custom case-sensitive config for dameng (#6510)
• [Improve][Jdbc] Increase tyepe converter when auto creating tables (#6617)
• [Improve][CDC] Optimize split state memory allocation in increment phase (#6554)
• [Improve][CDC] Improve read performance when record not contains schema field (#6571)
• [Improve][Jdbc] Add quote identifier for sql (#6669)
• [Improve] Add disable 2pc in SelectDB cloud sink (#6266)
• [Doc][Improve] Add Support Kerberos Auth For Kafka Connector #6653

CI

• [CI] Fix error repository name in ci config files (#4795)

Zeta(ST-Engine)

• [Improve][Zeta] Add classloader cache mode to fix metaspace leak (#6355)
• [Improve][Test] Fix test unstable on ResourceManger and EventReport module (#6620)
• [Improve][Test] Run all test when code merged into dev branch (#6609)
• [Improve][Test] Make classloader cache testing more stable (#6597)
• [Improve][Zeta][storage] update hdfs configuration, support more parameters (#6547)
• [Improve][Zeta]Optimize the logic of RestHttpGetCommandProcessor#getSeaTunnelServer() (#6666)

Transformer

• [Improve][Transform] Sql transform support inner strucy query (#6484)
• [Improve][Transform] Remove Fallback during parsing Transform process (#6644)
• [Improve][Transform] Remove can't find field exception (#6691)

Feature

Core

• [Feature][Tool] Add connector check script for issue 6199 (#6635)
• [Feature][Core] Support listening for message delayed events in cdc source (#6634)
• [Feature][Core] Support event listener for job (#6419)

Connector-V2

• [Feature][connector-v2] add xugudb connector (#6561)
• [Feature][Connector-V2] Support multi-table sink feature for paimon #5652 (#6449)
• [Feature][Connectors-V2][File]support assign encoding for file source/sink (#6489)
• [Feature][Connector]update pgsql-cdc publication for add table (#6309)
• [Feature][Paimon] Support specify paimon table write properties, partition keys and primary keys (#6535)
• [Feature][Feature] Support nanosecond in Doris DateTimeV2 type (#6358)
• [Feature][Feature] Support nanosecond in SelectDB DateTimeV2 type (#6332)
• [Feature][Feature] Supports iceberg sink #6198 (#6265)

Zeta(ST-Engine)

• [Zeta] Support config job retry times in job config (#6690)

Docs

• [Docs] fix kafka format typo (#6633)
• [Fix][Doc] Fixed links in some documents (#6673)
• [Fix][Doc] Fix some spell errors (#6628)
• [Fix][Doc] Fixed typography error in starrocks sink document (#6579)
• [Hotfix][Doc][Chinese] Fix invalid link about configure logging related parameters (#6442)
• [Fix][Doc]Seatunnel Engine/checkpoint-storage.md doc error(#6369)

2.3.4 Release Note

Bug Fix

Core

• [Core] [API] Fixed generic class loss for lists (#4421)
• [Starter] Fix the problem of "," being divided in [] (#5401)
• [Core] [API] Fix ReadonlyConfig lose key error (#5565)
• [Core] [API] Fix get bytes from LinkHashMap (#5622)
• [Core] [API] Fix log error when multi-table sink close (#5683)
• [Core] [API] Fix MultiTableSink return committer but sink do not support (#5710)
• [Core] [API] Fix the error msg when parse schema with unsupported type (#5790)
• [Core] [API] Fix flaky test OptionUtilTest.test (#5894)
• [Core] [API] Fix SaveModeHandler not be closed (#5843)
• [Core] [API] Fix MultiTableSinkWriter thread index always 1 (#5832)
• [Core] [API] Fix SeaTunnelRow::getBytesSize not support map interface (#5990)
• [Core] [Common] Fix FileUtils::createNewFile not create new file (#5943)
• [Core] [API] Fix Debezium format cannot parse date/time/timestamp (#5887)
• [Starter] When inside double quotes,',' are treated as normal characters instead of delimiters (#6042
• [Core] [Common] Replace CommonErrorCodeDeprecated.JSON_OPERATION_FAILED (#5978)
• [Core] [API] Fix Object.class option value can not return normal value (#6247)

Transformer-V2

• [All] Fix PrimaryKey in transform (#5704)
• [All] Fix cast to timestamp, date, time bug (#5812)

Formats

• [Text] Allow the entry in the map to be null and allow the key in the entry to be null (#5277)

Connector-V2

• [Connector-V2] [Clickhouse] Fix clickhouse old version compatibility (#5326)
• [Connector-V2] [Clickhouse] Fix http header cover (#5446)
• [Connector-V2] [StarRocks] Fix starrocks template sql parser (#5332)
• [Connector-V2] [Hive] Fix the bug that hive-site.xml can not be injected in HiveConf (#5261)
• [Connector-V2] [Clickhouse] Fix clickhouse sink flush bug (#5448)
• [Connector-V2] [Hive] Fix An error occurred reading an empty directory (#5427)
• [Connector-V2] [Hive] Fix An error occurred reading an empty directory (#5427)" (#5487)
• [Connector-V2] [Oss jindo] Fix the problem of jindo driver download failure. (#5511)
• [Connector-V2] [Oss jindo] Remove useless code (#5540)
• [Connector-V2] [File] Fix WriteStrategy parallel writing thread unsafe issue (#5546)
• [Connector-V2] [CDC] Fix the cdc bug about NPE when the original table deletes a field (#5579)
• [Connector-V2] [Jdbc] oracle catalog create table repeat and oracle pg null point (#5517)
• [Connector-V2] [CDC] Fix thread-unsafe collection container in cdc enumerator (#5614)
• [Connector-V2] [Mongodb] Fixed unsupported exception caused by bsonNull (#5659)
• [Connector-V2] [File] Fix file sink isPartitionFieldWriteInFile occurred exception when no columns are given (#5508)
• [Connector-V2] [Doris] Fix RestService report NullPointerException (#5319)
• [Connector-V2] [MaxCompute] Fix MaxCompute use not exist SCHEMA option (#5708)
• [Connector-V2] [Doris] Using try-with-resources to simplify the code. (#4995)
• [Connector-V2] [Clickhouse] Fixed an out-of-order BUG with output data fields of clickhouse-sink (#5346)
• [Connector-V2] [Jdbc] support postgresql xml type (#5724)
• [Connector-V2] [Jdbc] Nullable Column source have null data could be unexpected results. (#5560)
• [Connector-V2] [Iceberg] Iceberg source lost data with parallelism option (#5732)
• [Connector-V2] [Jdbc] Fix PG will not create index when using auto create table #5721
• [Connector-V2] [Jdbc] Fix database identifier (#5756)
• [Connector-V2] [CDC] Fix MultiTableSink restore failed when add new table (#5746)
• [Connector-V2] [Postgres CDC] Fix Postgres create table test case failed (#5778)
• [Connector-V2] [CDC] Clean unused code (#5785)
• [Connector-V2] [CDC] Fix state recovery error when switching a single table to multiple tables (#5784)
• [Connector-V2] [ElasticSearch] Fixed conversion exception of elasticsearch array format (#5825)
• [Connector-V2] [Jdbc] Fix read from Oracle Date type value lose time (#5814)
• [Connector-V2] [Pulsar] Fix: update IDENTIFIER = Pulsar for pulsar-datasource on project:seatunnel-web (#5852)
• [Connector-V2] [Jdbc] Fix Hive-Jdbc use krb5 overwrite kerberosKeytabPath (#5891)
• [Connector-V2] [InfluxDB] Resolve invalid SQL in initColumnsIndex method caused by direct QUERY_LIMIT appendage with 'tz' function. (#4829)
• [Connector-V2] [Jdbc] Fix cdc updates were not filtering same primary key (#5923)
• [Connector-V2] [File] Parquet reader parsing array type exception. (#4457)
• [Connector-V2] [Http] Fix bug http config no schema option and improve e2e test add case (#5939)
• [Connector-V2] [Doris] Fix DorisCatalog not implement name method (#5988)
• [Connector-V2] [TDengine] Fix the degree of multiple parallelism affects driver loading (#6020)
• [Connector-V2] [Jdbc] Fix jdbc setFetchSize error (#6005)
• [Connector-V2] [CDC] Fix CDC job cannot consume incremental data After restore run (#625) (#6094)
• [Connector-V2] [File] Fix the Issue of Abnormal Data Reading from Excel Files (#5932)
• [Connector-V2] [CDC] Fix NPE caused by adding a table for restore job (#6145)
• [Connector-V2] [Jdbc] Fix dameng catalog query table sql (#6141)
• [Connector-V2] [Jdbc] update pgsql catalog for save mode (#6080)
• [Connector-V2] [Jdbc] Fix Spliter Error in Case of Extensive Duplicate Data (#6026)
• [Connector-V2] [CDC] Fix added columns cannot be parsed after job restore (#6118)
• [Connector-V2] [CDC] Fix negative values in CDCRecordEmitDelay metric (#6259)
• [Connector-V2] [Oracle CDC] Fix invalid split key when no primary key (#6251)

Zeta(ST-Engine)

• [Zeta] Fix NotifyTaskRestoreOperation npe (#5362)
• [Zeta] Fix Zeta will close task twice error (#5422)
• [Zeta] Disable CheckpointTimeOutTest (#5438)
• [Zeta] Fix CDC task restore throw NPE (#5507)
• [Zeta] Multiple sink actions of the same type have the same name (#5499)
• [Zeta] Checkpoint exception status messages exclude state data (#5547)
• [Zeta] Fix memory leak issues related to checkpoints (#5539)
• [Zeta] Fix the checkpoint be blocked with long time (#5695)
• [Zeta] Fix the problem of unstable job status (#5450)
• [Zeta] Fix submit job api (#5702)
• [Zeta] Set default DeployMode to DeployMode.CLIENT (#5783)
• [Zeta] rest api submit a job with chinese name return Garbled code name (#5870)
• [Zeta] Fix CheckpointCoordinator report NPE when ack not existed pending checkpoint (#5909)
• [Zeta] Fix submit job has the same job name error. (#6041)
• [Zeta] Fixed the problem that the return list is empty due to no status parameters (#6040)
• [Zeta] Fix zeta scheduler bug (#6050)
• [Zeta] Fix job can not restore when last checkpoint failed (#6193)
• [Zeta] [Rest-API] Submit or stop job from an inactive master node (#6217)

E2E

• [E2E] [Common] Update test container version of seatunnel engine (#5323)
• [E2E] [Jdbc] Fix not remove docker image after test finish on jdbc suite (#5586)
• [E2E] [ClickHouse] Enhance ClickHouse E2E testing to trigger multiple checkpoints (#5476)
• [E2E] Fix not remove docker image after test finish on jdbc suite (#5586)
• [E2E] Fix ConnectorPackageServiceContainer miss implement getSavePointCommand/getRestoreCommand (#5780)
• [E2E] Fix build failed cause by JdbcHiveIT and SparkSinkTest (#5798)
• [E2E] Fix submit job case error (#6059)
• [E2E] Fixed action related err (#6264)
• [E2E] Lock mysql container version to 8.0 (#6263)

CI

• [CI] Fix jindo oss connector name (#5385)
• [Build] Fix error msg when fork repository not up to date. (#5497)
• [CI] Fix file change not be check when CI run in fork repository (#5515)
• [CI] remove jindo dependencies (#5528)
• [CI] Fix phoenix ci error (#5530)
• [Build] Update build version to 2.3.4-SNAPSHOT (#5619)
• [Build] Ensure install-plugin.sh compatibility with sh on Debian #5630 (#5631)
• [CI] [Chore] Remove useless sonar check script (#5665
• [Chore] Remove DISCLAIMER file (#5673)
• [CI] Fix CI unstable problem (#5896)
• [Build] Fix empty line in config/plugin_config causes BUILD FAILURE (#5921)
• [CI] Fix CI not run Kudu/AmazonSQS IT when not change api (#5955)
• [CI] Split doris e2e into separate modules (#5999)
• [CI] Fix Dead Links checker failure (#6016)
• [CI] Fix e2e error (#6018)
• [Build] Updated pom.xml (#6113)
• [Build] Solve the problem of example running failure (#6173)
• [Build] Fix build error (#6196)
• [CI] Fix engine client not close  (#6241)

Examples

• [Examples] modify the transform URL link (#5298)

Improve

• [Improve][CheckStyle] Remove useless 'SuppressWarnings' annotation of checkstyle. (#5260)
• [Improve][CheckStyle] Adjust the phase of spotless plugin for release plugin. (#5607)

Core

• [Core] [API] Remove CatalogTable field in CatalogTableUtil (#5521)
• [Core] [API] Move get schema logic from Config to ReadonlyConfig (#5534)
• [Starter] Throw IllegalArgumentException when find multiple connector jar for one pluginIdentifier (#5551)
• [Core] [API] Refactor CatalogTable and add SeaTunnelSource::getProducedCatalogTables (#5562)
• [Core] [API] Support config column/primaryKey/constraintKey in schema (#5564)
• [Core] [API] Remove useless ReadonlyConfig flatten feature (#5612)
• [Core] [Flink & Spark] Refactor Spark/Flink execution processor (#5595)
• [Core] [API] Mark SeaTunnelPluginLifeCycle as deprecated (#5625
• [Core] [API] Support config tableIdentifier for schema (#5628)
• [Core] [Pom] Add junit4 to the root pom (#5611)
• [Core] [API] Remove catalog tag for config file (#5645)
• [Core] [API] Remove useless transform code come from setTypeInfo (#5647)
• [Core] [API] Make sure CatalogTable options and partitionKeys are mutable (#5681)
• [Core] [API] Add default implement for SeaTunnelSource::getProducedType (#5670)
• [Core] [API] Add default implement for SeaTunnelSink::setTypeInfo (#5682)
• [Core] [API] Add warning for use fall back keys (#5753)
• [Core] [API] Adjust the sleep mode of flink and spark engine to be consistent with zeta (#5698)
• [Core] [API] Remove Factory option to avoid useless info (#5754)
• [Core] [API] Add field name to DataTypeConvertor to improve error message (#5782)
• [Core] [API] Remove use SeaTunnelSink::getConsumedType method and mark it as deprecated (#5755)
• [Core] [Common] Remove assert key word. (#5915)
• [Core] [Common] Clean flow control code (#5991)
• [Core] [Common] Adapt FILE_OPERATION_FAILED to CommonError (#5928)
• [Core] [API] Add serialVersionUID to Column
• [Core] [Common] Extend SupportResourceShare to spark/flink (#5847)
• [Core] [API] Remove checkpoint timeout check if checkpoint disabled (#6231)

Formats

• [Json]Use a static object mapper instead of creating it every time (#5460)
• [Json] Remove assert key word. (#5919)
• [Formats] Replace CommonErrorCodeDeprecated.JSON_OPERATION_FAILED. (#5948)
• [Formats] Refactor exception catch for ignoreParseErrors. (#6065)
• [Formats] Using number format for Decimal type in seatunnel-format-compatible-debezium-json (#5803)
• [Text] add dateTimeFormatter to parse ISO8601 (#5974)
• [Formats] Replace CommonErrorCodeDeprecated.JSON_OPERATION_FAILED (#5948)

Connector-V2

• [Connector-V2] [IoTDB] Remove scheduler in IoTDB sink (#5270)
• [Connector-V2] [InfluxDB] Remove scheduler in InfluxDB sink (#5271)
• [Connector-V2] [Dynamodb] Remove scheduler in Dynamodb sink (#5248)
• [Connector-V2] [StarRocks] Remove scheduler in StarRocks sink (#5269)
• [Connector-V2] [CDC] avoid listing tables under unnecessary databases (#5365)
• [Connector-V2] [Jdbc] Refactor AbstractJdbcCatalog (#5096)
• [Connector-V2] [CDC] Support flink running cdc job (#4918)
• [Connector-V2] [Assert] support 'DECIMAL' type and Fix 'Number' type precision issue (#5479)
• [Connector-v2] [Redis] Redis support select db (#5570)
• [Connector-v2] [CDC] Use Source to output the CatalogTable (#5626)
• [Connector-v2] [SqlServer CDC] add dataType datetimeoffset (#5548)
• [Connector-v2] [Jdbc] Support read multiple tables (#5581)
• [Connector-v2] [SqlServer CDC] Unified sqlserver TypeUtils type conversion mode (#5668)
• [Connector-v2] [Http] improve http e2e test (#5655)
• [Connector-v2] [AmazonDynamicDB] add amazondynamicdb source split (#5275)
• [Connector-v2] [File] parquet use system timezone (#5605)
• [Connector-v2] [Amazonsqs] Change amazonsqs to AmazonSqs as connector identifier (#5742)
• [Connector-v2] [File] unifiy option between file source/sink and update document (#5680)
• [Connector-v2] [AmazonDynamicDB] Code clean for AmazonDynamoDB connector (#5791)
• [Connector-v2] [MongoDB] Implement TableSourceFactory to create mongodb source
• [Connector-v2] [Jdbc] Optimize catalog-table metadata merge logic (#5828)
• [Connector-v2] [Jdbc] Rename getCountSql to getExistDataSql (#5838)
• [Connector-v2] [ClickHouse] Speed up ClickhouseFile Local generate a mmap object (#5822)
• [Connector-v2] [Jdbc] Improve Jdbc connector error message when datatype unsupported (#5864)
• [Connector-v2] [Jdbc] Reduce the time cost of getCatalogTable in jdbc (#5908)
• [Connector-v2] [StarRocks] Support create varchar field type in StarRocks (#5911)
• [Connector-v2] [StarRocks] add http socket timeout. (#5918)
• [Connector-v2] [File] Clean memory buffer of JsonWriteStrategy & ExcelWriteStrategy (#5925)
• [Connector-v2] [StarRocks] StarRocks support create table template with unique key (#5905)
• [Connector-v2] [CDC] Disable memory buffering when exactly_once is turned off (#6017)
• [Connector-v2] [Doris] Add batch flush in doris sink (#6024
• [Connector-v2] [Paimon] Adaptation Paimon 0.6 Version (#6061)
• [Connector-v2] [File] Make Oss implement source factory and sink factory (#6062)
• [Connector-v2] [File] Disable HDFSFileSystem cache (#6039)
• [Connector-v2] [Jdbc] Shade hikari in jdbc connector (#6116)
• [Connector-v2] [Jdbc] Supports Sqlserver Niche Data Types (#6122)
• [Connector-v2] [Kafka] Remove useless code for kafka connector (#6157)
• [Connector-v2] [Doris] Improve doris sink to random use be (#6132)
• [Connector-v2] [Http] Increase custom configuration timeout. (#6223)
• [Connector-v2] [Pulsar] Improve pulsar throughput performance. (#6234)
• [Connector-v2] [SqlServer CDC] Support int identity type in sql server (#6186)
• [Connector-v2] [Doris] Doris stream load use FE instead of BE (#6235)
• [Connector-v2] [Postgres CDC] Fix name typos (#6248)
• [Connector-v2] [Tdengine] support read bool column from tdengine (#6025)
• [Connector-v2] [Jdbc] Use PreparedStatement to sample data from column (#6242)

CI

• [CI] update sql-udf document (#5197)
• [CI][E2E][Zeta] Increase Zeta checkpoint timeout to avoid connector-file-sftp-e2e failed frequently (#5339)
• [CI] Fix phoenix ci error
• [Build] Put seatunnel-hadoop3-3.1.4-uber.jar into release binary package (#5743)
• [Test] Make sure the value in spark not be reused. (#5767)
• [Test] Move MaxCompute test case file (#5786)
• [CI] Always run all module unit test (#5800)
• [Test] Change System.out.println to log output. (#5912)
• [Test] Add some test case for command useage
• [Test] Fix sql server catalog test case failed (#6128)
• [Test] Fix JobMetricsTest unstable (#6152)
• [Test] Fix ConnectorSpecificationCheckTest not work (#5820

E2E

• [E2E] Remove unnecessary code to reduce disk pressure (#5613)
• [E2E] Enable IT case for Oceanbase Mysql mode (#5697)
• [E2E] load driver class from url on demand (#5712)
• [E2E] Jdbc test checking data consistency. (#5734)
• [E2E] Enable e2e log output and disable console sink log (#5879)
• [E2E] Improve e2e log for all engines (#5936)
• [E2E] Enhanced stability of Kudu E2E (#6258)

Zeta(ST-Engine)

• [Zeta] Optimize Test Cases CheckpointTimeOutTest.testJobLevelCheckpointTimeOut (#5403)
• [Zeta] Improve dependency packages (#5624)
• [Zeta] Change hard code config key to reference (#5618)
• [Zeta] Change class name for RestJobExecutionEnvironment implement (#5671)
• [Zeta] Change default Zeta client JVM heap value (#5674)
• [Zeta] Move generate_client_protocol.sh to engine module (#5667)
• [Zeta] Optimize SeaTunnel Zeta engine Jar package upload logic (#5542)
• [Zeta] Move RestJobExecutionEnvironment to rest package (#5764)
• [Zeta] Remove result_table_name from action name(checkpoint state key) (#5779)
• [Zeta] Refactor jar package service module (#5763)
• [Zeta] Expose client cluster-connect-timeout-millis to yaml (#5868)
• [Zeta] Reduce checkpoint completed log (#5916)
• [Zeta] Remove assert key words (#5947)
• [Zeta] Adjusting the log level for factory validation implementation. (#6153)
• [Zeta] Ignore useless wrong target slot error (#6135)
• [Zeta] Add restore when commit failed (#6101)

Transformer-V2

• [All] Code clean for SeaTunnel transform (#5810)
• [All] Add common error for transform (#5815)

Feature

Core

• [Core] [API] Add job-level configuration for checkpoint timeout. (#5222)
• [Core] [API] Catalog add Case Conversion Definition (#5328)
• [Core] [API] Add InMemoryCatalog for test and add new getCatalogTableFromConfig method (#5485)
• [Core] [Flink] Support Decimal Type with configurable precision and scale (#5419)
• [Core] [API] Add init and restoreCommit method in SinkAggregatedCommitter (#5598)
• [Core] [Flink] Support flow control in Flink (#5509)
• [Core] [Spark] Support SeaTunnel Time Type. (#5188)
• [Core] [Flink] Remove useless stageType. (#5650)
• [Core] [API] Support multi-table sink (#5620)
• [Core] [Spark] Support flow control in Spark (#5510)
• [Core] [Flink] Add external configuration parameters (#5480)
• [Core] [API] Remove all useless prepare, getProducedType method (#5741)
• [Core] [Common] Introduce new error define rule (#5793)
• [Core] [Common] Remove useless DeserializationFormatFactory and its implement (#5880)
• [Core] [API] Replace SeaTunnelRowType with TableSchema in the JdbcRowConverter
• [Core] [Flink] Upgrade flink source translation (#5100)
• [Core] [API] Add unsupported datatype check for all catalog (#5890)
• [Core] [Flink] Support record metrics in flink engine (#6035)

Connector-V2

• [Connector-V2] [SqlServer CDC] Support multi-table read (#4377)
• [Connector-V2] [Jdbc] Jdbc database support identifier (#5089)
• [Connector-V2] [Jdbc] jdbc connector supports Kingbase database (#4803)
• [Connector-V2] [Jdbc] Add tidb datatype convertor (#5440)
• [Connector-V2] [Jdbc] Add Dameng catalog (#5451)
• [Connector-V2] [File] Supports writing column names when the output type is file (CSV) (#5459)
• [Connector-V2] [File] FILE_FORMAT_TYPE is text/csv ,add parameter BaseSinkConfig.ENABLE_HEADER_WRITE: #5566 (#5567)
• [Connector-V2] [CDC] Support for preferring numeric fields as split keys (#5384)
• [Connector-V2] [File] Support read empty directory (#5591)
• [Connector-V2] [Fake&Assert] Add table-names from FakeSource/Assert to produce/assert multi-table (#5604)
• [Connector-V2] [Jdbc] Add OceanBase catalog (#5439)
• [Connector-V2] [File] Support LZO compress on File Read (#5083)
• [Connector-V2] [MongoDB CDC] Support MongoDB CDC running on flink (#5644)
• [Connector-V2] [Jdbc] Supporting more ways to configure connection parameters. (#5388)
• [Connector-V2] [Kafka] KafkaSource use Factory to create source (#5635)
• [Connector-V2] [Jdbc] Add connector amazonsqs (#5367)
• [Connector-V2] [Jdbc] Support catalog in MaxCompute Source (#5283)
• [Connector-V2] [Kudu] Refactor Kudu functionality and Sink support CDC data. (#5437)
• [Connector-V2] [Mysql CDC] Optimize the default value range of mysql server-id to reduce conflicts. (#5550)
• [Connector-V2] [Http] HTTP supports page increase #5477 (#5561)
• [Connector-V2] [Jdbc] Add Save Mode function and Connector-JDBC (MySQL) connector has been realized (#5663)
• [Connector-V2] [Jdbc] Support XMLTYPE data integration #5716 (#5723)
• [Connector-V2] [Jdbc] Support Hive JDBC Source Connector (#5424)
• [Connector-V2] [Http] Http paramter support custom encryption (#5727)
• [Connector-V2] [Kudu] Support TableSourceFactory/TableSinkFactory on kudu (#5789)
• [Connector-V2] [File] LocalFileSource support multiple table
• [Connector-V2] [Fake] FakeSource support generate different CatalogTable for MultipleTable (#5766)
• [Connector-V2] [Kudu] Support multi-table source read (#5878
• [Connector-V2] [Http] Support TableSourceFactory/TableSinkFactory on http (#5816)
• [Connector-V2] [Redis] Support TableSourceFactory/TableSinkFactory on redis (#5901)
• [Connector-V2] [Jdbc] Fix split key not support BigInteger type
• [Connector-V2] [File] LocalFile sink support multiple table (#5931)
• [Connector-V2] [Doris] Doris Catalog (#5175)
• [Connector-V2] [Kudu] Support multi-table sink feature for kudu (#5951)
• [Connector-V2] [File] Support using multiple hadoop account (#5903)
• [Connector-V2] [File] Put Multiple Table File API to File Base Module (#6033)
• [Connector-V2] [Paimon] Flink table store failed to prepare commit (#6057)
• [Connector-V2] [File] Add multiple table file sink to base (#6049)
• [Connector-V2] [Jdbc] add JDBC source support string type as partition key (#6079)
• [Connector-V2] [File] Support read .xls excel file (#6066)
• [Connector-V2] [CDC] Support read no primary key table (#6098)
• [Connector-V2] [Assert] Support check the precision and scale of Decimal type. (#6110)
• [Connector-V2] [Hbase] support array data. (#6100)
• [Connector-V2] [File] FTP source/sink add ftp connection mode (#6077) (#6099)
• [Connector-V2] [Jdbc] update sqlserver catalog for save mode (#6086)
• [Connector-V2] [CDC] Support custom table primary key (#6106)
• [Connector-V2] [Doris] Support SaveMode on Doris (#6085)
• [Connector-V2] [Jdbc] update oracle catalog for save mode (#6092)
• [Connector-V2] [ElasticSearch] add elasticsearch save_mode (#6046) (#6092)
• [Connector-V2] [Jdbc] Improve get column sql compatibility (#5664)
• [Connector-V2] [Jdbc] Improve get column sql compatibility (#5664)
• [Connector-V2] [Pulsar] Add Pulsar Sink Connector. (#4382)
• [Connector-V2] [StarRocks] add starrocks save_mode (#6029)
• [Connector-V2] [Oracle CDC] Support for oracle cdc (#5196)
• [Connector-V2] [Doris] Add Doris ConnectorV2 Source (#6161)
• [Connector-V2] [Jdbc] Support uuid in postgres jdbc (#6185)
• [Connector-V2] [Oracle CDC] Support read no primary key table (#6209)
• [Connector-V2] [Oracle CDC] Fix jdbc setFetchSize error (#6210)
• [Connector-V2] [Oracle CDC] Fix state recovery error when switching a single table to multiple tables (#6211)
• [Connector-V2] [Oracle CDC] Clean unused code (#6212)
• [Connector-V2] [File] add s3file save mode function (#6131)
• [Connector-V2] [Oracle CDC] Support custom table primary key (#6216)
• [Connector-V2] [CDC] Add date type and float type column split support (#6160)
• [Connector-V2] [Postgres CDC] Support for Postgres cdc (#5986)
• [Connector-V2] [Postgres CDC] Update jdbc fetchsize (#6245)
• [Connector-V2] [CDC] Disable exactly_once by default to improve stability (#6244)
• [Connector-V2] [CDC] Support Short and Byte Type in spliter (#6027)
• [Connector-V2] [Jdbc] Improve query Approximate Total Row Count of a Table (#5972)

Zeta(ST-Engine)

• [Zeta] Add the UNKNOWABLE job status. (#5303)
• [Zeta] Support flow control in zeta (#5502)
• [Zeta] [REST-API] Stop a running job. (#5512)
• [Zeta] Support Zeta engine on Kubernetes (#5594)
• [Zeta] In batch mode, checkpoint can be disabled (#5914)
• [Zeta] Change skip trigger checkpoint log level to debug (#5954)
• [Zeta] add new job status DOING_SAVEPOINT and SAVEPOINT_DONE (#5917)
• [Zeta] Add waitForJobCompleteV2 api (#5965)
• [Zeta] Can submit job with rest api to Zeta master node automaticly (#5950)
• [Zeta] [REST-API] get finished jobs info (#5949)
• [Zeta] Fix transform action return same name (#6034)
• [Zeta] Unify job env parameters (#6003)
• [Zeta] Add TaskGroupLocation to the thread name of the TaskExecutionService (#6095)
• [Zeta] Using G1 as defaut garbage colletor in zeta (#6114)
• [Zeta] Fix start with savepoint with no checkpoint file error (#6215)
• [Zeta] Support hocon style declare row type in generic type (#6187)

CI

• [Bin] Add .bat script for all shell script. (#5445)
• [INFRA] Move CI to running on fork repository container (#5495)
• [Build] Remove connector/seatunnel directory (#5489)
• [INFRA] Update PR template to add test and user change question (#5486)
• [INFRA] Add log for notify_test_workflow.yml to trace error reason
• [INFRA] Fix notify_test_workflow.yml not stable
• [Test]Remove docker image after test finish on jdbc suite (#5568)
• [Test] Add test for ResourceManager to keep task will be deployed in different node (#5518)
• [Chore] Remove useless .scalafmt.conf file (#5616)
• [LICENSE] Add hadoop license (#6067)
• [Build] Add seatunnel-spark-3-starter.jar into release package (#6044)
• [Test] Reduce repeated catalog test times (#6207)
• [CI] Make sure notify_test_workflow.yml error will be throwed (#6226)

Formats

• [Ogg] Support read ogg format message #4201 (#4225)
• [Json] Remove assert key word. (#5919)
• [Avro] support avro format (#5084)
• [Formats] Refactor exception catch for ignoreParseErrors. (#6065)
• [Avro] improve avro format convert (#6082)

Transformer-V2

• [All] add JsonPath transform (#5632)
• [All] Support SqlTransform Not Like Expression (#5768)
• [All] add from_unixtime function (#5462)
• [All] Support case when Expression (#6123)

Docs

• [Docs] Refactor connector-v2 docs using unified format Feishu (#5343)
• [Docs] Refactor IoTDB sink docs (#5306)
• [Docs] Correct word errors (#5360)
• [Docs] Improved iceberg documentation (#5335)
• [Docs] Use short url https://s.apache.org/seatunnel-slack to replace long URL (#5363)
• [Docs] improve http doc param body des (#5368)
• [Docs] Refactor connector-v2 docs using unified format Slack (#5344)
• [Docs] update sql-udf document (#5197)
• [Docs] Refactor MySQL-CDC docs (#5302)
• [Docs] Replace username by user in the options description of FtpFile (#5421)
• [Docs] update iotdb document (#5404)
• [Docs] Added the mysql Connector Document version title Example pr (#5249)
• [Docs] add Parallel parallelism (#5310)
• [Docs] Http source option key poll_interval_ms different in source code (#5430)
• [Docs] Fix error examples of kafka sink (#5527)
• [Docs] Improve Console Sink document (#5230)
• [Docs] Add how to change logging configuration of e2e test (#5589)
• [Docs] Add RocketMq connector (#5361)
• [Docs] Fix build status in README.md not update (#5574)
• [Docs] hdfsFile file_format changed to file_format_type (#5653)
• [Docs] Improved README.md (#5662)
• [Docs] Add FakeSource connector documentation (#5255)
• [Docs] Introduce SeaTunnel web project in README.md (#5634)
• [Docs] Added Table of Contents and FAQs to README (#5693)
• [Docs] Update quick-start-spark.md (#5795)
• [Docs] Add Socket connector doc #5255 (#5287)
• [Docs] Improve file sink doc (#5799)
• [Docs] Add SqlServer connector documentation (#5498)
• [Docs] update (#5808)
• [Docs] Add hive jdbc reference value (#5882)
• [Docs] Checkpoint-Storage description is incorrect. (#5883)
• [Docs] Reconstruct the OssFile connector document (#5233)
• [Docs] Fix oss connector can not run bug (#6010)
• [Docs] update doc for jdbc-connector (#5765)
• [Docs] Add V2 connector jdbc document parameter can speed up data import PR (#6176)
• [Docs] Modify some document title specifications (#6237)
• [Docs] Reconstruct the Socket Source and SftpFile connector document (#5386)
• [Docs] Improve doc of driver put path
• [Docs] Correct the introduction of array element types and map key types (#6261)

2.3.3 Release Note

Bug fix

Core

• [Translation] [Spark] Fix SeaTunnelRowConvertor fail to convert when schema contains row type (#5170)

Connector

• [Connector] [Jdbc] Fix the problem of JdbcOutputFormat database connection leak (#4802)
• [Connector-V2] [AmazonDynamoDB] Fix the problem that all table data cannot be obtained (#5146)
• [Connector-V2] [CDC-base] Fix cdc connection does not close (#4922)
• [Connector-V2] [Clickhouse] Fix clickhouse source connector read Nullable() type is not null,example:Nullable(Float64) while value is null the result is 0.0 (#5080)
• [Connector-V2] [ClickhouseFile] Fix ClickhouseFile write file failed when field value is null (#4937)
• [Connector-V2] [Doris] Add streamload Content-type for doris URLdecode error (#4880)
• [Connector-V2] [E2E] Fix maven scope (#4901)
• [Connector-V2] [File-Base] Fix Hadoop Kerberos authentication related issues (#5171)
• [Connector-V2] [Flink-runtime] Flink register table Environment The running mode is set tojob.mode (#4826)
• [Connector-V2] [HbaseSink] Fix default timestamp (#4958)
• [Connector-V2] [Hive] Support user-defined hive-site.xml (#4965)
• [Connector-V2] [Jdbc] Support postgresql json type (#5194)
• [Connector-V2] [Jdbc] Fix XA Transaction bug (#5020)
• [Connector-V2] [Jdbc] Modify error comment in JdbcSourceConfigFactory (#5088)
• [Connector-V2] [Jdbc-Oracle] Fix the Oracle number type mapping problem (#5209)
• [Connector-V2] [Jdbc-Postgres] PostgreSQL versions below 9.5 are compatible use cdc sync problem (#5120)
• [Connector-V2] [JindoOssFile] Fix plugin-mapping.properties (#5215)
• [Connector-V2] [LocalFileSink] Fix LocalFile Sink file_format_type (#5118)
• [Connector-V2] [Mongodb] Fix document error content and remove redundant code (#4982)
• [Connector-V2] [Mongodb] Compatible with historical parameters (#4997)
• [Connector-V2] [MongoDB-CDC] Incremental data kind error in snapshot phase (#5184)
• [Connector-V2] [MongoDB-CDC] Solve startup resume token is negative (#5143)
• [Connector-V2] [MongoDB-CDC] Refine data format to adapt to universal logic (#5162)
• [Connector-V2] [MongoDB-CDC] Fix array index anomalies caused by #5057 (#5195)
• [Connector-V2] [MySql-CDC] Fix jdbc connection leak for mysql (#5037)
• [Connector-V2] [MySql-CDC] Fix mysql bit column to java byte (#4817)
• [Connector-V2] [MySql-CDC] Fix listener not released when BinlogClient reuse (#5011)
• [Connector-V2] [MySql-CDC] ServerId from int to long (#5033) (#5035)
• [Connector-V2] [Paimon] Bump paimon-bundle version to 0.4.0-incubating (#5219)
• [Connector-V2] [Redis] Fix redis nodes format error (#4981)
• [Connector-V2] [StarRocks] Fix StarRocksJsonSerializer will transform array/map/row to string (#5281)
• [Connector-V2] [StarRocks] Fix code style (#4966)

Zeta(ST-Engine)

• [Zeta] Resolved the issue causing checkpoints to halt on tolerable-failure=0 (#5263)
• [Zeta] Fix the issue of two identical IDs appearing when executing seatunnel.sh -l as the job resumes (#5191)
• [Zeta] Fix SinkFlowLifeCycle without init lastCommitInfo (#5152)
• [Zeta] Fix MultipleTableJobConfigParser ignore env option (#5067)
• [Zeta] Fix checkpoint error report without msg (#5137)
• [Zeta] [REST-API] Fix class not found exception (#5063)
• [Zeta] [REST-API] Fix browser long type intercept (#5267)
• [Zeta] Fix task state memory leak (#5139)
• [Zeta] Fix when execute seatunnel.sh, console may print error (#4984)
• [Zeta] Fixed the issue of duplicated metrics caused by job fault tolerance or restore (#5214)
• [Zeta] Avoid Redundant Job Submissions by Checking Job Status (#5229)
• [Zeta] Fix the serialization issue of GetMetricsOperation during multi-node operation (#5206)
• [Zeta] Fix after the savepoint job is restored, the checkpoint file cannot be generated #4985 (#5051)
• [Zeta] Fixed multi-table job data loss and latency issues (#149) (#5031)
• [Zeta] Fix the HdfsStorage can not delete checkpoint file #5046 (#5054)
• [Zeta] Fix if the initialization task throws an error and the task can not return to master (#4888)

TEST

• [Test] Split updated modules integration test for part 5 (#5208)
• [Test] Add code style check when docs changed (#5183)
• [Test] Turn on the testSavepoint test (#5199)
• [Test] Fix the CI Job name error (#5032)
• [Test] [E2E] Fix testGetErrorInfo case error (#5282)
• [Test] [Kafka] Kafka e2e error (#5200)
• [Test] [MongoDB-CDC] Fix mongodb cdc e2e instability (#5128)
• [Test] Declare files that will always have UNIX line endings on checkout (#5221)
• [Test] Fixed a minor bug (#5274)
• [Test] Fixed IMap file storage e2e bug (#5237)
• [Test] Split updated modules integration test for part 4 (#5028)
• [Test] Fix zeta ci error (#5254)

Others

• Update the project description (#4967)
• Update version to 2.3.3-SNAPSHOT (#5043)
• Fix com.google.common.base.Preconditions to seatunnel shade one (#5284)
• Fix style error (#5280)
• Fix code style (#5092)

Improve

Core

• [API & Zeta] Using connector custom serializer encode/decode states (#5238)
• [API]Improve savemode api (#4767)

Connector

• [CDC Base] Add some split parameters to the optionRule (#5161)
• [CDC Base] Support exactly-once of cdc and fix the BinlogOffset comparing bug (#5057)
• [CDC Base] Add a fast sampling method that supports character types (#5179)
• [CDC Base] Change driver scope to provider (#5002)
• [CDC Base] Remove driver for cdc connector (#4952)
• [Connector-V2] [File] Optimize files commit order (#5045)
• [Connector-V2] [HiveSink] Remove drop partition when abort (#4940)
• [Connector-V2] [Jdbc] Remove scheduler in JDBC sink #4736 (#5168)
• [Connector-V2] [Jdbc] Check url not null throw friendly message (#5097)
• [Connector-V2] [Jdbc] Support for Decimal types as splict keys (#4634)
• [Connector-V2] [Mongodb] Sink support transaction update/writing (#5034)
• [Connector-V2] [Mongodb] Optimize reading logic (#5001)
• [Connector-V2] [Neo4j] Supports neo4j sink batch write and update docs (#4841)
• [Connector-V2] [OSS-Jindo] Optimize jindo oss connector (#4964)
• [Connector-V2] [Tablestore sink] Remove scheduler in Tablestore sink (#5272)

Transform

• [Transform-V2] [SQL] Support use catalogTableName as SQL expression (#5273)

CI

• [CI/CD] Remove 'paths-ignore', enable the code style check for markdown files (#5286)

Zeta(ST-Engine)

• [Zeta] Upgrade the key log output level (#4993)
• [Zeta] Don't trigger handleSaveMode when restore (#5192)
• [Zeta] Move checkpoint notify complete in checkpoint stage (#5185)
• [Zeta] Add sleep for Task to reduce CPU cost (#5117)
• [Zeta] Add an interface for batch retrieval of JobMetrics (#4576)
• [Zeta] Move imap storage file dependency packages to submodules (#5218)
• [Zeta] Improve CheckpointCoordinator log error when report error from task (#178) (#5134)
• [Zeta] Improve CheckpointCoordinator notify complete when restore (#5136)

Feature

Connector

• [Connector-V2] [CDC-base] Support tables without primary keys (with unique keys) (#163) (#5150)
• [Connector-V2] [CDC-base] Support disable/enable exactly once for INITIAL (#4921)
• [Connector-V2] [CDC-base] Change the time zone to the default time zone (#5030)
• [Connector-V2] [CDC-base] Support string type shard fields (#5147)
• [Connector-V2] [Clickhouse] Add clickhouse connector time zone key,default system time zone (#5078)
• [Connector-V2] [Clickhouse] Clickhouse writes with checkpoints (#4999)
• [Connector-V2] [Doris] Add doris factory type convertor (#5061)
• [Connector-V2] [File] Add config of 'file_filter_pattern', which used for filtering files (#5153)
• [Connector-V2] [File] Add cos source&sink (#4979)
• [Connector-V2] [Hive] Hive add option because web need (#5154)
• [Connector-V2] [Jdbc] Add JDBC Catalog auto create table (#4917)
• [Connector-V2] [Jdbc] Add oceanbase dialect factory (#4989)
• [Connector-V2] [Jdbc] JDBC source support string type as partition key (#4947)
• [Connector-V2] [Jdbc-DM] Add DM upsert support (#5073)
• [Connector-V2] [Kafka] Support read debezium format message from kafka (#5066)
• [Connector-V2] [Kafka] Connector-kafka source support data conversion extracted by kafka connect source (#4516)
• [Connector-V2] [Mongodb] Mongodb support cdc sink (#4833)
• [Connector-V2] [Mongodb-CDC] Support source mongodb cdc (#4923)
• [Connector-V2] [Mysql-CDC] Conversion of tinyint(1) to bool is supported (#5105)
• [Connector-V2] [Redis] Support redis to set expiration time (#4975)

Zeta(ST-Engine)

• [Zeta] [CDC-base] Support schema evolution framework(DDL) (#5125)
• [Zeta] [REST-API] Add REST API To Submit Job (#5107)
• [Zeta] Checkpoint support hdfs ha mode (#4942)
• [Zeta] The expiration time of a historical Job can be config (#5180)
• [Zeta] Support history service record job execute error (#5114)
• [Zeta] Configuration files support user variable replacement (#4969)

Transform

• [Transform-V2] [SQL] Support 'select *' and 'like' clause for SQL Transform plugin (#4991)

CI

• [CI] Update action config to support run CI on fork repo (#5065)
• [E2E] [FtpFile] Add ftp file e2e test case (#4647)

Docs

• [Connector-V2][Clickhouse] Reconstruct the clickhouse connector doc (#5085)
• [Connector-V2][Doris] Reconstruct the Doris connector document (#4903)
• [Connector-V2][HDFS] Refactor connector-v2 docs using unified format HDFS (#4871)
• [Connector-V2][Http] Reconstruct the Http connector document (#4962)
• [Connector-V2][Hudi] Reconstruct the Hudi connector document (#4905)
• [Connector-V2][Jdbc] Change the line boundary store value type to BigDecimal (#4900)
• [Connector-V2][Jdbc-DB2、Jdbc-Vertica] Improve DB2 Source Vertica Source & DB2 Sink Vertica Sink document (#5102)
• [Connector-V2][Jdbc-Oracle] Add JDBC Oracle Documentation (#5239)
• [Connector-V2][Kafka] Reconstruct the kafka connector document (#4778)
• [Connector-V2][MyHours] Reconstruct the MyHours connector document (#5129)
• [Connector-V2][PostgreSQL] Refactor connector-v2 docs using unified format PostgreSQL #4590 (#4757)
• [Connector-V2][S3File] Improve S3File Source & S3File Sink document (#5101)
• [Connector-V2][SelectDB-Cloud] Reconstruct the SelectDB-Cloud connector document (#5130)
• [Connector-V2][StarRocks] Reconstruct the StarRocks connector document (#5132)
• [Connector-V2][StarRocks] StarRocks nodeUrls property name fix (#4951)
• [README] Remove useless github workflow, and adjust description of 'engineering structure' (#4305)
• [Zeta] Add savepoint doc (#5081)
• Update bug-report.yml (#5160)
• Add Value types in Java to Schema feature (#5087)
• Documentation and partial word optimization (#4936)
• Change repository name from incubator-seatunnel to seatunnel (#4868)
• Update .asf.yaml (#5242)
• Update the Readme (#4968)
• Set Up with Kubernetes, dockerfile document error in constructing docker image (#5022)
• Fix Dockerfile and seatunnel-flink.yaml in Set Up with Kubernetes (#4793)
• Change the transform website url (#4954)
• Update quick-start-seatunnel-engine.md (#4943)
• Remove incubating keyword in document (#5257)

2.3.2 Release Note

Bug fix

Core

• [Core] [API] Fixed generic class loss for lists (#4421)
• [Core] [API] Fix parse nested row data type key changed upper (#4459)

Connector-V2

• [Json-format] [Canal-Json] Fix json deserialize NPE (#4195)
• [Connector-V2] [Jdbc] Field aliases are not supported in the query of jdbc source. (#4210)
• [Connector-V2] [Jdbc] Fix connection failure caused by connection timeout. (#4322)
• [Connector-V2] [Jdbc] Set default value to false of JdbcOption: generate_sink_sql (#4471)
• [Connector-V2] [JDBC] Fix TiDBCatalog without open (#4718)
• [Connector-V2] [Jdbc] Fix XA DataSource crash(Oracle/Dameng/SqlServer) (#4866)
• [Connector-V2] [Pulsar] Fix the bug that can't consume messages all the time. (#4125)
• [Connector-V2] [Eleasticsearch] Document description error (#4390)
• [Connector-V2] [Eleasticsearch] Source deserializer error and inappropriate (#4233)
• [Connector-V2] [Kafka] Fix KafkaProducer resources have never been released. (#4302)
• [Connector-V2] [Kafka] Fix the permission problem caused by client.id. (#4246)
• [Connector-V2] [Kafka] Fix KafkaConsumerThread exit caused by commit offset error. (#4379)
• [Connector-V2] [kafka] Fix the problem that the partition information can not be obtained when kafka is restored (#4764)
• [Connector-V2] [SFTP] Fix incorrect exception handling logic (#4720)
• [Connector-V2] [File] Fix read temp file (#4876)
• [Connector-V2] [CDC Base] Solving the ConcurrentModificationException caused by snapshotState being modified concurrently. (#4877)
• [Connector-V2] [Doris] update last checkpoint id when doing snapshot (#4881)
• [Connector-v2] [kafka] Fix the short interval of pull data settings and revise the format (#4875)
• [Connector-v2] [RabbitMQ] Fix reduplicate ack msg bug and code style (#4842)
• [Connector-V2] [Jdbc] Fix the error of extracting primary key column in sink (#4815)
• [Connector-V2] [Jdbc] Fix reconnect throw close statement exception (#4801)
• [Connector-V2] [Jdbc] Fix sqlserver system table case sensitivity (#4806)
• [Connector-v2] [File] Fix configuration file format and error comments (#4762)
• [Connector-v2] [Jdbc] Fix oracle sql table identifier (#4754)
• [Connector-v2] [Clickhouse] fix get clickhouse local table name with closing bracket from distributed table engineFull (#4710)

Zeta(ST-Engine)

• [Zeta] Fix LogicalDagGeneratorTest testcase (#4401)
• [Zeta] Fix MultipleTableJobConfigParser parse only one transform (#4412)
• [Zeta] Fix missing common plugin jars (#4448)
• [Zeta] Fix handleCheckpointError be called while checkpoint already complete (#4442)
• [Zeta] Fix job error message is not right bug (#4463)
• [Zeta] Fix finding TaskGroup deployment node bug (#4449)
• [Zeta] Fix the bug of conf (#4488)
• [Zeta] Fix Connector load logic from zeta (#4510)
• [Zeta] Fix conflict dependency of hadoop-hdfs (#4509)
• [Zeta] Fix TaskExecutionService synchronized lock will not release (#4886)
• [Zeta] Fix TaskExecutionService will return not active ExecutionContext (#4869)
• [Zeta] Fix deploy operation timeout but task already finished bug (#4867)
• [Zeta] Fix restoreComplete Future can't be completed when cancel task (#4863)
• [Zeta] Fix IMap operation timeout bug (#4859)
• [Zeta] fix pipeline state not right bug (#4823)
• [Zeta] Fix the incorrect setting of transform parallelism (#4814)
• [Zeta] Fix master active bug (#4855)
• [Zeta] Fix completePendingCheckpoint concurrent action (#4854)
• [Zeta] Fix engine runtime error (#4850)
• [Zeta] Fix TaskGroupContext always hold classloader so classloader can't recycle (#4849)
• [Zeta] Fix task notifyTaskStatusToMaster failed when job not running or failed before run (#4847)
• [Zeta] Fix cpu load problem (#4828)
• [zeta] Fix the deadlock issue with JDBC driver loading (#4878)

E2E

• [E2E] [Kafka] Fix kafka e2e testcase (#4520)
• [Container Version] Fix risk of unreproducible test cases #4591
• [E2e] [Mysql-cdc] Removing the excess MySqlIncrementalSourceIT e2e reduces the CI time (#4738)

Improve

Core

• [Core] [Spark] Push transform operation from Spark Driver to Executors (#4503)
• [Core] [Starter] Optimize code structure & remove redundant code (#4525)
• [Core] [Translation] [Flink] Optimize code structure & remove redundant code (#4527)
• [Core] [Starter] Add check of sink and source config to avoid null pointer exception. (#4734)

Connector-V2

• [Connector-V2] [CDC] Improve startup.mode/stop.mode options (#4360)
• [Connector-V2] [CDC] Optimize jdbc fetch-size options (#4352)
• [Connector-V2] [CDC] Fix chunk start/end parameter type error (#4777)
• [Connector-V2] [SQLServer] Fix sqlserver catalog (#4441)
• [Connector-V2] [StarRocks] Improve StarRocks Serialize Error Message (#4458)
• [Connector-V2] [Jdbc] add the log for sql and update some style (#4475)
• [Connector-V2] [Jdbc] Fix the table name is not automatically obtained when multiple tables (#4514)
• [Connector-V2] [S3 & Kafka] Delete unavailable S3 & Kafka Catalogs (#4477)
• [Connector-V2] [Pulsar] Support Canal Format
• [Connector-V2] [CDC base] Implement Sample-based Sharding Strategy with Configurable Sampling Rate (#4856)
• [Connector-V2] [SelectDB] Add a jobId to the selectDB label to distinguish between tasks (#4864)
• [Connector-V2] [Doris] Add a jobId to the doris label to distinguish between tasks (#4839) (#4853)
• [Connector-v2] [Mongodb]Refactor mongodb connector (#4620)
• [Connector-v2] [Jdbc] Populate primary key when jdbc sink is created using CatalogTable (#4755)
• [Transform-V2] Optimize SQL Transform package and Fix Spark type conversion bug of transform (#4490)

CI

• [CI] Fix error repository name in ci config files (#4795)

Zeta(ST-Engine)

• [Zeta] Support run the server through daemon mode (#4161)
• [Zeta] Change ClassLoader To Improve the SDK compatibility of the client (#4447)
• [Zeta] Client Support Async Submit Job (#4456)
• [Zeta] Add more detailed log output. (#4446)
• [Zeta] Improve seatunnel-cluster.sh (#4435)
• [Zeta] Reduce CPU Cost When Task Not Ready (#4479)
• [Zeta] Add parser log (#4485)
• [Zeta] Remove redundant code (#4489)
• [Zeta] Remove redundancy code in validateSQL (#4506)
• [Zeta] Improve JobMetrics fetch performance (#4467)
• [Zeta] Reduce the operation count of imap_running_job_metrics (#4861)
• [Zeta] Speed up listAllJob function (#4852)
• [Zeta] async execute checkpoint trigger and other block method (#4846)
• [Zeta] Reduce the number of IMAPs used by checkpointIdCounter (#4832)
• [Zeta] Cancel pipeline add retry to avoid cancel failed. (#4792)
• [Zeta] Improve Zeta operation max count and ignore NPE (#4787)
• [Zeta] Remove serialize(deserialize) cost when use shuffle action (#4722)

Feature

Core

• [Core] [API] Support convert strings as List<T> option (#4362)
• [Core] [API] Add copy method to Catalog codes (#4414)
• [Core] [API] Add options check before create source and sink and transform in FactoryUtil (#4424)
• [Core] [Shade] Add guava shade module (#4358)

Connector-V2

• [Connector-V2] [CDC] [SQLServer] Support multi-table read (#4377)
• [Connector-V2] [Kafka] Kafka source supports data deserialization failure skipping (#4364)
• [Connector-V2] [Jdbc] [TiDB] Add TiDB catalog (#4438)
• [Connector-V2] [File] Add file excel sink and source (#4164)
• [Connector-v2] [Snowflake] Add Snowflake Source&Sink connector (#4470)
• [Connector-V2] [Pular] support read format for pulsar (#4111)
• [Connector-V2] [Paimon] Introduce paimon connector (#4178)
• [Connector V2] [Cassandra] Expose configurable options in Cassandra (#3681)
• [Connector V2] [Jdbc] Supports GEOMETRY data type for PostgreSQL (#4673)
• [Transform-V2] Add UDF SPI and an example implement for SQL Transform plugin (#4392)
• [Transform-V2] Support copy field list (#4404)
• [Transform-V2] Add support CatalogTable for FieldMapperTransform (#4423)
• [Transform-V2] Add CatalogTable support for ReplaceTransform (#4411)
• [Transform-V2] Add Catalog support for FilterRowKindTransform (#4420)
• [Transform-V2] Add support CatalogTable for FilterFieldTransform (#4422)
• [Transform-V2] Add catalog support for SQL Transform plugin (#4819)

Zeta(ST-Engine)

• [Zeta] Support for mixing Factory and Plugin SPI (#4359)
• [Zeta] Add get running job info by jobId rest api (#4140)
• [Zeta] Add REST API To Get System Monitoring Information (#4315)
• [Transform V2 & Zeta] Make SplitTransform Support CatalogTable And CatalogTable Evolution (#4396)
• [Zeta] Move driver into lib directory and change operation count (#4845)
• [Zeta] Add Metaspace size default value to config file (#4848)
• [Zeta] Reduce the frequency of fetching data from imap (#4851)
• [Zeta] Add OSS support for Imap storage to cluster-mode type (#4683)
• [Zeta] Improve local mode startup request ports (#4660)

Docs

• [Docs] Optimizes part of the Doris and SelectDB connector documentation (#4365)
• [Docs] Fix docs code style (#4368)
• [Docs] Update jdbc doc and kafka doc (#4380)
• [Docs] Fix max_retries default value is 0. (#4383)
• [Docs] Fix markdown syntax (#4426)
• [Docs] Fix Kafka Doc Error Config Key "kafka." (#4427)
• [Docs] Add Transform to Quick Start v2 (#4436)
• [Docs] Fix Mysql sink format doc (#4800)
• [Docs] Add the generate sink sql parameter for the jdbc sink document (#4797)
• [Docs] Add the generate sink sql parameter And example (#4769)
• [Docs] Redshift add defaultRowFetchSize (#4616)
• [Docs] Refactor connector-v2 docs using unified format Mysql (#4590)

2.3.1-release

Bug fix

Core

• [Core] [Shade] [Hadoop] Fix hadoop shade dependency can not be used (#3835)
• [Core] Fix Handover using linked blocking queue cause the oom (#3469)
• [Core] Fix a bug of Spark Translation when has a timestamp/date type filed in source (#4226)
• [Core] Remove unnecessary row conversion (#4335)

Connector-V2

• [Connecor-V2] [JDBC] Fix the problem that can not throw exception correctly (#3796)
• [Connector-V2] [JDBC] Fix xa transaction commit failure on pipeline restore (#3809)
• [Connector-V2] [Clickhouse] Fix the serializable problems of Committer (#3803)
• [Connector-V2] [Clickhouse] Fix the performance bug (#3910)
• [Connector-v2] [Clickhouse] Fix clickhouse write cdc changelog update event (#3951)
• [Connector-V2] [Clickhouse] Clickhouse File Connector failed to sink to table with settings like storage_policy (#4172)
• [Connector-V2] [Clickhouse] Clickhouse File Connector not support split mode for write data to all shards of distributed table (#4035)
• [Connector-V2] [File] Fix the error type of option rule path (#3804)
• [Connector-V2] [File] Text And Json WriteStrategy lost the sinkColumnsIndexInRow (#3863)
• [Connector-V2] [Kafka] Fix the bug that can not parse offset format (#3810)
• [Connecor-V2] [Kafka] Fix the default value of commit_on_checkpoint (#3831)
• [Connector-V2] [Kafka] Json deserialize exception log no content (#3874)
• [Connector-V2] [Kafka] Fix commit kafka offset bug. (#3933)
• [Connector-V2] [Kafka] Fix the bug that kafka consumer is not close (#3836)
• [Connector-V2] [Kafka] Fix config option error (#4244)
• [Connector-V2] [CDC] Guaranteed to be exactly-once in the process of switching from SnapshotTask to IncrementalTask (#3837)
• [Connector-V2] [CDC] Fix concurrent modify of splits (#3937)
• [Connector-V2] [CDC] Fix jdbc sink generate update sql (#3940)
• [Connector-V2] [CDC] Fix cdc option rule error (#4018)
• [Connector-V2] [CDC] Fix cdc base shutdown thread not cleared (#4327)
• [Connector-V2] [ALL] Fix ConcurrentModificationException when snapshotState based on SourceReaderBase (#4011)
• [Connector-V2] Fix connector source snapshot state NPE (#4027)
• [Connector-v2] [Pulsar] Fix pulsar option topic-pattern bug (#3989)
• [Connector-V2] [ElasticSearch] Fix es source no data (#4076)
• [Connector-V2] [Hive] Fix hive unknownhost (#4141)
• [Connector-V2] [Pulsar] Fix Pulsar source consumer ack exception (#4237)
• [Connector-V2] [Maxcompute] Fix failed to parse some maxcompute type (#3894)
• [Connector-V2] [Doris] Fix Content Length header already present (#4277)
• [Connector-V2] [ElasticSearch] Fix es field type not support binary (#4274)
• [Connector-V2] [JDBC] Field aliases are not supported in the query of jdbc source (#4210)
• [Json-format] [Canal] Fix json deserialize NPE (#4195)

Zeta(ST-Engine)

• [Zeta] Fix the bug that checkpoint will be continued trigger after job CANCELED (#3808)
• [Zeta] Fix the bug that the source split can not be deserialized in local mode (#3817)
• [Zeta] Fix Engine Metrics will lose when Job be canceled. (#3797)
• [Zeta] Fix CheckpointIDCounter thread not safe problem (#3875)
• [Zeta] Fix SeatunnelChildFirstClassLoader load jackson error (#3884)
• [Zeta] Fix local mode can not generate logs (#3917)
• [Zeta] Fix actionSubtaskState can be null error (#3902)
• [Zeta] A checkpoint exception may be thrown when the active cancel task terminates with an exception (#3915)
• [Zeta] Fix NPE when scheduling sub plan fails. (#3909)
• [Zeta] Fix NPE when starting from checkpoint. (#3904)
• [Zeta] Fix metrics lose on pipeline restart (#3977)
• [Zeta] Fix Job will lost control when JobMaster init failed (#4045)
• [Zeta] Fix ResourceManager Assign Task Not Random (#4078)
• [Zeta] Fix NullPointerException when registering Reader with the enumerator (#4048)
• [Zeta] Fix clean TaskGroupContext Error when target node is offline (#4086)
• [Zeta] Fix Slot Status Not Sync After Restart When All Node Down (#4047)
• [Zeta] Fix parse job mode bug and improve doc (#4091)
• [Zeta] Fix Client Have Error Can't be Shutdown (#4099)
• [Zeta] Fix zeta bugs
• [Zeta] Fix checkpoint storage namespace (#4260)
• [Zeta] Fix read checkpoint file data is incomplete (#4263)
• [Zeta] The pipeline needs to wait for the CheckpointCoordinator to end (#4272)
• [Zeta] Fix CheckpointCoordinator Can't Trigger Timeout Task (#4276)
• [Zeta] Fix job crash when run it on Seatunnel Engine cluster (#4299)
• [Zeta] Fix Default Cluster Not Working In Config File (#3770)
• [Zeta] Adapt StarRocks With Multi-Table And Single-Table Mode (#4324)
• [Zeta] Fix TaskExecutionService Deploy Failed The Job Can't Stop (#4265)
• [Zeta] Fix cancelJob and checkpoint complete error (#4330)

E2E

• [Connector-V2] [SQLServer-CDC] Fix the bug of SQLServer-CDC it case (#3807)
• [Connector-V2] [Clickhouse] Fix the bug of clickhouse e2e case (#3985)

Improve

Core

• [Core] [API] Add parallelism and column projection interface (#3829)
• [Core] [Connector-V2] Add get source method to all source connector (#3846)
• [Core] [Shade] [Hadoop] Improve hadoop shade by including classes in package com.google.common.cache.* (#3858)
• [Core] Use ReadonlyConfig to avoid option being changed (#4056)
• [Core] Give the maven module a human readable name (#4114)
• [Core] Unified the checkpoint setting key of Flink (#4296)

Connector-V2

• [Connector-V2] [MySQL-CDC] Add mysql-cdc source factory (#3791)
• [Connector-V2] [MySQL-CDC] Ennable binlog watermark compare (#4293)
• [Connector-V2] [Kafka] Support user-defined client id (#3783)
• [Connector-V2] [Kafka] Support extract topic from SeaTunnelRow field (#3742)
• [Connector-V2] [Kafka] Add Kafka catalog (#4106)
• [Connector-V2] [JDBC] Remove unused options that in jdbc source factory (#3794)
• [Connector-V2] [JDBC] Support exactly-once semantics for JDBC source connector (#3750)
• [Connector-V2] [JDBC] Improve option rule (#3802) (#3864)
• [Connector-V2] [MongoDB] Support use source query in connector (#3697)
• [Connector-V2] [File] Add the exception stack detail for log output (#3805)
• [Connector-V2] [File] Improve file connector option rule and document (#3812)
• [Connector-V2] [File] Support skip number when reading text csv files (#3900)
• [Connector-V2] [File] Allow the user to set the row delimiter as an empty string (#3854)
• [Connector-V2] [File] Support compress (#3899)
• [Connector-V2] [File & Hive] Support kerberos in hive and hdfs file connector (#3840)
• [Connector-V2] [Hive] Improve config check logic (#3886)
• [Connector-V2] [Hive] Support assign partitions (#3842)
• [Connector-V2] [Hive] Support read text table & Column projection (#4105)
• [Connector-V2] [Clickhouse & Kafka & Rabbitmq & StarRocks] Change connector custom config prefix to map (#3719)
• [Connector-V2] [Clickhouse] Special characters in column names are supported (#3881)
• [Connector-V2] [Clickhouse] Remove Clickhouse Fields Config (#3826)
• [Connector-V2] [Email] Unified exception for email connector (#3898)
• [Connector-V2] [Iceberg] Unified exception for iceberg source connector (#3677)
• [Connector-v2] [StarRocks] Support write cdc changelog event(INSERT/UPDATE/DELETE) (#3865)
• [Connector-V2] [Fake] Improve fake connector (#3932)
• [Connector-V2] [Fake] Optimizing Data Generation Strategies (#4061)
• [Connector-V2] [InfluxDB] Unifie InfluxDB source fields to schema (#3897)
• [Connector-V2] [IoTDB] Unifie IoTDB source fields to schema (#3896)
• [Transform-V2] Add transform factory test (#3887)
• [Connector-V2] [SQLServer-CDC] Add sqlserver cdc optionRule (#4019)
• [Connector-V2] [Elasticsearch] Support https protocol (#3997)
• [Connector-V2] [Elasticsearch] Add ElasticSearch catalog (#4108)
• [Connector-V2] [Elasticsearch] Support dsl filter (#4130)
• [Connector-V2] [S3] Add S3Catalog (#4121)
• [Connector-V2] [Doris] Refactor some Doris Sink code as well as support 2pc and cdc (#4235)
• [Connector-V2] [CDC] Optimize options & add docs for compatible_debezium_json (#4351)

CI

• [CI] Imprve CI/CD process, split all connector it cases to 4 (#3832)
• [CI] Imprve CI/CD process, split all connector it cases to 5 (#4065)
• [E2E] Improve CI stability (#4068)
• [CI] Improve ci steps (#4314) (#4342)

Zeta(ST-Engine)

• [Zeta] Remove unnecessary dependencies in pom (#3795)
• [Zeta] Add the restart job logic when all nodes down (#3784)
• [Zeta] Add the logic that cancel CheckpointCoordinator first before cancel task (#3838)
• [Zeta] Remove seatunnel-api from engine storage (#3834)
• [Zeta] Suppress delete data log level (#4001)
• [Zeta] Set the write data timeout to be configurable (#4059)
• [Zeta] Job clean before JobMaster future complete (#4087)
• [Zeta] Add Slot Sequence To Avoid Active Check Error (#4097)
• [Zeta] Improve Client Job Info Message
• [Zeta] Client Job Info Message Add Order By Submit Time
• [Zeta] JVM parameters distinguish between client and server (#4297)

E2E

• [Connector-V2] [Iceberg] Refactor iceberg connector e2e test cases (#3820)
• [Connector-V2] [Datahub] Refactor DataHub connector e2e test cases (#3866)
• [Connector-V2] [MongoDB] Refactor MongoDB connector e2e test (#3819)
• [Connector-V2] [ES & InfluxDB & Redis] Add a 'nonNull' check to avoid npe when executing 'tearDown'. (#3967)
• [Connector-V2] [Http] Refactor the e2e test of http (#3969)
• [Connector-V2] Remove the use of scala.Tuple in the e2e module (#3974)
• [Zeta] Change E2E To support ClusterFaultToleranceIT (#3976)
• [Zeta] Statistics server job and system resource usage (#3982)
• [Transform-v2] Merge e2e tests and config files of SQL transform plugin (#4278)
• [Connector-V2] [Jdbc] Reactor jdbc e2e with new api, then remove the useless e2e case. For better performance, we split the jdbc e2e module into three modules. (#4165)

Feature

Core

• [Core] [Transform-V2] Support transform-v2 for flink engine (#3396)
• [Core] [Transform-V2] Support transform-v2 for spark (#3409)
• [Core] [Starter] Refactor starter module (#3798)
• [Core] [Transform-V1] Remove old version transform module (#3911)
• [Core] [API] Remove old version apis (#3920)
• [Core] [API] Add savemode feature API (#3885)
• [Core] [Shade] Add seatunnel-jackson module (#3947)
• [Core] Upgrade flink version to 1.14.6 (#3963)
• [Core] Support flink 1.16 (#3979)
• [Core] [Spark] Support spark 3.3 (#3984)
• [Core] [API] Support Other Config Type Adapter Extends SPI (#4016)
• [Core] [Catalog] Support create/drop table, create/drop database in catalog (#4075)
• [Core] Add the spotless plugin for unified code style (#3939) (#4101)
• [Core] [API] Add Metrics for Connector-V2 (#4017)
• [Core] Support config encryption (#4134)

Connector-V2

• [Connector-V2] [JDBC] Add SAP HANA connector (#3017)
• [Connector-V2] [Persistiq] Add Persistiq source connector (#3460)
• [Connector-V2] [TDEngine] Add tdengine source (#2832)
• [Connector-V2] [SelectDB Cloud] Support SelectDB Cloud Sink Connector (#3958)
• [Transform-V2] Add FieldMapper transform (#3781)
• [Transform-V2] Add SimpleSQL transform plugin (#4148)
• [Json-format] [canal] Support read canal format message (#3950)
• [Connector-V2] [CDC] Support export debezium-json format to kafka (#4339)
• [Connector-V2] [CDC] MySQL CDC supports deserialization of multi-tables (#4067)
• [Connector-V2] [Hbase] Introduce hbase sink connector (#4049)
• [Connector-V2] [MySQL-CDC] Support all database sync to StarRocks
• [Connector-V2] [StarRocks] Support auto create table when sink to it
• [Connector-V2] [MySQL-CDC] Support read database list (#4255)
• [Connector-V2] [CDC] Support add & dorp tables when restore cdc jobs (#4254)

Zeta(ST-Engine)

• [Zeta] Add close engine instance shell script (#3776)
• [Zeta] Zeta support print metrics information (#3913)
• [Zeta] Add savepoint and restore with savepoint (#3930)
• [Zeta] Support Get Error Message From Client When Job Failed (#3928)
• [Zeta] Add oss support for checkpoint storing (#3732)
• [Zeta] Add Zeta Client ShutdownHook To Cancel Job (#3946)
• [Zeta] Add a jvm.properties file to define the SeaTunnel Zeta JVM Options (#3771)
• [Zeta] Add rest api for runningjob detail (#4072)
• [Zeta] Support get cluster metrics from seatunnel zeta client (#4139)

Docs

• [Docs] Add transform v2 doc & remove transform v1 doc (#3786)
• [Docs] Add some tips in jdbc sink doc (#3916)
• [Docs] Add imap store doc (#4008)
• [Docs] SeaTunnel run with flink operator (#4024)
• [Docs] Add SQLServer cdc connector doc (#3993)
• [Docs] Update usage docs (#4030)
• [Docs] Modify the error of some documents (#4037)
• [Docs] Improve kafka sink docs (#4243)
• [Docs] Add the lost error codes in docs (#4275)
• [Docs] Improve StarRocks Sink Doc With Template Feature. (#4290)
• [Docs] Improve MySQL-CDC Doc For connect.timeout.ms field (#4292)

In this version, we have fixed numerous bugs in the Zeta engine, improving its stability and reliability. We have also improved the stability of the CI/CD process in the engineering aspect, optimizing the contributor's experience. In terms of connectors, we have added several new connectors, fixed hidden bugs in commonly used connectors, and refactored some of them to improve the stability of data transmission and enhance the user experience. In this release, we have also added support for MySQL CDC full-table synchronization to StarRocks, and automatic table creation is now possible on the StarRocks end.

2.3.0-release

Bug fix

Core

• [Core] [Starter] Fix the bug of ST log print failed in some jdk versions #3160
• [Core] [Shell] Fix bug that shell script about downloading plugins does not work #3462

Connector-V2

• [Connector-V2] [Jdbc] Fix the bug that jdbc source can not be stopped in batch mode #3220
• [Connector-V2] [Jdbc] Fix the bug that jdbc connector reset in jdbc connector #3670
• [Connector-V2] [Jdbc] Fix the bug that jdbc connector exactly-once it will throw NullPointerException #3730
• [Connector-V2] [Hive] Fix the following bugs of hive connector: 1. write parquet NullPointerException 2. when restore write from states getting error file path #3258
• [Connector-V2] [File] Fix the bug that when getting file system throw NullPointerException #3506
• [Connector-V2] [File] Fix the bug that when user does not config the fileNameExpression it will throw NullPointerException #3706
• [Connector-V2] [Hudi] Fix the bug that the split owner of Hudi connector may be negative #3184

ST-Engine

• [ST-Engine] Fix bug data file name will duplicate when use SeaTunnel Engine #3717
• [ST-Engine] Fix job restart of all nodes down #3722
• [ST-Engine] Fix the bug that checkpoint stuck in ST-Engine #3213
• [ST-Engine] Fix the bug that checkpoint failed in ST-Engine #3769

E2E

• [E2E] [Spark] Corrected spark version in e2e container #3225

Improve

Core

• [Core] [Starter] [Flink] Upgrade the method of loading extra jars in flink starter #2982
• [Core] [Pom] [Package] Optimize package process #3751

Connector-V1

• [Connector-V1] Remove connector v1 related codes from dev branch #3450

Connector-V2

• [Connector-V2] Add split templates for all connectors #3335
• [Connector-V2] [Redis] Support redis cluster mode & user authentication #3188
• [Connector-V2] [Clickhouse] Support nest type and array type in clickhouse connector #3047
• [Connector-V2] [Clickhouse] Support geo type in clickhouse connector #3141
• [Connector-V2] [Clickhouse] Improve double convert that in clickhouse connector #3441
• [Connector-V2] [Clickhouse] Improve float long convert that in clickhouse connector #3471
• [Connector-V2] [Kafka] Support setting read start offset or message time in kafka connector #3157
• [Connector-V2] [Kafka] Support specify multiple partition keys in kafka connector #3230
• [Connector-V2] [Kafka] Support dynamic discover topic & partition in kafka connector #3125
• [Connector-V2] [Kafka] Support text format for kafka connector #3711
• [Connector-V2] [IotDB] Add the parameter check logic for iotDB sink connector #3412
• [Connector-V2] [Jdbc] Support setting fetch size in jdbc connector #3478
• [Connector-V2] [Jdbc] Support upsert config in jdbc connector #3708
• [Connector-V2] [Jdbc] Optimize the commit process of jdbc connector #3451
• [Connector-V2] [Jdbc] Release jdbc resource when after using #3358
• [Connector-V2] [Oracle] Improve data type mapping of Oracle connector #3486
• [Connector-V2] [Http] Support extract complex json string in http connector #3510
• [Connector-V2] [File] [S3] Support s3a protocol in S3 file connector #3632
• [Connector-V2] [File] [HDFS] Support setting hdfs-site.xml #3778
• [Connector-V2] [File] Support file split in file connectors #3625
• [Connector-V2] [CDC] Support write cdc changelog event in elsticsearch sink connector #3673
• [Connector-V2] [CDC] Support write cdc changelog event in clickhouse sink connector #3653
• [Conncetor-V2] [CDC] Support write cdc changelog event in jdbc connector #3444

ST-Engine

• [ST-Engine] Improve statistic information print format that in ST-Engine #3492
• [ST-Engine] Improve ST-Engine performance #3216
• [ST-Engine] Support user-defined jvm parameters in ST-Engine #3307

CI

• [CI] Improve CI process #3179 #3194

E2E

• [E2E] [Flink] Support execute extra commands on task-manager container #3224
• [E2E] [Jdbc] Increased Jdbc e2e stability #3234

Feature

Core

• [Core] [Log] Integrate slf4j and log4j2 for unified management logs #3025
• [Core] [Connector-V2] [Exception] Unified exception API & Unified connector error tip message #3045
• [Core] [Shade] [Hadoop] Add hadoop shade package for SeaTunnel #3755

Connector-V2

• [Connector-V2] [Elasticsearch] Add elasticsearch source connector #2821
• [Connector-V2] [AmazondynamoDB] Add AmazondynamoDB source & sink connector #3166
• [Connector-V2] [StarRocks] Add StarRocks sink connector #3164
• [Connector-V2] [DB2] Add DB2 source & sink connector #2410
• [Connector-V2] [Transform] Add transform-v2 api #3145
• [Connector-V2] [InfluxDB] Add influxDB sink connector #3174
• [Connector-V2] [Cassandra] Add Cassandra Source & Sink connector #3229
• [Connector-V2] [MyHours] Add MyHours source connector #3228
• [Connector-V2] [Lemlist] Add Lemlist source connector #3346
• [Connector-V2] [CDC] [MySql] Add mysql cdc source connector #3455
• [Connector-V2] [CDC] [SqlServer] Add sqlserver cdc source connector #3686
• [Connector-V2] [Klaviyo] Add Klaviyo source connector #3443
• [Connector-V2] [OneSingal] Add OneSingal source connector #3454
• [Connector-V2] [Slack] Add slack sink connector #3226
• [Connector-V2] [Jira] Add Jira source connector #3473
• [Connector-V2] [Sqlite] Add Sqlite source & sink connector #3089
• [Connector-V2] [OpenMldb] Add openmldb source connector #3313
• [Connector-V2] [Teradata] Add teradata source & sink connector #3362
• [Connector-V2] [Doris] Add doris source & sink connector #3586
• [Connector-V2] [MaxCompute] Add MaxCompute source & sink connector #3640
• [Connector-V2] [Doris] [Streamload] Add doris streamload sink connector #3631
• [Connector-V2] [Redshift] Add redshift source & sink connector #3615
• [Connector-V2] [Notion] Add notion source connector #3470
• [Connector-V2] [File] [Oss-Jindo] Add oss jindo source & sink connector #3456

ST-Engine

• [ST-Engine] Support print job metrics when job finished #3691
• [ST-Engine] Add metrics statistic in ST-Engine #3621
• [ST-Engine] Support IMap file storage in ST-Engine #3418
• [ST-Engine] Support S3 file system for IMap file storage #3675
• [ST-Engine] Support save job restart status information in ST-Engine #3637

E2E

• [E2E] [Http] Add http type connector e2e test cases #3340
• [E2E] [File] [Local] Add local file connector e2e test cases #3221

Docs

• [Docs] [Connector-V2] [Factory] Add TableSourceFactory & TableSinkFactor docs #3343
• [Docs] [Connector-V2] [Schema] Add connector-v2 schema docs #3296
• [Docs] [Connector-V2] [Quick-Manaul] Add error quick reference manual #3437
• [Docs] [README] Improve README and refactored other docs #3619

2.3.0 Beta

[Connector V2]

[New Connector V2 Added]

• [Source] [Kafka] Add Kafka Source Connector (2953)
• [Source] [Pulsar] Add Pulsar Source Connector (1980)
• [Source] [S3File] Add S3 File Source Connector (3119)
• [Source] [JDBC] [Phoenix] Add Phoenix JDBC Source Connector (2499)
• [Source] [JDBC] [SQL Server] Add SQL Server JDBC Source Connector (2646)
• [Source] [JDBC] [Oracle] Add Oracle JDBC Source Connector (2550)
• [Source] [JDBC] [GBase8a] Add GBase8a JDBC Source Connector (3026)
• [Source] [JDBC] [StarRocks] Add StarRocks JDBC Source Connector (3060)
• [Sink] [Kafka] Add Kafka Source Connector (2953)
• [Sink] [S3File] Add S3 File Sink Connector (3119)

[Improve & Bug Fix]

• [Source] [Fake]

  • [Improve] Supports direct definition of data values(row) (2839)
  • [Improve] Improve fake source connector: (2944)
    • Support user-defined map size
    • Support user-defined array size
    • Support user-defined string length
    • Support user-defined bytes length
  • [Improve] Support multiple splits for fake source connector (2974)
  • [Improve] Supports setting the number of splits per parallelism and the reading interval between two splits (3098)

• [Source] [Clickhouse]

  • [Improve] Clickhouse Source random use host when config multi-host (3108)

• [Source] [FtpFile]

  • [BugFix] Fix the bug of incorrect path in windows environment (2980)
  • [Improve] Support extract partition from SeaTunnelRow fields (3085)
  • [Improve] Support parse field from file path (2985)

• [Source] [HDFSFile]

  • [BugFix] Fix the bug of incorrect path in windows environment (2980)
  • [Improve] Support extract partition from SeaTunnelRow fields (3085)
  • [Improve] Support parse field from file path (2985)

• [Source] [LocalFile]

  • [BugFix] Fix the bug of incorrect path in windows environment (2980)
  • [Improve] Support extract partition from SeaTunnelRow fields (3085)
  • [Improve] Support parse field from file path (2985)

• [Source] [OSSFile]

  • [BugFix] Fix the bug of incorrect path in windows environment (2980)
  • [Improve] Support extract partition from SeaTunnelRow fields (3085)
  • [Improve] Support parse field from file path (2985)

• [Source] [IoTDB]

  • [Improve] Improve IoTDB Source Connector (2917)
    • Support extract timestamp、device、measurement from SeaTunnelRow
    • Support TINYINT、SMALLINT
    • Support flush cache to database before prepareCommit

• [Source] [JDBC]

  • [Feature] Support Phoenix JDBC Source (2499)
  • [Feature] Support SQL Server JDBC Source (2646)
  • [Feature] Support Oracle JDBC Source (2550)
  • [Feature] Support StarRocks JDBC Source (3060)
  • [Feature] Support GBase8a JDBC Source (3026)

• [Sink] [Assert]

  • [Improve] 1.Support check the number of rows (2844) (3031):
    • check rows not empty
    • check minimum number of rows
    • check maximum number of rows
  • [Improve] 2.Support direct define of data values(row) (2844) (3031)
  • [Improve] 3.Support setting parallelism as 1 (2844) (3031)

• [Sink] [Clickhouse]

  • [Improve] Clickhouse Support Int128,Int256 Type (3067)

• [Sink] [Console]

  • [Improve] Console sink support print subtask index (3000)

• [Sink] [Enterprise-WeChat]

  • [BugFix] Fix Enterprise-WeChat Sink data serialization (2856)

• [Sink] [FtpFile]

  • [BugFix] Fix the bug of incorrect path in windows environment (2980)
  • [BugFix] Fix filesystem get error (3117)
  • [BugFix] Solved the bug of can not parse '\t' as delimiter from config file (3083)

• [Sink] [HDFSFile]

  • [BugFix] Fix the bug of incorrect path in windows environment (2980)
  • [BugFix] Fix filesystem get error (3117)
  • [BugFix] Solved the bug of can not parse '\t' as delimiter from config file (3083)

• [Sink] [LocalFile]

  • [BugFix] Fix the bug of incorrect path in windows environment (2980)
  • [BugFix] Fix filesystem get error (3117)
  • [BugFix] Solved the bug of can not parse '\t' as delimiter from config file (3083)

• [Sink] [OSSFile]

  • [BugFix] Fix the bug of incorrect path in windows environment (2980)
  • [BugFix] Fix filesystem get error (3117)
  • [BugFix] Solved the bug of can not parse '\t' as delimiter from config file (3083)

• [Sink] [IoTDB]

  • [Improve] Improve IoTDB Sink Connector (2917)
    • Support align by sql syntax
    • Support sql split ignore case
    • Support restore split offset to at-least-once
    • Support read timestamp from RowRecord
  • [BugFix] Fix IoTDB connector sink NPE (3080)

• [Sink] [JDBC]

  • [BugFix] Fix JDBC split exception (2904)
  • [Feature] Support Phoenix JDBC Source (2499)
  • [Feature] Support SQL Server JDBC Source (2646)
  • [Feature] Support Oracle JDBC Source (2550)
  • [Feature] Support StarRocks JDBC Source (3060)

• [Sink] [Kudu]

  • [Improve] Kudu Sink Connector Support to upsert row (2881)

• [Sink] [Hive]

  • [Improve] Hive Sink supports automatic partition repair (3133)

[Connector V1]

[New Connector V1 Added]

[Improve & Bug Fix]

• [Sink] [Spark-Hbase]
  • [BugFix] Handling null values (3099)

[Starter & Core & API]

[Feature & Improve]

• [Improve] [Sink] Support define parallelism for sink connector (2941)
• [Improve] [all] change Log to @slf4j (3001)
• [Improve] [format] [text] Support read & write SeaTunnelRow type (2969)
• [Improve] [api] [flink] extraction unified method (2862)
• [Feature] [deploy] Add Helm charts (2903)
• [Feature] [seatunnel-text-format] (2884)

[Bug Fix]

• [BugFix] Fix assert connector name error in config/plugin_config file (3127)
• [BugFix] [starter] Fix connector-v2 flink & spark dockerfile (3007)
• [BugFix] [core] Fix spark engine parallelism parameter does not working (2965)
• [BugFix] [build] Fix the invalidation of the suppression file of checkstyle in the win10 (2986)
• [BugFix] [format] [json] Fix jackson package conflict with spark (2934)
• [BugFix] [build] Fix the invalidation of the suppression file of checkstyle in the win10 (2986)
• [BugFix] [build] Fix the invalidation of the suppression file of checkstyle in the win10 (2986)
• [BugFix] [seatunnel-translation-base] Fix Source restore state NPE (2878)

[Docs]

• Add coding guide (2995)

[SeaTunnel Engine]

[Feature & Improve]

[Cluster Manager]

• Support Run SeaTunnel Engine in stand-alone.
• Support Run SeaTunnel Engine cluster.
• Do not rely on third-party services(zookeeper etc) to realize the master-worker architecture.
• Autonomous cluster (non centralized).
• Automatic discovery of cluster members.

[Core]

• Support submit Job to SeaTunnel Engine in local mode.
• Support submit Job to SeaTunnel Engine in cluster mode.
• Support Batch Job.
• Support Stream Job.
• Supports batch stream integration, and the batch stream integration feature of all SeaTunnel V2 Connectors can be guaranteed in SeaTunnel Engine.
• Support Distributed Snapshot algorithm Chandy Ramport algorithm and Two-phase Commit. Exactly-Once semantics based on these implementations.
• Support pipeline granularity job scheduling, Ensure that the job can be started under limited resources.
• Support pipeline granularity job restore.
• Sharing threads between tasks to achieve real-time synchronization of a large number of small datasets.

[Feature & Improve]

• Connector V2 API, Decoupling connectors from compute engines
  • [Translation] Support Flink 1.13.x
  • [Translation] Support Spark 2.4
  • [Connector-V2] [Fake] Support FakeSource (#1864)
  • [Connector-V2] [Console] Support ConsoleSink (#1864)
  • [Connector-V2] [ElasticSearch] Support ElasticSearchSink (#2330)
  • [Connector-V2] [ClickHouse] Support ClickHouse Source & Sink (#2051)
  • [Connector-V2] [JDBC] Support JDBC Source & Sink (#2048)
  • [Connector-V2] [JDBC] [Greenplum] Support Greenplum Source & Sink(#2429)
  • [Connector-V2] [JDBC] [DM] Support DaMengDB Source & Sink(#2377)
  • [Connector-V2] [File] Support Source & Sink for Local, HDFS & OSS File
  • [Connector-V2] [File] [FTP] Support FTP File Source & Sink (#2774)
  • [Connector-V2] [Hudi] Support Hudi Source (#2147)
  • [Connector-V2] [Icebreg] Support Icebreg Source (#2615)
  • [Connector-V2] [Kafka] Support Kafka Source (#1940)
  • [Connector-V2] [Kafka] Support Kafka Sink (#1952)
  • [Connector-V2] [Kudu] Support Kudu Source & Sink (#2254)
  • [Connector-V2] [MongoDB] Support MongoDB Source (#2596)
  • [Connector-V2] [MongoDB] Support MongoDB Sink (#2649)
  • [Connector-V2] [Neo4j] Support Neo4j Sink (#2434)
  • [Connector-V2] [Phoenix] Support Phoenix Source & Sink (#2499)
  • [Connector-V2] [Redis] Support Redis Source (#2569)
  • [Connector-V2] [Redis] Support Redis Sink (#2647)
  • [Connector-V2] [Socket] Support Socket Source (#1999)
  • [Connector-V2] [Socket] Support Socket Sink (#2549)
  • [Connector-V2] [HTTP] Support HTTP Source (#2012)
  • [Connector-V2] [HTTP] Support HTTP Sink (#2348)
  • [Connector-V2] [HTTP] [Wechat] Support Wechat Source Sink(#2412)
  • [Connector-V2] [Pulsar] Support Pulsar Source (#1984)
  • [Connector-V2] [Email] Support Email Sink (#2304)
  • [Connector-V2] [Sentry] Support Sentry Sink (#2244)
  • [Connector-V2] [DingTalk] Support DingTalk Sink (#2257)
  • [Connector-V2] [IotDB] Support IotDB Source (#2431)
  • [Connector-V2] [IotDB] Support IotDB Sink (#2407)
  • [Connector-V2] [Hive] Support Hive Source & Sink(#2708)
  • [Connector-V2] [Datahub] Support Datahub Sink(#2558)
• [Catalog] MySQL Catalog (#2042)
• [Format] JSON Format (#2014)
• [Spark] [ClickHouse] Support unauthorized ClickHouse (#2393)
• [Binary-Package] Add script to automatically download plugins (#2831)
• [License] Update binary license (#2798)
• [e2e] Improved e2e start sleep (#2677)
• [e2e] Container only copy required connector jars (#2675)
• [build] delete connectors*-dist modules (#2709)
• [build] Dependency management split (#2606)
• [build] The e2e module don't depend on the connector*-dist module (#2702)
• [build] Improved scope of maven-shade-plugin (#2665)
• [build] make sure flatten-maven-plugin runs after maven-shade-plugin (#2603)
• [Starter] Use the public CommandLine util class to parse the args (#2470)
• [Spark] [Redis] Self-Achieved Redis Proxy which is not support redis function of "info replication" (#2389)
• [Flink] [Transform] support multi split,and add custome split function name (#2268)
• [Test] Upgrade junit to 5.+ (#2305)

[Bugfix]

• [Starter] Ensure that output paths constructed from zip archive entries are validated to prevent writing files to unexpected locations (#2843)
• [Starter] Let the SparkCommandArgs do not split the variable value with comma (#2523)
• [Spark] fix the problem of calling the getData() method twice (#2764)
• [e2e] Fix path split exception in win10,not check file existed (#2715)

[Docs]

• [Kafka] Update Kafka.md (#2863)
• [JDBC] Fix inconsistency between document (#2776)
• [Flink-SQL] [ElasticSearch] Updated prepare section (#2634)
• [Contribution] add CheckStyle-IDEA Plugin introduction (#2535)
• [Contribution] Update new-license.md (#2494)

[Feature & Improvement]

[Connector][Flink][Fake] Supported BigInteger Type (#2118)

[Connector][Spark][TiDB] Refactored config parameters (#1983)

[Connector][Flink]add AssertSink connector (#2022)

[Connector][Spark][ClickHouse]Support Rsync to transfer clickhouse data file (#2074)

[Connector & e2e][Flink] add IT for Assert Sink in e2e module (#2036)

[Transform][Spark] data quality for null data rate (#1978)

[Transform][Spark] Add a module to set default value for null field #1958

[Chore]a more understandable code,and code warning will disappear #2005

[Spark] Use higher version of the libthrift dependency (#1994)

[Core][Starter] Change jar connector load logic (#2193)

[Core]Add plugin discovery module (#1881)

[BUG]

[Connector][Hudi] Source loads the data twice

[Connector][Doris]Fix the bug Unrecognized field "TwoPhaseCommit" after doris 0.15 (#2054)

[Connector][Jdbc]Fix the data output exception when accessing Hive using Spark JDBC #2085

[Connector][Jdbc]Fix JDBC data loss occurs when partition_column (partition mode) is set #2033

[Connector][Kafka]KafkaTableStream schema json parse #2168

[seatunnel-core] Failed to get APP_DIR path bug fixed (#2165)

[seatunnel-api-flink] Connectors dependencies repeat additions (#2207)

[seatunnel-core] Failed to get APP_DIR path bug fixed (#2165)

[seatunnel-core-flink] Updated FlinkRunMode enum to get the proper help message for run modes. (#2008)

[seatunnel-core-flink]fix same source and sink registerplugin librarycache error (#2015)

[Command]fix commandArgs -t(--check) conflict with flink deployment target (#2174)

[Core][Jackson]fix jackson type convert error (#2031)

[Core][Starter] When use cluster mode, but starter app root dir also should same as client mode. (#2141)

Docs

source socket connector docs update (#1995)

Add uuid, udf, replace transform to doc (#2016)

Update Flink engine version requirements (#2220)

Add Flink SQL module to website. (#2021)

[kubernetes] update seatunnel doc on kubernetes (#2035)

Dependency upgrade

Upgrade common-collecions4 to 4.4

Upgrade common-codec to 1.13

[Feature]

• Add Spark webhook source
• Support Flink application mode
• Split connector jar from core jar
• Add Replace transforms for Spark
• Add Uuid transform for Spark
• Support Flink dynamic configurations
• Flink JDBC source support Oracle database
• Add Flink connector Http
• Add Flink transform for register user define function
• Add Flink SQL Kafka, ElasticSearch connector

[Bugfix]

• Fixed ClickHouse sink data type convert error
• Fixed first execute Spark start shell can't run problem
• Fixed can not get config file when use Spark on yarn cluster mode
• Fixed Spark extraJavaOptions can't be empty
• Fixed the "plugins.tar.gz" decompression failure in Spark standalone cluster mode
• Fixed Clickhouse sink can not work correctly when use multiple hosts
• Fixed Flink sql conf parse exception
• Fixed Flink JDBC Mysql datatype mapping incomplete
• Fixed variables cannot be set in Flink mode
• Fixed SeaTunnel Flink engine cannot check source config

[Improvement]

• Update Jackson version to 2.12.6
• Add guide on how to Set Up SeaTunnel with Kubernetes
• Optimize some generic type code
• Add Flink SQL e2e module
• Flink JDBC connector add pre sql and post sql
• Use @AutoService to generate SPI file
• Support Flink FakeSourceStream to mock data
• Support Read Hive by Flink JDBC source
• ClickhouseFile support ReplicatedMergeTree
• Support use Hive sink to save table as ORCFileFormat
• Support Spark Redis sink custom expire time
• Add Spark JDBC isolationLevel config
• Use Jackson replace Fastjson