Dlink Versions Save

Feature: [Feature-1709][core] Support use FlinkSQL CTE to preview data [Feature-1774][flink] Support Apache Flink 1.17

Fix: [Fix-797][web] Fix no fragment parameters were passed when getting lineage [Fix-1731][gateway] Fix deploy flink k8s session error [Fix-1787][metadata] Fix error in automatic table creation using CDCSOURCE Mysql field unsigned type problem [Fix-1805][admin] Fix NPE occurred when publishing the job [Fix][cdcsource] Fix data sending and receiving issues with synchronizing multiple Sinks in CDCSOURCE [Fix][SqlSegment] Fix the issue of SQL configuration parsing failure in specific scenarios [Fix-1784][admin] Fix task is deleted but task instances and history are not deleted consecutively [Fix-1849][client] Fix CDCSOURCE kafkaSink transactionalIdPrefix NullPointerException [Fix-1847][metadata] Fix postgreSql get DDL script error [Fix][metadata] Fix generating MySQL DDL statements without quotes for normal default values

Optimization & Improve: [Optimization-1715][admin] Optimize job monitor reconnect [Optimization-1776][core,metadata] Fix typo [Optimization-1657][client] Optimize MysqlJsonDebeziumDeserializationSchema use shaded lib in flink 1.13 [Optimization][client] Support scan.incremental.snapshot.chunk.size [Optimization][client] Analyze lineage from the original relNode, not the optimized relNode, and add unit tests [Optimization-1851][flink] Upgrade all versions of Flink to the latest version

Contributors: @aiwenmo @AntChen27 @cygnusdark @HamaWhiteGG @LiuHao0606

Feature: [Feature-1593] Add CDCSOURCE support sink catalog [Feature-1608] Support get task lineage by taskid with openapi [Feature-1641] Add Flink row level permissions [Feature-1647] Yarn session support applicationId and resource manager config [Feature-1547] Flink Jar support alert

Fix: [Fix-1615] Fix submit task error but return succeed when using dolphinscheduler [Fix-1624] Fix lineage api in data explore module returns empty result [Fix-1652] Fix the task using the FlinkSQLEnv cannot obtain lineage [Fix-1662] Fix OpenApi notLoginException [Fix] Fix wrong Python type [Fix-1672] Fix failed to get metadata due to jdbc session timeout in CDCSOURCE [Fix-1638] Fix dlink web could not correct display Boolean type data [Fix-1686] Fix ZipFile got wrong in K8s native app mode

Optimization & Improve: [Optimization-1631] Optimize auto.sh and remove useless files [Optimization-1666] Add job reconnecting status and optimize job monitor [Optimization-1682] Optimize setting named tinyInt1isBit for mysql metadata

Contributors: @aiwenmo @liaotian1005

Fix: [Fix-1494] Fix tinyint(1) convet error [Fix-1575] Fix unknown column 'tenant_id' in 'field list' [Fix] Fix i18n [Fix] Fix UDF function case insensitive [Fix] Fix the bug that the schema name contains middleLine [Fix-1556] Fix job submission succeeded but failed [Fix-1503] Fix use yarn application mode to submit job reported null pointer exception [Fix-1563] Fix NPE caused by querying data [Fix] Fix add k8s ClusterConfiguration npe [Fix] Fix parse type error when use bigint types

Optimization & Improve: [Optimization-1548] Optimize datasource registration should can save config before validate [Optimization] Optimize clear console and add clear console at right click [Optimization] Update org.apache.httpcomponents:httpclient 4.4.1 to 4.5.13

Contributors: @aiwenmo @leechor @zackyoungh @zhoumengyks @zhu-mingye

Feature: Supports Apache Flink 1.16.0 Add build java udf jar Support Flink session mode to automatically load udf jars Support Flink per-job mode to automatically load udf jars Support Flink application mode to automatically load udf jars Support python udf online development Support scala udf online development Support custom k8sapp submit at studio Add file upload in flinkjar task Add Column Lineage By Logic Plan in Flink all version Flink JDBC support data filter Flink JDBC scan partition support datetime Add multi tenant management Add select tenant at login Add dolphinScheduler auto create task Add system log console Add execution progress console Add Flink UDF template Add Presto data source Add the function of deleting job tree directory level Add new sink connector 'datastream-doris-ext' for CDCSOURCE to support metadata writing DataStudio metadata add refesh button Add database copying Add frontend internationalization Add backend internationalization Add K8S auto deploy application Add local environment with web UI Add FlinkSQL built-in date global variable Add Kerberos Verification on session model Add CDCSOURCE supports Doris schema evolution Add user can change password Add kill cluster Add deploy session cluster by cluster configuration Add jmx monitoring Add clear console

Fix: Resolve the exception when starting from the specified savepoint Fix StarRocks databases display Fix query exceptions caused by the system's failure to automatically clean up the selected table information when switching data sources Fix Exception in obtaining SQLDDL of view Fix invalid oracle validationQuery Fix failed to get schema of PG database metadata Fix kafka sink properties not enable Fix jobConfig useAutoCancel parameter pass wrong value Fix job monitoring bugs caused by multi tenancy Fix the deletion of a cluster instance causes existing tasks to fail to stop Fix application mode missing database variable Fix continuous click task item will open multiply tabs problem Fix cdcsource kafka product transactionalIdPrefix disable Fix cluster display nothing when alias is not set Fix task version query error Fix The status of per-job and application always is unknown when the job completes Fix guide page link error Fix open API does not have tenant Fix switch editor tab doesn't work

Optimization & Improve: Add parameter configuration of MySQL CDC Optimizate datastream kafka-json and datastream starrocks Support metadata cache to memory or redis Add uuid after doris label prefix Optimize tenant selection Change resource center to authentication center Add spotless plugin Optimize SQL files and differentiate versions by directory Improve the automatic creation of MySQL tables Improve postgres metadata information Improve the generation of postgres table building statements Optimizate Flink Oracle Connector Optimizate maven assembly and profile Compatible java 11 Remove duplicated init of datasource Upgrade mysql-connector-java to 8.0.28 Upgrade Flink 1.14.5 to 1.14.6 Upgrade Guava and Lombok Version Upgrade jackson and sa-token version Optimize lineage to support watermark and udf and LOCALTIMESTAMP and cep and ROW_NUMBER The authentication center is not visible under non-admin users

Document： Optimizate readme.md Update website Optimize deploy doc Add flink metrics monitor and optimize deploy document

Contributors: @admxj @aiwenmo @billy-xing @boolean-dev @chengchuen @czy868 @dzygcc @Forus0322 @gujincheng @HamaWhiteGG @hxp0618 @ikiler @leechor @lewnn @jinyanhui2008 @nylqd @ren-jq101 @rookiegao @siriume @tgluon @wellCh4n @wfmh @wmtbnbo @wuzhenhua01 @zackyoungh @zhongjingq @zhu-mingye @ziqiang-wang @zq0757 @Zzih @zzzzzzzs

Feature： [Feature-775][admin] Add tenant implementation [Feature-789][admin,web] One click online and offline operation [Feature-812][admin,web] Add FragmentVariable manager && resourcecenter page [Feature] Added NameSpaceForm PasswordForm [Feature-823][web] Render multi tenant forms on login [Feature-823][web] Multi tenant front end implementation [Feature-868][common] Added github workflow to check the checkstyle, test and build of each PR [Feature-861][metadata] Add alibaba druid connection pooling to fix jdbc multi-connection problem [Feature-890][admin,web] Realization of user empowerment role function [Feature-907][pom] Change Flink base version into 1.14 [Feature-905][admin,web] Implementation of global variable management [Feature][client] Add SqlServer CDCSOURCE [Feature-915][admin,core] Add global variables takes effect in flinksql [Feature-923][*] Build column lineage base on flink logical plan [Feature][client] Add posgresql CDCSOURCE [Feature][test] Modify checkstyle to be a required item [Feature] Add swagger api doc [Feature] CDCSOURCE supports multi-sink [feature][admin] File upload [Feature-987][admin] ClusterConfig and jar add upload file [Feature-989][metadata] Add StarRocks datasource [Feature-946][alert] Alarm after task monitoring retry [Feature][web] Add data development task information log details button

Fix: [Fix][admin] Modify the problem of repeated judgment of task monitoring code [Fix] Fix mail alert params bug [Fix-804][admin] GetJobInfoDetail and refreshJobInfoDetail maybe return a error result [Fix-818][connectors] Fix dlink-connector-doris-1.13 when the data is flushed with an exception and no new data is entered subsequently [Fix-833][client] Error with SQLSinkBuilder.buildRow [Fix] Fix execute error by flink 1.14.4 [Fix] Fix cluster submission taskId is empty [Fix] Fix yarn per job can't release resources [Fix] Fix multi tenant add role and delete role [Fix] Fix dlink-conector-pulsar-1.14 can't find SubscriptionType [Fix] Fix the jackjson problem in flink1.14 savepoint [Fix-803][client] Fix TypeConvert-ColumnType Enumeration usage error [Fix] The full database sync kafkasinkbuilder does not implement serialization and cause an error [Fix-840][web] fix registration document type filter condition error [Fix] Fix yarn perjob/application and k8s application cluster configuration [Fix] Fix k8s application submit error and add Maximum waiting time [Fix] Fix the banner of dlink admin app [Fix][function] Fix udf and udtaf in Flink 1.14 [Fix][app] Fix yarn application task separator error [Fix][admin] Failed to save job after repairing rename job [Fix][web] No content when repairing the second pop-up of the submitted history

Optimize: [Optimization-764][dlink-web,docs] Optimization dlink-web,docs [Optimization-780][admin] Optimization The task version was not deleted after deleting the task [Optimization-781][web] Optimization Overflow in job list tree after importing file [Optimization-801][web] Optimization StudioConsole's StudioProcess [Optimization-773][dlink-client] Optmizatiion cdcsource filter to process [Optimization-809][Common] Added ignore of logs folder in git ignore file. [Optimization-810][Common] Added Maven Wrapper [Optimization-816][Common] Fix Chinese README link error and add English README [Optimization] Optimization of multi tenant && Optimize form rendering [Optimization] Remove sensitive info for more api [Optimization][admin] Optimized for multi-tenant [Optimization-819][client] CDCSOURCE with timestamp and timezone [Optimization-849][client,executor] Replace sql separator and change default sql separator into ;\n [Optimization][Style] Optimization code style import order [Optimization][Git] Added .DS_Store git file ignore [Optimization] Optimization multi tenant of delete roles && optimization multi tenant web render [Optimization] Optimize user associated character rendering [Optimization][Style] Added dlink-admin module code style [Optimization][Style] Added dlink-alert module code style [Optimization][Style] Added dlink-common module code style [Optimization][Style] Added dlink-catalog module code style [Optimization][Style] Added dlink-client module code style [Optimization][Style] Added dlink-app module code style [Optimization][metadata] Optimized connection pooling and connection creation [Optimization][Style] Added dlink-connectors module code style [Optimization][Style] Added dlink-core module code style [Optimization][Style] Added dlink-daemon module code style [Optimization][Style] Added dlink-executor module code style [Optimization][Style] Added dlink-function and dlink-gateway module code style [Optimization][Style] Added dlink-metadata module code style [Optimization][license] Add a license description to the pom file [Optimization-932][pom] Optimizate package and auto.sh by loading classpaths [Optimization] dlink-client-hadoop add ServicesResourceTransformer [Optimization-943][pom] Optimizate config and static dir packaging [Optimization][style] Configure global checkstyle validation [Optimization][client] Add sqlserver date type convert [Optimization][metadata] Optimizate postgresql schema_name query [Optimization-981][metadata] Doris support more syntax [Optimization-983][client] Optimizate Doris datastream sink [Optimization][web] Optimizate Some problems in front-end and some tips [Optimization-882][web] Collapse all directories by default on datastudio's directory panel [Optimization-881][datastudio] Optimizate FlinkSql explain exception message [Optimization-1014][client] Optimizate Doris sink and type convert and upgrade flink to 1.15.2 [Optimization][flink] Upgrade Flink 1.15 to 1.15.2 [Optimization][metadata] Optimize SqlServer field type query

Document: [Document][docs] Migrate dlink docs from dinky-website to docs [Document] Dlink add Flink1.15 docs [Document-769][docs] The whole database synchronization document repair [Document-766][docs] Add import and export job docs [Document-793][doc] Optimization some docs [Document-835][doc] Update the home page and basic information of the document [Document-832][doc] Add Practice Sharing of FlinkSQL Extending Redis [Document][docs] Optimizing Deployment Documents

Contributor： @aiwenmo @byd-android-2017 @chrofram @complone @dylenWu @dzygcc @gaogao110 @hxp0618 @hzymarine @ikiler @leo65535 @lnnlab @JanCong @jinyanhui2008 @mengyejiang @mydq @rafaelxie @sunyanqing01 @tgluon @Toms1999 @walkhan @wangzhonggui @XiaoF-Liu @zackyoungh @zhu-mingye @zhuangchong @zhujunjieit @ziqiang-wang

Feature： [Feature-685][web,admin] add job historyversion list of DevOps [Feature-692][web] Add history version comparison in data development [Feature][catalog] Add Flink MySql Catalog [Feature-704][catalog,admin] Add default mysql catalog in FlinkSQLEnv [Feature][connector] Added version 1.13 Doris connection is hidden by _DORIS_DELETE [Feature][connector] Add dlink-connector-pulsar [Feature][web,admin] Select checkpoint restart [Future][flink] Update flink1.15 to flink1.15.1 [Feature-728][admin,web] Add meta store feature [Feature-733][admin,web,client] Add Flink meta store info and column details [Feature-738][*] Add and update Licenses [Feature-750][admin,web] Add import and export tasks json

Fix: [Fix][connctor] flink-connector-phoenix and Update PhoenixDynamicTableFactory [Fix][admin] Fix job instance cause OOM [Fix][bug] Update flink version and fix Flink and CDC version compatibility bug [Fix-696][web] Fix the problem of time-consuming parsing errors of task instances [Fix-709] [catalog] Fix catalog SPI bug and sql bug [Fix-719][web] Fix checkpoint has error of devops && add savepoints info [Fix-714][client] Catch translateToPlan exception in SQLSinkBuilder [Fix-736][client] An exception occurred during the packaging process [Fix-738][admin] Fix CopyrightUtil [Fix-741][app] Fix application mode submit failed [Fix-745][admin] Fix alert instance delete bug [Fix][admin] Allow circular references config [Fix-750][admin] Delete unreferenced classes [Fix][admin] Fix jobhistory field null

Optimize: [Improvement][jobplan] Prompt returned when failed to obtain the optimized flinksql execution graph [Optimization][admin] Move sensitive info(password) from api [Optimization-713][dlink-admin] Fix the problem of incorrect display information of deleted user login [Optimization-719][web] Optimization checkpoints page

Contributors： @a279780399 @aiwenmo @Arnu- @darren-da @dzygcc @Forus0322 @gaogao110 @JPengCheng @mydq @syyangs799 @wmtbnbo @zhu-mingye

Feature: [Feature][connector]Add flink connecotr phoenix 1.14 [Feature-609][metadata]Supports generating FlinkSQL with null and not null through metadata [Feature-635][admin]Add job instance info api [Feature-644][admin,web]Add Flink snapshot info and JobManager config in DevOps [Feature-649][admin,web]Add TaskManager info in DevOps [Feature-654][web]Add task info tab [Feature-649][admin,web]Add TaskManager table and form in DevOps [Feature-662][admin,web]Add task version history [Feature-661][web]Add job checkpoint history [Feature-666][client]Capture column type conversion exception details in CDCSOURCE [Feature-668][web]Add task manager info [Feature-663][admin,web]Add task copy feature Fix: [Fix-577][web]Fix datastudio datasource alias is "" [Fix-574][web]Fix datastudio tabs close_other can not close first tab [Fix-576][admin]Fix the exception caused by metadata switching [Fix][gateway]Fix flinkLibPath in K8S configuration is null and cluster-id in test k8s-cluster-config is null [Fix][common]Fix semicolon exception at the end of FlinkSql [Fix][core]Fix K8S cluster configuration can't get custom configuration [Fix-596][admin]Fix NPE on running yarn per-job app [Fix-603][admin]Fix task information will be cleared when refreshing tasks that have lost connection [Fix-140][core]Fix 'table.local-time-zone' parameter is invalid [Fix-607][dlink-alert-email]Fix ClassNotFoundException: javax.mail.Address [Fix-621][core]Fix can't explain 'show datatbases' [Fix][core]Fix setParentId method space determination error [Fix-608][client]Fix can't create a flink table when table name is a sql reserved word in CDCSOURCE [Fix-629][core]Fix lineage of the same field name cannot be correctly resolved [Fix][web]Fix cluster configuration page bug [Fix-638][alert]Fix the email alarm notification cannot customize the nickname [Fix][connector]Fix dlink-connector-phoenix-1.14 build error [Fix-670][metadata]Fix Oracle column nullable error [Fix-674][client]Support MySQL varbinary and binary in CDCSOURCE Optimization & Improve: [Optimization-575][web]Optimize tree search results highlight & & selected backgroundColor [Optimization-605][web]Optimize the alias is empty in several places [Optimization][client]Optimize explainSqlRecord [Optimization-627][web]Optimization cluster instance page [Optimization-633][web]Optimization lineage refresh [Optimization-641][admin]Optimize job instance api [Optimization-640][alert,web]Optimization all Alert of sendMsg

Feature: [Feature-506][client] The CDCSOURCE table parameter supports line feed and column sort by primary key [Feature-518][client] CDCSOURCE add log [Feature-525][alert,web] Add @mobile mode of dingTalk Alert [Feature-254][web] Add StreamGraph JSON Export function [Feature-534][admin,web] Add task open API page [Feature-356][web] Add datastudio help page [Feature-318][core] Add column lineage from db sql [Feature-545][admin] Add Task Pool to solve frequent database writes [Feature-552][web] Hide export StreamGraphPlan button when not FlinkSQL task [Feature-558][web] add delete button of database Manage [Feature-568][client] Add CDCSOURCE jdbc properties and upgrade flink cdc version

Fix: [Fix-491][admin,web] Fix refresh the status page with abnormal jitter [Fix-499][connector] Flink oracle connector can't cast CLOB to String [Fix-494][web] Fix the savepoint information does not change with the task information [Fix-503][gateway,client] Compatibility of different versions of the interface ClusterClient [Fix-509][metabase] Fix MySqlTypeConvert precision and scale is null bug [Fix-503][client] Remove initFunctions [Fix-518][client] Fix CDCSOURCE decimal bug [Fix-520][core] Fix get JobPlanInfo failed [Fix-522][client] Fix CDCSOURCE OracleCDC number can't be cast to Long [Fix-527][web] Fix the wechat alarm instance send testMsg error [Fix-529][web] Fix the task cannot be saved after switching tasks [Fix-541][web] Fix the problem that the datasource && metadata does not set alias [Fix-542][web] Fix the name of tab can't be modified when modify task name [Fix-556][adimin,gateway] Fix flinkLibPath in K8S configuration is null [Fix-571] [client] Fix CDCSOURCE String can't be cast to Timestamp

Optimization & Improve: [Optimization-504][doc] Optimization init sql [Optimization-550][assembly] Optimization package [Optimization-562][web] Remove preset-ui [Optimization-564][metadata] Optimization MySqlTypeConvert

Feature: [Feature-422][] CDCSource sync kafka topics [Feature-429][] OracleCDCSource sync kafka topics [Feature-435][client,executor] CDCSource sync doris [Feature-442][client,executor] CDCSource sync hudi [Feature-445][client] CDCSource sync add sink table-name rule [Feature-447][client] CDCSource sync sql [Feature-451][client] CDCSource sync field type convertion in Flink1.13 [Feature-461][client] CDCSource sync add sink table-name RegExp [Feature-469][client] Add MysqlCDCSource sync extended configuration [Feature-477][client] CDCSOURCE add pkList [Feature-275][client] Add Flink client 1.15 [Feature-488][*] Release v0.6.3

Fix: [Fix-427][admin] Task status error when task has stoped [Fix-425][metadata] Fix Oracle Unsupported character set [Fix-426][metadata] Fix Clickhouse metadata does not display [Fix-387][web] Metadata switching error [Fix-442][client,executor] Add Hudi SOURCE_AVRO_SCHEMA [Fix-454][metadata] Assign to DriverConfig's name [Fix-424][common] Modify the default value of the sqlSeparator to ';\r\n|;\n' [Fix][app] Batch execution with yarn-app mode using the useBatchModel parameter [Fix-457][web] Fix bug of modifying job name [Fix-472] [client] Mysqlcdc whole database sync Hudi error [Fix-479][docs] Fix some inaccessible documentation links [Fix-484] [core] Fix to job plan info was executed twice

Optimization & Improve: [Improve-434][admin] The release of Stream [Optimization-439][client] Optimize CDCSource sync doris [Improvement-456][dlink-doc] Fix start and stop conflicts when deploy more instances in same machine [Optimization-459][alert-wechat,web] Optimization wechat Webhook sendMsg title [Improvement-456][dlink-doc] Fix auto.sh LF and disable environment variables

[Feature-312][alert] Add FeiShu alert type [Feature-309][alert] Add Email alert type [Feature-339][admin,web] Add alarm msg sending test [Feature][root] Add docker support [Feature-355][admin,executor] Restore the job from a savepoint path in remote mode [Feature-377][] Modify version to 0.6.2-SNAPSHOT [Feature-389][client,executor] Add OracleCDCSourceMerge [Feature-401][] Modify version to 0.6.2

[Fix-239][core] Use "==" to judge whether the values of referenced variables are equal. Replace with the equals function [Fix-310][gateway] Configure hadoop_conf for Flink [Fix-347][alert] Fix to the latest configuration take effect during the alert test [fix-350][alert] Feishu alert when choose '@all' send msg failed bug [Fix-358][pom] Fix to install [fix-363][metadata-hive] Fix hive getSqlGeneration Exception [Fix-370][executor] Avoid replacement failure due to special characters [Fix-98][admin,web] Fix to show exception [Fix-127][core,executor] Add executeAsync datastream job [Fix-352][admin] Fix to modify cluster configuration [Fix-395][connector] Oracle connector oracle.sql.TIMESTAMP converte java.sql.Timestamp occur cast exception [Fix-402][admin] MybatisPlus PO boolean default false [fix-407][common,function] Use "==" to judge whether the values of referenced variables are equal. [Fix-405][common] Fix the default value of SystemConfiguration sqlSeparator to ';\r\n' [Fix-393][core] Program in loop when offline task failed [Fix-418][admin,core,sql] Fix submit task parameter exception

[Optimization-367][core] Optimization explainSql return all error [Optimization-371][metadata-hive] Optimization metadata-hive pom [Optimization-373][admin] Add httpclient to admin jar [Optimization-382][alert,metadata] Optimization SPI [Optimization-384][client] Optimization CDCSourceMerge [Optimization-397][daemon] SPI Optimization [Optimization-406][web] FlinkWebUI button Optimization && add flinkwebui button for clusertInstance Manager