Exchangis is a lightweight,highly extensible data exchange platform that supports data transmission between structured and unstructured heterogeneous data sources
Exchangis is a new data exchange tool jointly developed by WeDataSphere big data platform and community users to support the synchronization of structured and unstructured data transfer between heterogeneous data sources. Exchangis also abstracts a unified set of data source and synchronous job definition plugins, allowing users to quickly access new data sources and use them on the page with a simple configuration in the database. Based on the plugin framework design and computing middleware Linkis, Exchangis can quickly integrate and connect with the integrated data synchronization engine of Linkis, and convert the synchronization jobs of Exchangis into the data synchronization jobs of Linkis data synchronization engine. With the connectivity, reuse, and simplification capabilities of Linkis computing middleware, Exchange is born with financial level data synchronization capabilities of high concurrency, high availability, multi-tenant isolation, and resource management.
Github： https://github.com/WeBankFinTech/Exchangis
Compared with Exchangis 1.0.0, Exchangis 1.1.2 added DataX engine and supported Oracle datasource. At the same time, some common problems in project module and task module were fixed, which further improved the stability of the system. Exchangis1.1.2 provides the following new features： 1, [Job engine plug-in module] added DATAX engine to support a variety of data derivative links.
3, [Datasource module] Added Oracle data source, supported bidirectional derivatives with MySQL, Hive, MongoDB, and imported to ElasticSearch.
[Datasource module] data source connection sensitive information encryption, support the owner of the connection information encryption.
[Job configuration management module] Field rule mapping management function supports the management of field mapping rules and field matching strategies of different engines and different data sources, and renders at the front end when loading jobs.
6, [Job configuration management module] background storage engine configuration and mapping function, support background storage engine derivative configuration and task field mapping function list storage.
[Exchangis Plugin Management Module] Add the DataX engine.
[Exchangis Project Module] Adds the project permission synchronization function to synchronize DSS project permissions.
[Exchangis Datasource Server Module] Add an Oracle data source.
[Exchangis Datasource Server Module] Add encryption of sensitive data source connection information.
[Exchangis Job Builder Module] Add field rule mapping management.
[Exchangis Job Builder Module] Add the background storage engine configuration and mapping function list.
[Exchangis Project Module] Sets Project Name to uneditable when modifying a project on the front-end page.
[Exchangis Project Module] Checks whether tasks exist when deleting an item. If yes, the item cannot be deleted.
[Exchangis Job Execution Module] Optimizes subtask execution schedule statistics during task execution to ensure consistency of schedule data.
[DSS Dependency] Upgrade the DataSphereStudio (DSS) version to 1.1.2.
[Linkis Dependency] Upgraded Linkis version to 1.4.0, addressing some issues with concurrent execution of the engine.
[Exchangis Job Info Module] The Execute User value is automatically displayed during task editing.
[Exchangis Job Builder Module] Fixed an issue where the name of a subtask is too long to run a task for a long time.
[Exchangis Job Log Query Module] Fixed system exceptions when viewing non-current day logs.
[Exchangis Job Log Query Module] Fixed an error message displayed on the log TAB page after a log query is abnormal.
Exchange 1.1.2 can't be released without the contributors of the Exchange community. Thank you to all the contributors!
Exchangis 1.1.2 Installation package: Click Me to download
DSS1.1.2&Linkis1.4.0 One-click installation package: Click Me to download
Exchangis-1.0.0 version contains all Project Exchangis-1.0.0.
This release is the official version of Exchangis1.0.0, a data synchronization task tool. Compared with version 1.0.0-rc1, Appconn has been modified to adapt to DSS1.1.0, so as to implement data synchronization tasks in the form of workflow.
This version mainly includes the following contents:
1、Make modification of Appconn to adapt to DSS1.1.0;
2、The workflow node replication function is added, which can roll back the workflow node to other version history in DSS.
EJS: Exchangis Job Server EJB:Exchangis Job Builder EP:Exchangis Project EDS:Exchangis Datasource Server EXAPP:Exchangis Appconn
[EJS] Exchangis-315 Fixed the problem that field mapping could not be saved successfully.
[EXAPP] Exchangis-316 Fixed sqoop workflow node deletion failure.
[EXAPP] Exchangis-317 Fixed the problem of service error when forwarding POST request in Appconn, and the problem that labels label was not obtained correctly.
The release of Exchangis-1.0.0 is inseparable from the contributors of Exchangis community. Thanks to all community contributors! Including but not limited to the following Contributors：wushengyeyouya、Dlimeng、Davidhua1996、mingfengwang、yuxin-No1、ryanqin01、lucaszhu2zgf、FinalTarget、Liveipool、gjy1043、jefftlin.
Tencent Cloud:
Exchangis-1.1.0 Compiled package (.tar.gz) sqoop.zip exchangis-appconn.zip dist.zip
The Exchangis-1.0.0-RC release includes all Project Exchangis-1.0.0-RC.
This release is a new RC version of the Exchangis data synchronization task platform, which supports data synchronization between different types of data sources. The platform splits the data exchange process, abstracts concepts such as data sources, data exchange tasks, and task scheduling, and strengthens the linkage with linkis and DSS. It is a reconstructed version after version 0.X.
This version mainly includes the following contents:
[Submit data synchronization job] Refactoring data synchronization job execution
[Sub-task running status] View their various indicators information for the running or completed sub-jobs, so as to monitor the tasks in real time.
[Get real-time log] Data synchronization is in progress/finished to display real-time logs of jobs and tasks, and troubleshoot problems accurately and quickly
[Data source management module] Perform a series of management operations on data synchronization data sources
[Synchronization History Module] Manage the historical information of the synchronization data tasks that have been completed or are being executed.
[Project Management Module] Configure and manage data synchronization projects and tasks
[Data synchronization job building module] Build the information of data synchronization job configuration and form a form that can be submitted to linkis
Exchangis1.0 Appconn is connected to DSS, and supports the execution of data synchronization tasks in the form of workflow
EJS: Exchangis Job Server EJB:Exchangis Job Builder EP:Exchangis Project EDS:Exchangis Datasource Server EXAPP:Exchangis Appconn
[Exchangis-227] [EJS] Solve the problem that the job info service does not fit due to changes in the library table.
[Exchangis-185] [EJS-Execution] Avoid deadlock during netscape call in map calculation method: adjust scheduling task selection rules.
[Exchangeis-158] [EJS-Execution] Fix bugs updating progress, status and metrics.
[Exchangeis-165] [EDS] Implementation of client-side RPC and modification of namespaces for restful classes.
[Exchangis-292] [EJS] Fix the issue that hive's run_date time partition system variable fails to get.
[Exchangis-293] [EJS-Execution] Fix the problem that sqoop workflow node still executes in the background after canceling the task.
[Exchangis-282] [EJS-Execution] Fix the problem that the data source database table is changed after copying the subtask, causing the field mapping of the atomic task to be overwritten.
[Exchangis-140] [EJS-Execution] Fix the problem that submitting tasks to wtss for scheduling proxy users fails.
[Exchangis-241] [EJS] Fix missing subtask log.
[Exchangis-221] [EJS] Fix duplicate request issue when viewing subtasks.
[Exchangis-204] [EJB] When selecting the data source library table in the task configuration, add a search box for quick search.
[Exchangis-210] [EJS] Fix the problem that after clicking the execution ID to check the progress of the task, it is closed, and it switches to another page in time, and still pulls the progress of the task non-stop.
The release of EXCHANGIS 1.0.0-RC is inseparable from the contributors of the EXCHANGIS community, thanks to all the community contributors! Including but not limited to the following Contributors：wushengyeyouya、Dlimeng、Davidhua1996、mingfengwang、peacewong、Casion、xiaojie19852006、yuxin-No1、ryanqin01、lucaszhu2zgf、FinalTarget、Liveipool、gjy1043、