WeBankFinTech Qualitis Versions Save

Qualitis is a one-stop data quality management platform that supports quality verification, notification, and management for various datasource. It is used to solve various data quality problems caused by data processing. https://github.com/WeBankFinTech/Qualitis

release-1.0.0

4 months ago

功能新增:

  1. 新增规则执行参数模块化配置,支持动态引擎配置,执行变量配置,指标参数配置,告警参数配置等
  2. 新增数据源管理,维护数据源连接信息进行关系型数据质量校验
  3. 新增实例配置,是多实例服务任务刷新负载均衡
  4. 新增租户管理,支持多租户校验资源隔离

功能优化:

  1. UI 框架升级,界面重构
  2. 支持对项目列表的高级筛选
  3. 支持项目详情下对规则列表的高级筛选
  4. 支持编辑锁,防止多人同时编辑相同规则
  5. 支持配置规则执行参数进行批量配置
  6. 支持规则的批量启用和禁用
  7. 优化指标历史值趋势查看
  8. 规则模板配置模块化重构
  9. 支持对规则模板的高级筛选
  10. 优化字段一致性内置校验模板
  11. 优化行数据一致性内置校验模板
  12. 优化规则模板、数据源管理及指标管理的权限配置
  13. 优化任务日志详情的分页展示,提升页面响应速度
  14. 优化任务状态详情的分页展示,支持点击跳转规则详情
  15. 优化部门科室管理,支持自定义部门科室录入

New features:

  1. Added modular configuration of rule execution parameters, supporting dynamic engine configuration, execution variable configuration, indicator parameter configuration, alarm parameter configuration, etc.
  2. Add new data source management, maintain data source connection information, and perform relational data quality verification
  3. New instance configuration is added to refresh load balancing for multi-instance service tasks.
  4. Added new tenant management to support multi-tenant verification resource isolation

Function optimization:

  1. UI framework upgrade, interface reconstruction
  2. Supports advanced filtering of project lists
  3. Supports advanced filtering of rule lists under project details
  4. Supports edit lock to prevent multiple people from editing the same rules at the same time
  5. Supports configuring rule execution parameters for batch configuration
  6. Support batch enabling and disabling of rules
  7. View historical value trends of optimization indicators
  8. Modular reconstruction of rule template configuration
  9. Supports advanced filtering of rule templates
  10. Optimized field consistency built-in verification template
  11. Optimize row data consistency built-in verification template
  12. Optimize the permission configuration of rule templates, data source management and indicator management
  13. Optimize the paging display of task log details and improve page response speed
  14. Optimize the paging display of task status details and support click-to-jump rule details
  15. Optimize department management and support custom department entry

release-0.9.2

1 year ago
  1. 适配 DataSphere Studio 1.1.0 版本
  2. 适配 Apache Linkis 1.1.1 版本
  3. 首次安装,缺失依赖包请参考 release-0.9.1
  4. 工作流安装说明参考文档

BUG 修复

  1. 模板异常数据保存开关配置实时生效
  2. 系统变量 run_date - n,run_date_std - n 默认为执行日期前 n + 1 天
  3. 脱敏 UI 源码中的信息
  4. 修改部分文档错误内容

  1. Adapt to DataSphere Studio 1.1.0 version
  2. Adapt to Apache Linkis 1.1.1 version
  3. For the first installation, please refer to release-0.9.1 for missing dependencies
  4. Workflow Installation Instructions Reference Documentation

BUG fix

  1. The template exception data saving switch configuration takes effect in real time
  2. System variables run_date - n, run_date_std - n default to n + 1 days before the execution date
  3. Desensitize the information in the UI source code
  4. Modify some document errors

release-0.9.1

2 years ago
  1. 补充前端源码,位于根目录下的ui;
  2. 补充dss-qualitis-appconn源码,位于根目录下的appconn,在其resources目录下含有init.sql;
  3. 修复update_0.8.0-0.9.0.sql,支持基于release-0.8.0部署的版本数据库升级,位于web\app\src\main\resources\database\update;
  4. 提供org.springframework.boot:spring-boot-starter-data-jpa:2.0.4.RELEASE的依赖压缩包forgerock.zip,在本地.m2\repository\org解压,解决代码仓库拉取不到的问题。

  1. Supplement front-end source code, ui located in the root directory;
  2. Supplement dss-qualitis-appconn source code, which is located in appconn in the root directory, contains init.sql in its resources directory;
  3. Repair update_0.8.0-0.9.0.sql, support version database upgrade based on release-0.8.0 deployment, located in web\app\src\main\resources\database\update;
  4. Provide the dependency compressed package forgerock.zip of org.springframework.boot:spring-boot-starter-data-jpa:2.0.4.RELEASE, and decompress it locally in .m2\repository\org to solve the problems that cannot be pulled from the code repository question.

release-0.9.0

2 years ago

注:请直接使用最新的init.sql进行新版本部署,暂不支持0.8.0升级到0.9.0

新增关系型数据的质量校验 (需Linkis 1.0以上版本提供Datasource服务);

新增数据源管理 (需Linkis 1.0以上版本提供Datasource服务);

新增规则模板权限管理,系统管理员、部门管理员、普通用户可以新增权限范围内的模板;

新增数据质量业务指标管理功能;

新增多维度指标校验功能,支持自定义SQL校验;

支持库级数据全量一致性对比优化;

新增引擎参数管理 (需接入Linkis 1.0 以上版本),支持多 Linkis 集群切换;

新增任务执行参数配置管理;

支持数据质量分析统计结果导出到指定的存储目录;

优化规则管理-规则查询实时统计库表规则部署数量情况;

优化任务提交前数据源校验;

适配 DataSphere Studio 1.x,Appjoint 升级为 Appconn 并实现三级规范;

适配 Apache Linkis (Incubating) 1.x 版本。


Note: Please directly use the latest init.sql to deploy the new version. It is not supported to upgrade from 0.8.0 to 0.9.0.

Added the quality check of relational data (Linkis 1.0 or above is required to provide Datasource service);

Added data source management (requires Linkis 1.0 or above to provide Datasource service);

Added rule template authority management, system administrators, department administrators, and ordinary users can add templates within the scope of authority;

Added data quality business indicator management function;

Added multi-dimensional index verification function to support custom SQL verification;

Support database-level data consistency comparison and optimization;

Added engine parameter management (requires access to Linkis 1.0 or above), supports multi-Linkis cluster switching;

Added task execution parameter configuration management;

Support data quality analysis statistical results to be exported to the specified storage directory;

Optimize rule management - rule query real-time statistics on the number of rules deployment in database tables;

Optimize data source verification before task submission;

Adapted to DataSphere Studio 1.x, Appjoint was upgraded to Appconn and implemented the third-level specification;

Adapt to Apache Linkis (Incubating) 1.x version.

release-0.8.0

3 years ago

release-0.8.0

  1. 重构“规则查询”界面的前后台逻辑,规则查询结果展示由项目信息变成表信息,并关联具体的字段;
  2. 新增规则模板的管理功能,除系统初始化的默认模板可新增自定义的规则模板;
  3. 与最新开源的DataSphereStudio 0.9.0完成对接,主要是任务执行请求中任务阻断和发布中的导入导出功能;
  4. 丰富用户手册。

  1. Refactor the front and back logic of the "Rule Query" interface, and display the results of the rule query from project information to table information, and associate specific fields;
  2. Added the management function of rule templates. In addition to the default template initialized by the system, custom rule templates can be added;
  3. Complete the docking with the latest open source DataSphereStudio 0.9.0, Mainly the import and export functions of task blocking and publishing in task execution requests;
  4. Rich user manual.

release-0.7.0

4 years ago
  1. 删除HiveServer2,HiveMetastore以及Hadoop相关依赖。
  2. 删除异常数据存储数据库自动创建的机制
  3. 新增异常数据存储数据库配置
  4. 删除无效配置
  5. 删除前端编译后代码,换成压缩包
  6. 优化文档

  1. Delete HiveServer2, HiveMetastore and Hadoop related dependencies.
  2. Remove the mechanism of automatic creation of abnormal data storage database
  3. Added abnormal data storage database configuration
  4. Delete invalid configuration
  5. Delete the front-end compiled code and replace it with a compressed package
  6. Optimize the documentation

release-0.6.0

4 years ago
  1. 修改获取元数据方式,使用Linkis获取用户权限的元数据信息。
  2. 修改表名为qualitis开头,解决和visiualis的表名不一致产生的冲突。
  3. 增加Debug端口说明。
  4. 修复start.sh脚本的bug。
  5. 增加maven仓库,修复不能下载依赖的问题。
  6. 修改接入DSS文档。
  7. 增加接入Linkis文档。

  1. Modify the method of obtaining metadata, and use Linkis to obtain metadata information of user permissions.
  2. Modify the table name to begin with qualitis, and resolve conflicts caused by inconsistent table names with visiualis.
  3. Added Debug port description.
  4. Fix the bug of start.sh script.
  5. Add maven repository to fix the problem of unable to download dependencies.
  6. Modify the access DSS file.
  7. Added access Linkis guide.

release-0.5.0

4 years ago