LakeSoul Versions Save

LakeSoul is an end-to-end, realtime and cloud native Lakehouse framework with fast data ingestion, concurrent update and incremental data analytics on cloud storages for both BI and AI applications.

v2.5.3

1 month ago
  1. Add shaded packages for release
  2. Fix compaction may write to incorrect partition

v2.5.1

3 months ago
  1. Fix Flink sink parallelism for non-primary key table;
  2. Fix native io filter for non-ascii names and nested columns;
  3. Optimize compaction performance.

v2.5.0

4 months ago

LakeSoul 2.5.0 Release Note

What's New

  1. Python Reader supports PyTorch, PyArrow, Pandas, Ray, and distributed execution;
  2. Support Spark Gluten Vectorized Engine;
  3. Spark SQL supports Compaction, Rollback and other Call Procedures;
  4. Flink CDC’s entire database synchronization supports MySQL, PostgreSQL, PolarDB, and Oracle;
  5. Support streaming and batch export to MySQL, PostgreSQL, PolarDB, and Apache Doris;
  6. Optimized NativeIO performance.

更新内容

  1. Python Reader 支持 PyTorch、PyArrow、Pandas、Ray，支持分布式执行；
  2. 支持 Spark Gluten Vectorized Engine；
  3. Spark SQL 支持 Compaction、Rollback 等 Call Procedures；
  4. Flink CDC 整库同步支持 MySQL、PostgreSQL、PolarDB、Oracle；
  5. 支持流式、批式出湖至 MySQL、PostgreSQL、PolarDB、Apache Doris；
  6. 优化 NativeIO 性能.

What's Changed

New Contributors

Full Changelog: https://github.com/lakesoul-io/LakeSoul/compare/v2.4.1...v2.5.0

v2.4.1

7 months ago

What's Changed

Full Changelog: https://github.com/lakesoul-io/LakeSoul/compare/v2.4.0...v2.4.1

v2.4.0

7 months ago

What's New In This Release

  1. RBAC support for all query engines. doc
  2. Auto cleaning of old compaction data and partition TTL. doc
  3. Upgrade Flink version to 1.17 and support row level update/delete in batch sql.
  4. Optimize whole database Flink cdc sync throughput by 80%: #307
  5. Presto Reader; doc
  6. Python reader and integration with PyTorch and HuggingFace. doc

本次更新内容

  1. 支持 RBAC 角色权限控制，对所有引擎、所有语言API均有效；文档
  2. 自动清理旧的 compaction 数据，支持分区级生命周期（TTL）；文档
  3. 升级 Flink 版本到 1.17，并支持批模式下行级别更新和删除；
  4. 优化整库同步 Flink 作业，吞吐提升 80%： #307 ；
  5. 支持 Presto 读取；文档
  6. 支持原生 Python 读取，提供 PyTorch、HuggingFace 的集成。文档

What's Changed

Full Changelog: https://github.com/lakesoul-io/LakeSoul/commits/v2.4.0

v2.3.1

8 months ago
  • Fix jackson-core packaging for Flink package
  • Fix commons-lang class missing
  • Fix snapshot rollback/cleanup with local timezone

v2.3.0

10 months ago

v2.3.0 Release Notes

This is the first release after LakeSoul donated to Linux Foundation AI & Data. This release contains the following major new features:

  1. Flink Connector for Flink SQL/Table API to read or write LakeSoul in both batch and streaming mode, with the supports of Flink Changelog Stream semantics and row-level upsert and delete. See docs Flink Connector.
  2. Flink CDC Ingestion refactored to infer new tables and schema changes automatically from messages. This enables simpler CDC stream ingestion job development for any kinds of database or message queues.
  3. Global automatic compaction service. See docs Auto Compaction Service.

更新日志

这是 LakeSoul 捐赠给 Linux Foundation AI & Data 后的第一个发布版本。该版本包含以下重要更新：

  1. 全面支持 Flink SQL/Table API. LakeSoul 支持 Flink 流、批读写。流式读写完整支持 Flink Changelog 语义，支持行级别流式增删改。参考文档
  2. Flink CDC 整库同步重构，支持从消息中自动推断新表和 schema 变更。能够更简单的开发 CDC 入湖作业并支持消费任意数据库 CDC 流或消息队列流。
  3. 全局自动 Compaction 服务。参考文档：LakeSoul 全局自动压缩服务使用方法

What's Changed

v2.2.0

1 year ago

LakeSoul Release v2.2.0

v2.2.0 Release Notes

  1. Native IO is by default enabled for Flink CDC Sink and Spark SQL. Native IO uses arrow-rs and Datafusion with special IO optimizations based on arrow-rs' object store. Benchmarks show 3x IO throughput improvement over parquet-mr and Hadoop filesystem. Native IO supports both HDFS and S3 object storage (including S3 protocol compatible storages). Native IO supports all data types in Spark and Flink and has passed both TPC-H and CHBenchmark correctness tests.
  2. Snapshot read and incremental read support on Spark. LakeSoul's incremental read on spark supports both batch mode and microbatch streaming mode.
  3. Default supported Spark's version has been upgraded to Spark 3.3.

v2.2.0 发布日志

  1. Native IO 在 Flink 和 Spark 上默认启用。Native IO 使用 arrow-rs 和 [Datafusion] (https://github.com/apache/arrow-datafusion) 实现，并在 arrow-rs object store 上做了专门的性能优化。在实际测试中比 parquet-mr+hadoop filesystem 快 3 倍以上。Native IO 可以支持 HDFS 和 S3 存储，以及与 S3 兼容的存储系统。Native IO 经过了详细的测试，能够支持 Flink、Spark 所有数据类型，并通过了 TPC-H 和 CHBenchmark 的正确性校验。
  2. 在 Spark 上支持了快照读和增量读功能。增量读功能可以支持 batch 模式和 micro batch streaming 模式。
  3. 默认的 Spark 版本更新到 3.3.

What's Changed

Full Changelog: https://github.com/meta-soul/LakeSoul/commits/v2.2.0

v2.1.1

1 year ago

What's Changed

This is a bug fix release for v2.1.0.

Fixed bugs:

Full Changelog: https://github.com/meta-soul/LakeSoul/compare/2.1.0...v2.1.1

2.1.0

1 year ago

v2.1.0 Release Notes

LakeSoul 2.1.0 brings new Flink CDC sink implementation which supports all tables (with different schemas) in one entire MySQL database sync in one Flink job, automatic schema sync and evolution, automatic new table creation and exactly once guarantee. The currently supported flink version is 1.14.

In 2.1.0 we also reimplement Spark catalog so that it could be used as a standalone catalog rather than a session catalog extension. This change is to avoid some inconsistencies in Spark's v2 table commands, e.g. show tables cannot support v2 tables until 3.3.

Packages for Spark and Flink are separated into two maven submodules. The maven coordinates are com.dmetasoul:lakesoul-spark:2.1.0-spark-3.1.2 and com.dmeatsoul:lakesoul-flink:2.1.0-flink-1.14. All the required transitive dependencies have already been shaded into the released jars.

v2.1.0 发布日志

LakeSoul 2.1.0 增加了全新的 Flink CDC Sink 功能，支持 MySQL 数据库整库千表（支持不同 schema）同步，自动 Schema 变更同步，自动新表感知和严格一次（Exactly Once）语义保证。

Spark 支持部分重写了 Catalog 的实现，使得 Catalog 可以作为非 Session Catalog 扩展使用，主要目的是规避 Spark 在 3.3 版本之前，一些 DDL Command 不支持 V2 表的问题。

Spark 和 Flink 分别拆分成了两个 Maven 子模块。在工程中引用的 Maven 坐标分别是 com.dmetasoul:lakesoul-spark:2.1.0-spark-3.1.2 and com.dmeatsoul:lakesoul-flink:2.1.0-flink-1.14。他们各自的依赖库已经通过 shade 的方式打包到了发布的 jar 包中。

Merged Pull Requests

New Contributors

Full Changelog: https://github.com/meta-soul/LakeSoul/commits/2.1.0