Diennea Herddb Versions Save

Highlights:

• BRIN fix load/unload with locks (#821)
• Fix ignored index memory configuration (#818)

Higtlights:

• support for the new "vector of float" (FLOATA) data type
• new scalar functions cosine_similarity, euclidean_distance and dot_product
• Upgrade BookKeeper to 4.16.3

See #812

Sample table:

CREATE TABLE DOCUMENTS (
            FILENAME string
            CHUNKID int, "
            TEXT string, "
             EMBEDDINGSVECTOR floata,
             PRIMARY KEY(filename, chinkid)
)

Sample Vector Search query

SELECT text
FROM DOCUMENTS
ORDER BY cosine_similarity(embeddingsvector,cast(? as FLOAT ARRAY)) DESC
LIMIT 10

• Converted herddb-cli to a lightway jar, to start a server from cli now you require a herddb-services
• Added SASL username/password auth

• Fixed a memory leak within local JVM connections #798

Bumped the version of some libraries:

• netty from 4.1.72 to 4.1.85
• bouncycastle from 1.65 to 1.70
• calcite from 1.30.0 to 1.32.0
• jacson from 2.12.3 to 2.14.1
• zookeeper from 3.6.2 to 3.8.0
• curator from 5.1.0 to 5.4.0
• slf4j from 1.7.29 to 2.0.4

Notes:

• Upgrade Apache Calcite from 1.28.0 to 1.30.0
• Improved logging for data fetch errors

Notes:

• Upgrade Apache Calcite: this is a big jump, in contains the switch to the Search Operator (SARG)
• Upgrade other dependencies
• Some changes on the Docker image (remove log4j2, stick to Java Util Logging)

Docker image: https://hub.docker.com/layers/herddb/herddb/0.24.0/images/sha256-f2a461f5635311e7c25c70e0284dbbf0f970039e004a229a5e8e88446c76b260?context=explore

• Simplified memory configuration and added configurable separation between data and index memory spaces (#757)
• Upgrade Jetty to 9.4.41.v20210516 (#751)
• Upgrade BookKeeper to 4.14.2 (#750)
• Upgrade commons-io to 2.7 (#747)
• Upgrade Netty to 4.1.59.Final (#738)
• Beta support for real time capture data change (CDC), streaming data directly from BookKeeper

Compatibility: This release is 100% compatible with old versions

• Add support for Foreign Keys
• Upgrade BookKeeper to 4.12.0 and support "bookieId"
• Move Pull request validation to GitHub Actions, use Ephemeral ports for tests
• Enhance support for OpenJPA

Compatibility: This release is 100% compatible with old versions

Feature highlights

• implement REPEATABLE_READ transaction isolation, the default will be READ_COMMITTED, that holds less row-level locks
• implement "UNIQUE" indexes (experimental)
• improve SQL conformance of handling NULL values (this may be considered as a breaking change!)
• implement simple jSQLParser based planner (server.plannertype=jsqlparser)
• implement automatic fallback to Calcite in case of a query that is not plannable with the simple planner (server.plannertype=auto) default planner is still Apache Calcite based, as it offers full SQL support and cost based Volcano planner
• several improvements on the JDBC driver (thanks to @rmannibucau )
• several performance improvement for in-JVM communications (bypass Netty stack at all)
• performance improvements in "local" mode (skip auth, apply better defaults, skip useless logging) (thanks to @rmannibucau )
• ability to drop dependencies to most of Calcite third party dependencies in case of using server.plannertype=jsqlparser (in order to - make a smaller dependency set and resulting size of the JDBC Driver)
• new "DUAL" table like Oracle
• ability to replace the PK of a record
• remove JsonPath and Geometry dependencies
• cache ZooKeeper handle on the client to reduce the number of sessions

see https://github.com/diennea/herddb/milestone/5?closed=1

Compatibility: This release is 100% compatible with old versions, except from 0.20.0BETA, that was an experimental version.