Go Sstables Versions Save

New Features:

• move to Go 1.19 and generics
• skiplist is now generic
• sstables merging is supporting generics
• the priority queue moved to its own package, supports generics too
• recordio and sstables support DirectIO and aligned block writes
• recordio now support lzw compression
• simpledb is now covered with linearization tests
• simpledb's WAL can make use of DirectIO

Breaking Changes:

• generics in skiplist, priority queue and sstables merging ** this comes with changes in iterators and comparators
• simpledb.NotFound is now called simpledb.ErrNotFound

Housekeeping:

• benchmarking with ycsb
• new CI jobs
• update codeql to v2
• badge updates

minor patch release with bugfix from b83e8cab620b1937238ef587d5a3c938a98862fb and without directIO support

New Features:

• adding simpledb, a fault-tolerant embedded key-value store using all building blocks in this project
• adding simpledb crash tests, a test suite to exercise crash and recovery scenarios which can be reused for any other key/value store
• recordio now features buffer size tuning options and buffer reuse (directIO and io_uring improvements are in the making)
• improved performance in recordio - between 25x for snappy and up to 60x without compression, especially for small files
• new version (v1) for sstables removing protobuf for value serialization (internal change, old tables are still readable and backward compatible)
• improved performance in sstables: up to 10x faster scanning and merging of tables
• sstables is now merging via iterators, several new ways of compacting during merges - e.g. like in map/reduce
• wal features a non-fsync append for better throughput
• wal features manual rotations in addition to the size-based ones
• memstore has support for tombstones and sstable iterators to enable easier merging
• adding SuperSSTableReader: a wrapper for multiple sstables behind a single sstable interface
• adding EmptySSTableReader: for easier testing and nil-safe merging scenarios

Breaking Changes:

• errors are now properly wrapped per package for easier debugging, as per changes from golang 1.13 - some interfaces now require to handle errors via errors.Is compared to direct equality checks
• all New* methods now return the interface, previously it might've been a pointer to the struct

Notable Bug Fixes:

• 5b18807 fix in skiplist iteration and sstable reader - where an iterator would continue to iterate across the upper boundary

Housekeeping:

• updated documentation
• switched dev branch from master to main
• moved examples package to _examples
• adding some new badges
• add go vet in the CI

With the new release of the go protobuf library, I cleaned a couple TODOs, added more docs and updated all protos and make their appearance consistent.

That means there are some breaking changes in the API, they are fairly minor compared to the changes in the protobuf upgrade.

• v2 recordio format with vint compression
• adding sstable merging
• adding write ahead logs
• remove the vendor folder