Ceresdb Versions Save

HoraeDB is a high-performance, distributed, cloud native time-series database.

v2.0.0

3 weeks ago

Upgrade from 1.x.x to 2.0.0

The transition from CeresDB to Apache HoraeDB introduces several breaking changes. To facilitate upgrading from older versions to v2.0.0, specific alterations are necessary.

Upgrade Steps

  1. Setup required envs
export HORAEDB_DEFAULT_CATALOG=ceresdb
  1. Update config

Etcd's root should be configured both in horaedb and horaemeta

For horaedb

[cluster_deployment.etcd_client]
server_addrs = ['127.0.0.1:2379']
root_path = "/rootPath"

For horaemeta

storage-root-path = "/rootPath"
  1. Upgrade horaemeta

Horaedb will throw following errors, which is expected

2024-01-23 14:37:57.726 ERRO [src/cluster/src/cluster_impl.rs:136] Send heartbeat to meta failed, err:Failed to send heartbeat, cluster:defaultCluster, err:status: Unimplemented, message: "unknown service meta_service.MetaRpcService", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc"} }
  1. Upgrade horaedb

After all server upgraded, the cluster should be ready for read/write, and old data could be queried like before.

What's Changed

Breaking Changes

Features

Refactor

Fixed

Docs

Chore

New Contributors

Full Changelog: https://github.com/apache/incubator-horaedb/compare/v1.2.7...v2.0.0

v1.2.7

7 months ago

Major Features

Partition Table

  • Support random partition rule #1193
  • Avoid memory allocation during partition write requests #1208
  • Fix wrong text of show create table for partition table #1214
  • Improved partitioned table tests powered by tsbs #1195

Performance

  • Teach ceresdb to run the whole dist query process #1204
  • Support aggr push down in distributed query #1232
  • Store real time range in sst #1225
  • Use real time range to filter memtable #1233
  • Rewrite not in expr to in #1236
  • Dedup requests in proxy #1125
  • Support dedup execute physical plan #1237

Bug Fix

  • Fix deadlock when dedup stream read #1199
  • Fix panic when read data out of range by disk cache #1206
  • Fix lock contention on acquiring the arena stats #1207
  • Fix panic if dedupped query fails #1229

What's Changed

New Contributors

Full Changelog: https://github.com/CeresDB/ceresdb/compare/v1.2.6...v1.2.7

v1.2.6

8 months ago

Major Features

Query

  • Optimize datafusion plan, remove unnecessary node #1150
  • Optimize disk cache, avoid panic when cache file is corrupted #1130
  • Support PostgreSQL protocol #1138

Remote engine

  • Optimize remote server's protocol, reduce payload overhead when write batch is small #1146

WAL

  • Open wal parallelly #1129
  • Introduce columnar encoding

What's Changed

New Contributors

Full Changelog: https://github.com/CeresDB/ceresdb/compare/v1.2.5...v1.2.6

v1.2.5

9 months ago

Major Features

What's Changed

Full Changelog: https://github.com/CeresDB/ceresdb/compare/v1.2.4...v1.2.5

v1.2.4

10 months ago

Major Features

  • Support shard based recovery to improve performance #976

  • Improve the stability of Kafka based wal

    • Some enhancements on Kafka client #980 #1005
    • Refactor wal deletion algorithm #1064

  • Improve performance of query and write

    • Introduce parquet page filter to accelerate query #664
    • Optimize bloom filter building process to accelerate flush #967 #975

  • Support more object store backends

    • Support S3 as object store backend #969
    • Support OceanBase as object store backend(unstable) #887 #970 #971

  • Other new features

    • Support hex literal in sql #1030
    • Improve sst-metadata tool for better debugging #1019

What's Changed

Full Changelog: https://github.com/CeresDB/ceresdb/compare/v1.2.2...v1.2.4

v1.2.2

11 months ago

Major Features

  • Enhancement on proxy module:
    • #875 Support influxdb api with proxy
    • #886 #896 Refactor sql query with proxy
    • #932 Support querying partitioned sub-table by http query
    • #878 Support forwarding of Prometheus remote write request
  • Improvement of write performance:
    • #879 Support merge small write requests
    • #918 Skip building column schema when columns already exists
  • Enhancement on debugging tools:
    • #909 Support tokio console for debugging
    • #927 Add sst-metadata tool to query sst metadata

Bug Fix

  • Cluster
    • #908 Fix the problem that shard cannot be closed normally
    • #941 Fix the problem that the shard cannot be opened normally due to deadlock
  • Compaction
    • #910 Limit input sst size when compact for old bucket
    • #915 Ensure pick at least 2 files for compaction
  • Proxy
    • #911 Fix auto create table without CeresMeta
  • PromQL
    • #901 Fix reserve column case when build plan

What's Changed

New Contributors

Full Changelog: https://github.com/CeresDB/ceresdb/compare/v1.2.0...v1.2.2

v1.2.0

1 year ago

Upgrade Guide

NOTE: this guide is only used for upgrading CeresDB v1.1.0 to CeresDB v1.2.0, ignore it if you want to deploy a brand new CeresDB cluster with v1.2.0.

In v1.2.0, some incompatible changes are contained, so it's important to upgrade carefully:

  1. First, stop all the instances of CeresDB and CeresMeta;
  2. Upgrade the CeresMeta first by referring to the Upgrade Guide of CeresMeta;
  3. When upgrade the CeresDB, the config should be updated:
  • Change the config section [analytic.compaction_config] to [analytic.compaction] if you use it;
  • Add the config section about the [cluster_deployment.etcd_client] if your CeresDB cluster is in WithMeta mode:
[cluster_deployment.etcd_client]
server_addrs = ['127.0.0.1:2379']
root_path = "/rootPath"

NOTE: the root_path must be /rootPath if upgrade from v1.1.0. 4. After updating CeresDB config, start the CeresDB server;

Major Features

  • Enhancement on InfluxQL support:
    • Support query with aggregators;
    • #854 optimize influxql planner to load all tables on demand instead of loading them when initializing the planner;
    • Replace influxdb_iox with CeresDB/influxql to remove unnecessary dependencies introduced by influxdb_iox;
  • Enhancement on proxy module:
    • Implement the proxy as a separate module;
    • Support forward table requests in proxy;
    • Support read and write on partition table in proxy;
    • Recover the metadata of partition table from CeresMeta instead of CeresDB in proxy;
  • Improvement of write performance:
    • #822 solves the problem that compaction schedule triggered by flush procedure may block the write procedure;
    • #814 is a big change set, and replaces the write queue with the lock on table level for less write contentions;
    • #843 adjusts the flush strategy to avoid frequent write stall;
    • #861 brings the level 1 to SSTs, and currently the SST of the level 0, which is generated by flushing, won't contain complex indexes, e.g. xor-filter, leading to faster flushing and less write stall;
  • Enhancement on observability:
    • #774 introduces the hotspot recorder that can be used to find out the top tables with the highest write/read throughput in a specific time window;
    • #827 #831 provides more metrics for all the stages of writing procedure, which can be used to troubleshoot write performance problems, and the grafana dashboard config has been already updated.
    • #817 introduces the CPU profiler, and the flamegraph of CPU can be generated easily just by an HTTP request to CeresDB server;
  • Support the new mechanism of failover and load balancing, more details can refer to the [Release Note v1.2.0] of CeresMeta:
    • #706 #853 implements the distributed locks for shard based on ETCD, and opening and closing of shards is only allowed with the shard lock held, and after that, data corruption caused by multiple shard leaders will be avoided completely;
    • Support automatic failover of CeresDB nodes, that is to say, the service recovery can be handled automatically without any manual intervention;
    • Support automatic load balance based on consistent hashing, which can ensure that shards are evenly distributed on each node of the cluster when the number of the cluster nodes increases or decreases;

Thanks

Heartfelt thanks for @zouxiang1993's effort in helping troubleshooting write performance issues.

What's Changed

Full Changelog: https://github.com/CeresDB/ceresdb/compare/v1.1.0...v1.2.0

v1.1.0

1 year ago

Major features

  • Initial support for InfluxQL, usage can be found here.
  • Introduce proxy module:
    • Support auto update schema when a new column occurs in the write request;
    • Support forward stream sql query now, usage can be found here.
  • Optimize SST write/read process with less memory consumption.
  • explain analyze [SQL] statement is able to show the details of scan procedure.

What's Changed

New Contributors

Full Changelog: https://github.com/CeresDB/ceresdb/compare/v1.0.0...v1.1.0

v1.0.0

1 year ago

Major features

  • Support prometheus remote storage protocol
  • Query performance improvement and resource control
    • Replace bloom filter with XOR8 filter
    • Add timeout for query
    • Add route cache in remote engine client in table partition
    • Support locate partition for in sql expression in table partition
  • Internal refactor
    • Refactor sst module for better extensibility
    • Refactor manifest module
  • Refactor grpc storage service
  • Add intergration test for cluster mode
  • Bug fix
    • Fix sql identifier case-sensibility
    • Correct the order of sync meta snapshot and clean logs in wal on kafka
    • Update flushed_sequence_num after compaction
    • Fix varbinary type error

What's Changed

New Contributors

Full Changelog: https://github.com/CeresDB/ceresdb/compare/v1.0.0-alpha02...v1.0.0

v1.0.0-alpha02

1 year ago

Major features

  • Table Partition
    • Support key partition(MySQL-like syntax).
    • Implement PartitionTable to support query and write for partitioned tables.
  • Improve query performance
    • Add disk based object store cache. (unstable)
    • Implement parallel get_byte_ranges for ObjectStoreReader.
    • Scan row groups in one sst file parallelly.
    • Support bloom filter in hybrid format.
    • Support MergeIterator to pull data concurrently.
  • Support auto query forwarding for grpc.
  • Normalize the case of SQL and make clear that all SQL cases are sensitive.
  • Chore
    • Migrate current harness tests to sqlness. by @dust1
    • Support memory usage limit on background compaction.
    • Make the bloom filter optional in sst meta.
  • Bug fix
    • Fix wrong primary key when define tsid and timestamp key as primary key.
    • Fix wrong path in the result from StoreWithPrefix.
    • Fix lru-weighted-cache` memory leak.
    • Fix some bugs in background compaction.
    • Fix wrong profile output os path for heap profiling.

What's Changed

Full Changelog: https://github.com/CeresDB/ceresdb/compare/v1.0.0-alpha01...v1.0.0-alpha02