Questdb Versions Save

An open source time-series database for fast ingest and SQL queries

7.3.2

8 months ago

Overview

Maintenance release improving stability of the deduplication logic, which we introduced in 7.3. For anyone relying on deduplication or considering using it, we recommend to upgrade at the earliest convenience.

What is new

SQL

What is fixed

CORE

SQL

PGWire

HTTP

UI

New Contributors

Full Changelog: https://github.com/questdb/questdb/compare/7.3.1...7.3.2

7.3.1

9 months ago

Overview

Follow-up release for the data deduplication logic and IPv4 type, first released in 7.3. This version brings the dedup logic for CSV Import alongside minor fixes and additional functions compatible with IPv4.

We also update the Web Console with a new "create table" UI to create new tables interactively. We would love your feedback on this new UI item; let us know your thoughts.

What is new

SQL

UI

What is fixed

CORE

SQL

HTTP

PGWire

UI

New Contributors

Full Changelog: https://github.com/questdb/questdb/compare/7.3...7.3.1

7.3

9 months ago

Overview

Data deduplication

This release includes the highly anticipated data deduplication logic, which is now supported across all ingestion protocols. Its primary objective is to simplify the process of restarting data streams. If a data stream is interrupted, clients can easily resolve the issue by re-publishing an arbitrarily long window of data. The server will then handle the task of retaining only the missing data.

Data deduplication functions as an upsert operation by preserving the newer version of a record when duplicates are encountered.

To control data deduplication the following SQL syntax is available:

-- deduplicate on 2 columns
CREATE TABLE no_ts_dups_table (timestamp ts, l long, symbol sym, uuid id, string s) timestamps(ts) 
PARTITION BY DAY WAL 
DEDUPLICATE UPSERT KEYS(ts, id)

-- deduplication on 3 columns
ALTER TABLE no_ts_dups_table DEDUP UPSERT KEYS(ts, id, l);

-- deduplication on 4 columns
ALTER TABLE no_ts_dups_table DEDUP UPSERT KEYS(ts, sym, id, l);

We are thrilled to assure you that data deduplication has a negligible impact on ingestion performance, allowing it to be enabled at all times.

New IPV4 data type

This release introduces a new data type called ipv4. The primary objective behind this addition is to assist users working with IPv4 addresses in alleviating storage constraints and enhancing the overall SQL execution performance. Please see this the original pull request for more details.

What is new

CORE

SQL

What is fixed

CORE

SQL

UI

  • fix incorrect rendering of last few columns in Web Console grid

New Contributors

Full Changelog: https://github.com/questdb/questdb/compare/7.2.1...7.3

7.2.1

10 months ago

Overview

This is a maintenance release ahead of the next major release. Consider upgrading if you are affected by the issues fixed below.

What is new

SQL

What is fixed

CORE

SQL

PGWire

ILP

New Contributors

Full Changelog: https://github.com/questdb/questdb/compare/7.2...7.2.1

7.2

11 months ago

Overview

In QuestDB 7.2, we address a key issue faced by our users regarding write amplification and performance under heavy loads.

Previously, QuestDB utilized fixed-size time partitions to store data in ascending chronological order. However, this approach resulted in write amplification, particularly towards the end of each time interval. To tackle this problem, we have introduced implicit variable-size time partitions in this release.

Before

image

After

image

The new feature intelligently splits partitions at strategic points, ensuring smaller partition sizes and reducing the surface area for copy-on-write operations. This minimizes write amplification and maintains high write performance, even during heavy load scenarios.

To further optimize the system, we have implemented a sophisticated statistical model to balance the split and squash logic. As a result, partitions statistically unlikely to be modified again are asynchronously squashed, reducing the number of files on disk and alleviating stress on the OS metadata catalog.

Write amplification is reduced by orders of magnitude. Users can now experience sustained write performance, even in demanding environments.

Gotchas

This is a breaking change. If you decide to downgrade from this release, partitions, created by 7.2 may not be understood by earlier versions. To proceed with the downgrade you may need to "squash" partitions on each table. To do that, run the following SQL before the downgrade:

ALTER TABLE <tab> SQUASH PARTITIONS

What is new

SQL

ILP

UI

What is fixed

CORE

SQL

ILP

HTTP

UI

New Contributors

Full Changelog: https://github.com/questdb/questdb/compare/7.1.3...7.2

7.1.3

11 months ago

Overview

This release includes several minor hotfixes before the next major release, 7.2.

What we fixed

CORE

SQL

ILP

UI

New Contributors

Full Changelog: https://github.com/questdb/questdb/compare/7.1.2...7.1.3

7.1.2

1 year ago

Overview

We ship a new user interface for CSV imports in the Web Console. This stability release collates bug fixes for issues raised by our community. We decided to cut this release ahead of the feature-packed 7.2.

What is new

UI

CORE

SQL

HTTP

What we fixed

CORE

SQL

HTTP

PGRWire

New Contributors

Full Changelog: https://github.com/questdb/questdb/compare/7.1.1...7.1.2

7.1.1

1 year ago

Overview

This release was prompted by the library dependency that QuestDB 7.1 introduced in the Web Console. This dependency has affected a lot of users, and we have decided to fix it as soon as possible.

What is new

SQL

What we fixed

CORE

SQL

UI

HTTP

New Contributors

Full Changelog: https://github.com/questdb/questdb/compare/7.1...7.1.1

7.1

1 year ago

Overview

This release introduces fixes for bugs found in the WAL storage mechanism (introduced in 7.0.1) as a result of thousands of hours of fuzz testing. We also fixed all of the issues reported on GitHub and by users in our public Slack channel. It also includes an improved UI for the Web Console.

What is new

CORE

SQL

UI
PGWire

What we fixed

CORE

SQL

PGWire

ILP

HTTP

New Contributors

Full Changelog: https://github.com/questdb/questdb/compare/7.0.1...7.1

7.0.1

1 year ago

Overview

QuestDB 7.0.1 is the production-ready implementation of our new storage type, which makes data ingestion faster and more reliable. In this release, we introduce a Write-Ahead-Log (WAL) to ingest data. Newly created tables will now be WAL tables by default. Existing tables will use legacy storage until explicitly migrated to a new storage type.

🚀 WAL Storage new features

Benchmark

Below are our latest benchmark results using the Time Series Benchmarking Suite:

  • Benchmark: CPU-only
  • Hardware: 32CPUs, 64GB RAM

The ingestion rate is measured in rows per second:

  QuestDB 7.0 influxDB 1.8.10 TimescaleDB 2.10 (tuned)
4 workers 1.458M 0.181M 0.304M
8 workers 2.874M 0.306M 0.464M
12 workers 4.155M 0.372M 0.517M
16 workers 4.373M 0.375M 0.510M

How to migrate from existing legacy tables to WAL-tables

To create a WAL-based table, the following syntax should be used (note WAL at the end):

create table ABC(x int, t timestamp) timestamp(t) partition by day WAL;

You can also convert your existing table to WAL (followed by an instance restart). The conversion touches table configuration (not data!) and is fully reversible.

To convert an existing table to WAL:

  • alter table ABC set type WAL;
  • restart instance

To convert a WAL table to a non-WAL table:

  • alter table ABC set type bypass WAL;
  • restart instance

What is new

CORE

SQL

HTTP

UI

What we fixed

CORE

SQL

ILP

Full Changelog: https://github.com/questdb/questdb/compare/7.0.0...7.0.1