An open source time-series database for fast ingest and SQL queries
Maintenance release improving stability of the deduplication logic, which we introduced in 7.3. For anyone relying on deduplication or considering using it, we recommend to upgrade at the earliest convenience.
alter table resume wal
not having the effect by @ideoma in https://github.com/questdb/questdb/pull/3715
update
sql execution by @ideoma in https://github.com/questdb/questdb/pull/3703
Full Changelog: https://github.com/questdb/questdb/compare/7.3.1...7.3.2
Follow-up release for the data deduplication
logic and IPv4
type, first released in 7.3.
This version brings the dedup logic for CSV Import alongside minor fixes and additional functions compatible with IPv4
.
We also update the Web Console with a new "create table" UI to create new tables interactively. We would love your feedback on this new UI item; let us know your thoughts.
Full Changelog: https://github.com/questdb/questdb/compare/7.3...7.3.1
This release includes the highly anticipated data deduplication logic, which is now supported across all ingestion protocols. Its primary objective is to simplify the process of restarting data streams. If a data stream is interrupted, clients can easily resolve the issue by re-publishing an arbitrarily long window of data. The server will then handle the task of retaining only the missing data.
Data deduplication functions as an upsert operation by preserving the newer version of a record when duplicates are encountered.
To control data deduplication the following SQL syntax is available:
-- deduplicate on 2 columns
CREATE TABLE no_ts_dups_table (timestamp ts, l long, symbol sym, uuid id, string s) timestamps(ts)
PARTITION BY DAY WAL
DEDUPLICATE UPSERT KEYS(ts, id)
-- deduplication on 3 columns
ALTER TABLE no_ts_dups_table DEDUP UPSERT KEYS(ts, id, l);
-- deduplication on 4 columns
ALTER TABLE no_ts_dups_table DEDUP UPSERT KEYS(ts, sym, id, l);
We are thrilled to assure you that data deduplication has a negligible impact on ingestion performance, allowing it to be enabled at all times.
IPV4
data typeThis release introduces a new data type called ipv4. The primary objective behind this addition is to assist users working with IPv4 addresses in alleviating storage constraints and enhancing the overall SQL execution performance. Please see this the original pull request for more details.
ipv4
as supported column type by @allegraharris in https://github.com/questdb/questdb/pull/3523
string
and char
comparison by @bziobrowski in https://github.com/questdb/questdb/pull/3607
Full Changelog: https://github.com/questdb/questdb/compare/7.2.1...7.3
This is a maintenance release ahead of the next major release. Consider upgrading if you are affected by the issues fixed below.
Full Changelog: https://github.com/questdb/questdb/compare/7.2...7.2.1
In QuestDB 7.2, we address a key issue faced by our users regarding write amplification and performance under heavy loads.
Previously, QuestDB utilized fixed-size time partitions to store data in ascending chronological order. However, this approach resulted in write amplification, particularly towards the end of each time interval. To tackle this problem, we have introduced implicit variable-size time partitions in this release.
The new feature intelligently splits partitions at strategic points, ensuring smaller partition sizes and reducing the surface area for copy-on-write operations. This minimizes write amplification and maintains high write performance, even during heavy load scenarios.
To further optimize the system, we have implemented a sophisticated statistical model to balance the split and squash logic. As a result, partitions statistically unlikely to be modified again are asynchronously squashed, reducing the number of files on disk and alleviating stress on the OS metadata catalog.
Write amplification is reduced by orders of magnitude. Users can now experience sustained write performance, even in demanding environments.
This is a breaking change. If you decide to downgrade from this release, partitions, created by 7.2 may not be understood by earlier versions. To proceed with the downgrade you may need to "squash" partitions on each table. To do that, run the following SQL before the downgrade:
ALTER TABLE <tab> SQUASH PARTITIONS
Full Changelog: https://github.com/questdb/questdb/compare/7.1.3...7.2
This release includes several minor hotfixes before the next major release, 7.2.
Full Changelog: https://github.com/questdb/questdb/compare/7.1.2...7.1.3
We ship a new user interface for CSV imports in the Web Console. This stability release collates bug fixes for issues raised by our community. We decided to cut this release ahead of the feature-packed 7.2.
COPY CANCEL
API by @bluestreak01 in https://github.com/questdb/questdb/pull/3298
nullif(long,long)
SQL function by @javier in https://github.com/questdb/questdb/pull/3188
rank()
analytical SQL function by @puzpuzpuz in https://github.com/questdb/questdb/pull/2929
QuestDB
to string returned by version()
function by @davidcooke2 in https://github.com/questdb/questdb/pull/3239
Full Changelog: https://github.com/questdb/questdb/compare/7.1.1...7.1.2
This release was prompted by the library dependency that QuestDB 7.1 introduced in the Web Console. This dependency has affected a lot of users, and we have decided to fix it as soon as possible.
Full Changelog: https://github.com/questdb/questdb/compare/7.1...7.1.1
This release introduces fixes for bugs found in the WAL storage mechanism (introduced in 7.0.1) as a result of thousands of hours of fuzz testing. We also fixed all of the issues reported on GitHub and by users in our public Slack channel. It also includes an improved UI for the Web Console.
LIKE
and ILIKE
operator by @SiyaoIsHiding in https://github.com/questdb/questdb/pull/3006
Full Changelog: https://github.com/questdb/questdb/compare/7.0.1...7.1
QuestDB 7.0.1 is the production-ready implementation of our new storage type, which makes data ingestion faster and more reliable. In this release, we introduce a Write-Ahead-Log (WAL) to ingest data. Newly created tables will now be WAL tables by default. Existing tables will use legacy storage until explicitly migrated to a new storage type.
Below are our latest benchmark results using the Time Series Benchmarking Suite:
The ingestion rate is measured in rows per second:
 | QuestDB 7.0 | influxDB 1.8.10 | TimescaleDB 2.10 (tuned) |
---|---|---|---|
4 workers | 1.458M | 0.181M | 0.304M |
8 workers | 2.874M | 0.306M | 0.464M |
12 workers | 4.155M | 0.372M | 0.517M |
16 workers | 4.373M | 0.375M | 0.510M |
To create a WAL-based table, the following syntax should be used (note WAL
at the end):
create table ABC(x int, t timestamp) timestamp(t) partition by day WAL;
You can also convert your existing table to WAL (followed by an instance restart). The conversion touches table configuration (not data!) and is fully reversible.
To convert an existing table to WAL:
alter table ABC set type WAL;
To convert a WAL table to a non-WAL table:
alter table ABC set type bypass WAL;
Full Changelog: https://github.com/questdb/questdb/compare/7.0.0...7.0.1