Zstandard - Fast real-time compression algorithm
This release highlights the deployment of Google Chrome 123, introducing zstd-encoding
for Web traffic, introduced as a preferable option for compression of dynamic contents. With limited web server support for zstd-encoding
due to its novelty, we are launching an updated Zstandard version to facilitate broader adoption.
ZSTD_c_targetCBlockSize
Using zstd
compression for large documents over the Internet, data is segmented into smaller blocks of up to 128 KB, for incremental updates. This is crucial for applications like Chrome that process parts of documents as they arrive. However, on slow or congested networks, there can be some brief unresponsiveness in the middle of a block transmission, delaying update. To mitigate such scenarios, libzstd
introduces the new parameter ZSTD_c_targetCBlockSize
, enabling the division of blocks into even smaller segments to enhance initial byte delivery speed. Activating this feature incurs a cost, both runtime (equivalent to -2% speed at level 8) and a slight compression efficiency decrease (<0.1%), but offers some interesting latency reduction, notably beneficial in areas with less powerful network infrastructure.
libzstd
provides build customization, including options to compile only the compression or decompression modules, minimizing binary size. Enhanced in v1.5.6
(source), it now allows for even finer control by enabling selective inclusion or exclusion of specific components within these modules. This advancement aids applications needing precise binary size management.
This release includes various minor enhancements and bug fixes to enhance user experience. Key updates include an expanded list of recognized compressed file suffixes for the --exclude-compressed
flag, improving efficiency by skipping presumed incompressible content. Furthermore, compatibility has been broadened to include additional chipsets (sparc64
, ARM64EC
, risc-v
) and operating systems (QNX
, AIX
, Solaris
, HP-UX
).
api: Promote ZSTD_c_targetCBlockSize
to Stable API by @felixhandte
api: new experimental ZSTD_d_maxBlockSize
parameter, to reduce streaming decompression memory, by @terrelln
perf: improve performance of param ZSTD_c_targetCBlockSize
, by @Cyan4973
perf: improved compression of arrays of integers at high compression, by @Cyan4973
lib: reduce binary size with selective built-time exclusion, by @felixhandte
lib: improved huffman speed on small data and linux kernel, by @terrelln
lib: accept dictionaries with partial literal tables, by @terrelln
lib: fix CCtx size estimation with external sequence producer, by @embg
lib: fix corner case decoder behaviors, by @Cyan4973 and @aimuz
lib: fix zdict prototype mismatch in static_only mode, by @ldv-alt
lib: fix several bugs in magicless-format decoding, by @embg
cli: add common compressed file types to --exclude-compressed
by @daniellerozenblit (requested by @dcog989)
cli: fix mixing -c
and -o
commands with --rm
, by @Cyan4973
cli: fix erroneous exclusion of hidden files with --output-dir-mirror
by @felixhandte
cli: improved time accuracy on BSD, by @felixhandte
cli: better errors on argument parsing, by @KapJI
tests: better compatibility with older versions of grep
, by @Cyan4973
tests: lorem ipsum generator as default content generator, by @Cyan4973
build: cmake improvements by @terrelln, @sighingnow, @gjasny, @JohanMabille, @Saverio976, @gruenich, @teo-tsirpanis
build: bazel support, by @jondo2010
build: fix cross-compiling for AArch64 with lld by @jcelerier
build: fix Apple platform compatibility, by @nidhijaju
build: fix Visual 2012 and lower compatibility, by @Cyan4973
build: improve win32 support, by @DimitriPapadopoulos
build: better C90 compliance for zlibWrapper, by @emaste
port: make: fat binaries on macos, by @mredig
port: ARM64EC compatibility for Windows, by @dunhor
port: QNX support by @klausholstjacobsen
port: MSYS2 and Cygwin makefile installation and test support, by @QBos07
port: risc-v support validation in CI, by @Cyan4973
port: sparc64 support validation in CI, by @Cyan4973
port: AIX compatibility, by @likema
port: HP-UX compatibility, by @likema
doc: Improved specification accuracy, by @elasota
bug: Fix and deprecate ZSTD_generateSequences (#3981), by @terrelln
DESTDIR
and BINDIR
separately by @paulmenzel in https://github.com/facebook/zstd/pull/3752
do { } while (0)
by @terrelln in https://github.com/facebook/zstd/pull/3831
targetCBlockSize
by @Cyan4973 in https://github.com/facebook/zstd/pull/3917
zstd::libzstd
CMake target if only static or dynamic linkage is specified. by @teo-tsirpanis in https://github.com/facebook/zstd/pull/3811
ZSTD_RowFindBestMatch
comment by @yoniko in https://github.com/facebook/zstd/pull/3947
utimensat()
on FreeBSD by @felixhandte in https://github.com/facebook/zstd/pull/3960
--output-dir-mirror
by @felixhandte in https://github.com/facebook/zstd/pull/3963
ZSTD_c_targetCBlockSize
Parameter to Stable API by @felixhandte in https://github.com/facebook/zstd/pull/3964
Full Changelog: https://github.com/facebook/zstd/compare/v1.5.5...v1.5.6
This is a quick fix release. The primary focus is to correct a rare corruption bug in high compression mode, detected by @danlark1 . The probability to generate such a scenario by random chance is extremely low. It evaded months of continuous fuzzer tests, due to the number and complexity of simultaneous conditions required to trigger it. Nevertheless, @danlark1 from Google shepherds such a humongous amount of data that he managed to detect a reproduction case (corruptions are detected thanks to the checksum), making it possible for @terrelln to investigate and fix the bug. Thanks !
While the probability might be very small, corruption issues are nonetheless very serious, so an update to this version is highly recommended, especially if you employ high compression modes (levels 16+).
When the issue was detected, there were a number of other improvements and minor fixes already in the making, hence they are also present in this release. Let’s detail the main ones.
--patch-from
modeV1.5.5
introduces memory-mapped dictionaries, by @daniellerozenblit, for both posix #3486 and windows #3557.
This feature allows zstd
to memory-map large dictionaries, rather than requiring to load them into memory. This can make a pretty big difference for memory-constrained environments operating patches for large data sets.
It's mostly visible under memory pressure, since mmap
will be able to release less-used memory and continue working.
But even when memory is plentiful, there are still measurable memory benefits, as shown in the graph below, especially when the reference turns out to be not completely relevant for the patch.
This feature is automatically enabled for --patch-from
compression/decompression when the dictionary is larger than the user-set memory limit. It can also be manually enabled/disabled using --mmap-dict
or --no-mmap-dict
respectively.
Additionally, @daniellerozenblit introduces significant speed improvements for --patch-from
.
An I/O
optimization in #3486 greatly improves --patch-from
decompression speed on Linux, typically by +50%
on large files (~1GB).
Compression speed is also taken care of, with a dictionary-indexing speed optimization introduced in #3545. It wildly accelerates --patch-from
compression, typically doubling speed on large files (~1GB), sometimes even more depending on exact scenario.
This speed improvement comes at a slight regression in compression ratio, and is therefore enabled only on non-ultra compression strategies.
The row-hash match finder introduced in version 1.5.0 for levels 5-12 has been improved in version 1.5.5, enhancing its speed in specific corner-case scenarios.
The first optimization (#3426) accelerates streaming compression using ZSTD_compressStream
on small inputs by removing an expensive table initialization step. This results in remarkable speed increases for very small inputs.
The following scenario measures compression speed of ZSTD_compressStream
at level 9 for different sample sizes on a linux platform running an i7-9700k cpu.
sample size | v1.5.4 (MB/s) |
v1.5.5 (MB/s) |
improvement |
---|---|---|---|
100 | 1.4 | 44.8 | x32 |
200 | 2.8 | 44.9 | x16 |
500 | 6.5 | 60.0 | x9.2 |
1K | 12.4 | 70.0 | x5.6 |
2K | 25.0 | 111.3 | x4.4 |
4K | 44.4 | 139.4 | x3.2 |
... | ... | ... | |
1M | 97.5 | 99.4 | +2% |
The second optimization (#3552) speeds up compression of incompressible data by a large multiplier. This is achieved by increasing the step size and reducing the frequency of matching when no matches are found, with negligible impact on the compression ratio. It makes mid-level compression essentially inexpensive when processing incompressible data, typically, already compressed data (note: this was already the case for fast compression levels).
The following scenario measures compression speed of ZSTD_compress
compiled with gcc-9
for a ~10MB incompressible sample on a linux platform running an i7-9700k cpu.
level | v1.5.4 (MB/s) |
v1.5.5 (MB/s) |
improvement |
---|---|---|---|
3 | 3500 | 3500 | not a row-hash level (control) |
5 | 400 | 2500 | x6.2 |
7 | 380 | 2200 | x5.8 |
9 | 176 | 1880 | x10 |
11 | 67 | 1130 | x16 |
13 | 89 | 89 | not a row-hash level (control) |
There are other welcome speed improvements in this package.
For example, @felixhandte managed to increase processing speed of small files by carefully reducing the nb of system calls (#3479). This can easily translate into +10% speed when processing a lot of small files in batch.
The Seekable format received a bit of care. It's now much faster when splitting data into very small blocks (#3544). In an extreme scenario reported by @P-E-Meunier, it improves processing speed by x90. Even for more "common" settings, such as using 4KB blocks on some "normally" compressible data like enwik
, it still provides a healthy x2 processing speed benefit. Moreover, @dloidolt merged an optimization that reduces the nb of I/O
seek()
events during reads (decompression), which is also beneficial for speed.
The release is not limited to speed improvements, several loose ends and corner cases were also fixed in this release. For a more detailed list of changes, please take a look at the changelog.
mmap
large dictionaries to save memory, by @daniellerozenblit--patch-from
mode (~+50%) (#3545) by @daniellerozenblitzstd
no longer crashes when requested to write into write-protected directory (#3541) by @felixhandte-o
(#3584, @Cyan4973) reported by @georgmucmake
does no longer require 3.18 as minimum version (#3510) by @koutests/fullbench
can benchmark multiple files (#3516) by @dloidoltf
-variants of chmod()
and chown()
by @felixhandte in https://github.com/facebook/zstd/pull/3479
setvbuf()
on Null File Pointer by @felixhandte in https://github.com/facebook/zstd/pull/3541
-std=c++11
When Default is Older by @felixhandte in https://github.com/facebook/zstd/pull/3574
dest
is valid for decompression by @daniellerozenblit in https://github.com/facebook/zstd/pull/3555
Full Changelog: https://github.com/facebook/zstd/compare/v1.5.4...v1.5.5
Zstandard v1.5.4
is a pretty big release benefiting from one year of work, spread over > 650 commits. It offers significant performance improvements across multiple scenarios, as well as new features (detailed below). There is a crop of little bug fixes too, a few ones targeting the 32-bit mode are important enough to make this release a recommended upgrade.
This release has accumulated a number of scenario-specific improvements, that cumulatively benefit a good portion of installed base in one way or another.
Among the easier ones to describe, the repository has received several contributions for arm
optimizations, notably from @JunHe77 and @danlark1. And @terrelln has improved decompression speed for non-x64 systems, including arm
. The combination of this work is visible in the following example, using an M1-Pro (aarch64
architecture) :
cpu | function | corpus | v1.5.2 |
v1.5.4 |
Improvement |
---|---|---|---|---|---|
M1 Pro | decompress | silesia.tar |
1370 MB/s | 1480 MB/s | + 8% |
Galaxy S22 | decompress | silesia.tar |
1150 MB/s | 1200 MB/s | + 4% |
Middle compression levels (5-12) receive some care too, with @terrelln improving the dispatch engine, and @danlark1 offering NEON
optimizations. Exact speed up vary depending on platform, cpu, compiler, and compression level, though one can expect gains ranging from +1 to +10% depending on scenarios.
cpu | function | corpus | v1.5.2 |
v1.5.4 |
Improvement |
---|---|---|---|---|---|
i7-9700k | compress -6 | silesia.tar |
110 MB/s | 121 MB/s | +10% |
Galaxy S22 | compress -6 | silesia.tar |
98 MB/s | 103 MB/s | +5% |
M1 Pro | compress -6 | silesia.tar |
122 MB/s | 130 MB/s | +6.5% |
i7-9700k | compress -9 | silesia.tar |
64 MB/s | 70 MB/s | +9.5% |
Galaxy S22 | compress -9 | silesia.tar |
51 MB/s | 52 MB/s | +1% |
M1 Pro | compress -9 | silesia.tar |
77 MB/s | 86 MB/s | +11.5% |
i7-9700k | compress -12 | silesia.tar |
31.6 MB/s | 31.8 MB/s | +0.5% |
Galaxy S22 | compress -12 | silesia.tar |
20.9 MB/s | 22.1 MB/s | +5% |
M1 Pro | compress -12 | silesia.tar |
36.1 MB/s | 39.7 MB/s | +10% |
Speed of the streaming compression interface has been improved by @embg in scenarios involving large files (where size is a multiple of the windowSize
parameter). The improvement is mostly perceptible at high speeds (i.e. ~level 1). In the following sample, the measurement is taken directly at ZSTD_compressStream()
function call, using a dedicated benchmark tool tests/fullbench
.
cpu | function | corpus | v1.5.2 |
v1.5.4 |
Improvement |
---|---|---|---|---|---|
i7-9700k | ZSTD_compressStream() -1 |
silesia.tar |
392 MB/s | 429 MB/s | +9.5% |
Galaxy S22 | ZSTD_compressStream() -1 |
silesia.tar |
380 MB/s | 430 MB/s | +13% |
M1 Pro | ZSTD_compressStream() -1 |
silesia.tar |
476 MB/s | 539 MB/s | +13% |
Finally, dictionary compression speed has received a good boost by @embg. Exact outcome varies depending on system and corpus. The following result is achieved by cutting the enwik8
compression corpus into 1KB blocks, generating a dictionary from these blocks, and then benchmarking the compression speed at level 1.
cpu | function | corpus | v1.5.2 |
v1.5.4 |
Improvement |
---|---|---|---|---|---|
i7-9700k | dictionary compress | enwik8 -B1K |
125 MB/s | 165 MB/s | +32% |
Galaxy S22 | dictionary compress | enwik8 -B1K |
138 MB/s | 166 MB/s | +20% |
M1 Pro | dictionary compress | enwik8 -B1K |
155 MB/s | 195 MB/s | +25 % |
There are a few more scenario-specifics improvements listed in the changelog
section below.
The 1.5.4 release improves IO performance of zstd
CLI, by using system buffers (macos
) and adding a new asynchronous I/O capability, enabled by default on large files (when threading is available). The user can also explicitly control this capability with the --[no-]asyncio
flag . These new threads remove the need to block on IO operations. The impact is mostly noticeable when decompressing large files (>= a few MBs), though exact outcome depends on environment and run conditions.
Decompression speed gets significant gains due to its single-threaded serial nature and the high speeds involved. In some cases we observe up to double performance improvement (local Mac machines) and a wide +15-45% benefit on Intel Linux servers (see table for details).
On the compression side of things, we’ve measured up to 5% improvements. The impact is lower because compression is already partially asynchronous via the internal MT mode (see release v1.3.4).
The following table shows the elapsed run time for decompressions of silesia
and enwik8
on several platforms - some Skylake-era Linux servers and an M1 MacbookPro. It compares the time it takes for version v1.5.2
to version v1.5.4
with asyncio on and off.
platform | corpus | v1.5.2 |
v1.5.4-no-asyncio |
v1.5.4 |
Improvement |
---|---|---|---|---|---|
Xeon D-2191A CentOS8 | enwik8 |
280 MB/s | 280 MB/s | 324 MB/s | +16% |
Xeon D-2191A CentOS8 | silesia.tar |
303 MB/s | 302 MB/s | 386 MB/s | +27% |
i7-1165g7 win10 | enwik8 |
270 MB/s | 280 MB/s | 350 MB/s | +27% |
i7-1165g7 win10 | silesia.tar |
450 MB/s | 440 MB/s | 580 MB/s | +28% |
i7-9700K Ubuntu20 | enwik8 |
600 MB/s | 604 MB/s | 829 MB/s | +38% |
i7-9700K Ubuntu20 | silesia.tar |
683 MB/s | 678 MB/s | 991 MB/s | +45% |
Galaxy S22 | enwik8 |
360 MB/s | 420 MB/s | 515 MB/s | +70% |
Galaxy S22 | silesia.tar |
310 MB/s | 320 MB/s | 580 MB/s | +85% |
MBP M1 | enwik8 |
428 MB/s | 734 MB/s | 815 MB/s | +90% |
MBP M1 | silesia.tar |
465 MB/s | 875 MB/s | 1001 MB/s | +115% |
libzstd
can now support external sequence producers via a new advanced registration function ZSTD_registerSequenceProducer()
(#3333).
This API allows users to provide their own custom sequence producer which libzstd invokes to process each block. The produced list of sequences (literals and matches) is then post-processed by libzstd to produce valid compressed blocks.
This block-level offload API is a more granular complement of the existing frame-level offload API compressSequences()
(introduced in v1.5.1
). It offers an easier migration story for applications already integrated with libzstd
: the user application continues to invoke the same compression functions ZSTD_compress2()
or ZSTD_compressStream2()
as usual, and transparently benefits from the specific properties of the external sequence producer. For example, the sequence producer could be tuned to take advantage of known characteristics of the input, to offer better speed / ratio.
One scenario that becomes possible is to combine this capability with hardware-accelerated matchfinders, such as the Intel® QuickAssist accelerator (Intel® QAT) provided in server CPUs such as the 4th Gen Intel® Xeon® Scalable processors (previously codenamed Sapphire Rapids). More details to be provided in future communications.
perf: +20% faster huffman decompression for targets that can't compile x64 assembly (#3449, @terrelln)
perf: up to +10% faster streaming compression at levels 1-2 (#3114, @embg)
perf: +4-13% for levels 5-12 by optimizing function generation (#3295, @terrelln)
pref: +3-11% compression speed for arm
target (#3199, #3164, #3145, #3141, #3138, @JunHe77 and #3139, #3160, @danlark1)
perf: +5-30% faster dictionary compression at levels 1-4 (#3086, #3114, #3152, @embg)
perf: +10-20% cold dict compression speed by prefetching CDict tables (#3177, @embg)
perf: +1% faster compression by removing a branch in ZSTD_fast_noDict (#3129, @felixhandte)
perf: Small compression ratio improvements in high compression mode (#2983, #3391, @Cyan4973 and #3285, #3302, @daniellerozenblit)
perf: small speed improvement by better detecting STATIC_BMI2
for clang
(#3080, @TocarIP)
perf: Improved streaming performance when ZSTD_c_stableInBuffer
is set (#2974, @Cyan4973)
cli: Asynchronous I/O for improved cli speed (#2975, #2985, #3021, #3022, @yoniko)
cli: Change zstdless
behavior to align with zless
(#2909, @binhdvo)
cli: Keep original file if -c
or --stdout
is given (#3052, @dirkmueller)
cli: Keep original files when result is concatenated into a single output with -o
(#3450, @Cyan4973)
cli: Preserve Permissions and Ownership of regular files (#3432, @felixhandte)
cli: Print zlib/lz4/lzma library versions with -vv
(#3030, @terrelln)
cli: Print checksum value for single frame files with -lv
(#3332, @Cyan4973)
cli: Print dictID
when present with -lv
(#3184, @htnhan)
cli: when stderr
is not the console, disable status updates, but preserve final summary (#3458, @Cyan4973)
cli: support --best
and --no-name
in gzip
compatibility mode (#3059, @dirkmueller)
cli: support for posix
high resolution timer clock_gettime()
, for improved benchmark accuracy (#3423, @Cyan4973)
cli: improved help/usage (-h
, -H
) formatting (#3094, @dirkmueller and #3385, @jonpalmisc)
cli: Fix better handling of bogus numeric values (#3268, @ctkhanhly)
cli: Fix input consists of multiple files and stdin
(#3222, @yoniko)
cli: Fix tiny files passthrough (#3215, @cgbur)
cli: Fix for -r
on empty directory (#3027, @brailovich)
cli: Fix empty string as argument for --output-dir-*
(#3220, @embg)
cli: Fix decompression memory usage reported by -vv --long
(#3042, @u1f35c, and #3232, @zengyijing)
cli: Fix infinite loop when empty input is passed to trainer (#3081, @terrelln)
cli: Fix --adapt
doesn't work when --no-progress
is also set (#3354, @terrelln)
api: Support for External Sequence Producer (#3333, @embg)
api: Support for in-place decompression (#3432, @terrelln)
api: New ZSTD_CCtx_setCParams()
function, set all parameters defined in a ZSTD_compressionParameters
structure (#3403, @Cyan4973)
api: Streaming decompression detects incorrect header ID sooner (#3175, @Cyan4973)
api: Window size resizing optimization for edge case (#3345, @daniellerozenblit)
api: More accurate error codes for busy-loop scenarios (#3413, #3455, @Cyan4973)
api: Fix limit overflow in compressBound
and decompressBound
(#3362, #3373, Cyan4973) reported by @nigeltao
api: Deprecate several advanced experimental functions: streaming (#3408, @embg), copy (#3196, @mileshu)
bug: Fix corruption that rarely occurs in 32-bit mode with wlog=25 (#3361, @terrelln)
bug: Fix for block-splitter (#3033, @Cyan4973)
bug: Fixes for Sequence Compression API (#3023, #3040, @Cyan4973)
bug: Fix leaking thread handles on Windows (#3147, @animalize)
bug: Fix timing issues with cmake/meson builds (#3166, #3167, #3170, @Cyan4973)
build: Allow user to select legacy level for cmake (#3050, @shadchin)
build: Enable legacy support by default in cmake (#3079, @niamster)
build: Meson build script improvements (#3039, #3120, #3122, #3327, #3357, @eli-schwartz and #3276, @neheb)
build: Add aarch64 to supported architectures for zstd_trace (#3054, @ooosssososos)
build: support AIX architecture (#3219, @qiongsiwu)
build: Fix ZSTD_LIB_MINIFY
build macro, which now reduces static library size by half (#3366, @terrelln)
build: Fix Windows issues with Multithreading translation layer (#3364, #3380, @yoniko) and ARM64 target (#3320, @cwoffenden)
build: Fix cmake
script (#3382, #3392, @terrelln and #3252 @Tachi107 and #3167 @Cyan4973)
doc: Updated man page, providing more details for --train
mode (#3112, @Cyan4973)
doc: Add decompressor errata document (#3092, @terrelln)
misc: Enable Intel CET (#2992, #2994, @hjl-tools)
misc: Fix contrib/
seekable format (#3058, @yhoogstrate and #3346, @daniellerozenblit)
misc: Improve speed of the one-file library generator (#3241, @wahern and #3005, @cwoffenden)
ip1
into Table by @felixhandte in https://github.com/facebook/zstd/pull/3129
wlog
when doing --long
by @zengyijing in https://github.com/facebook/zstd/pull/3226
make clean
list maintenance by adding a CLEAN
variable by @Cyan4973 in https://github.com/facebook/zstd/pull/3256
-E
flag in sed
by @haampie in https://github.com/facebook/zstd/pull/3245
ZSTD_count
call by @JunHe77 in https://github.com/facebook/zstd/pull/3199
zstd
CLI accepts bogus values for numeric parameters by @ctkhanhly in https://github.com/facebook/zstd/pull/3268
clang
by @MaskRay in https://github.com/facebook/zstd/pull/3273
__attribute__((aligned(1)))
for unaligned access by @Hello71 in https://github.com/facebook/zstd/pull/2881
-h
, -H
) formatting by @jonpalmisc in https://github.com/facebook/zstd/pull/3385
-Wdocumentation
by @terrelln in https://github.com/facebook/zstd/pull/3393
ZSTD_CCtx_setCParams()
entry point, to set all parameters defined in a ZSTD_compressionParameters
structure by @Cyan4973 in https://github.com/facebook/zstd/pull/3403
timefn
unit, restore support for clock_gettime()
by @Cyan4973 in https://github.com/facebook/zstd/pull/3423
stderr
is not the console by @Cyan4973 in https://github.com/facebook/zstd/pull/3458
Full Automated Changelog: https://github.com/facebook/zstd/compare/v1.5.2...v1.5.4
Zstandard v1.5.2 is a bug-fix release, addressing issues that were raised with the v1.5.1 release.
In particular, as a side-effect of the inclusion of assembly code in our source tree, binary artifacts were being marked as needing an executable stack on non-amd64 architectures. This release corrects that issue. More context is available in #2963.
This release also corrects a performance regression that was introduced in v1.5.0 that slows down compression of very small data when using the streaming API. Issue #2966 tracks that topic.
In addition there are a number of smaller improvements and fixes.
noexecstack
on All Architectures by @felixhandte in https://github.com/facebook/zstd/pull/2964
stderr
progress logging for decompression by @terrelln in https://github.com/facebook/zstd/pull/2982
stdout
by @floppym in https://github.com/facebook/zstd/pull/2998
Full Changelog: https://github.com/facebook/zstd/compare/v1.5.1...v1.5.2
Notice : it has been brought to our attention that the v1.5.1
library might be built with an executable stack on non-x64
architectures, which could end up being flagged as problematic by some systems with thorough security settings which disallow executable stack. We are currently reviewing the issue. Be aware of it if you build libzstd
for non-x64
architecture.
Zstandard v1.5.1 is a maintenance release, bringing a good number of small refinements to the project. It also offers a welcome crop of performance improvements, as detailed below.
PRs #2749, #2774, and #2921 refactor single-segment compression for ZSTD_fast
and ZSTD_dfast
, which back compression levels 1 through 4 (as well as the negative compression levels). Speedups in the ~3-5% range are observed. In addition, the compression ratio of ZSTD_dfast
(levels 3 and 4) is slightly improved.
v1.5.0
introduced major speed improvements for mid-level compression (from 5 to 12), while preserving roughly similar compression ratio. As a consequence, the speed scale became tilted towards faster speed. Unfortunately, the difference between successive levels was no longer regular, and there is a large performance gap just after the impacted range, between levels 12 and 13.
v1.5.1
tries to rebalance parameters so that compression levels can be roughly associated to their former speed budget. Consequently, v1.5.1
mid compression levels feature speeds closer to former v1.4.9
(though still sensibly faster) and receive in exchange an improved compression ratio, as shown in below graph.
Note that, since middle levels only experience a rebalancing, save some special cases, no significant performance differences between versions v1.5.0
and v1.5.1
should be expected: levels merely occupy different positions on the same curve. The situation is a bit different for fast levels (1-4), for which v1.5.1
delivers a small but consistent performance benefit on all platforms, as described in previous paragraph.
Our Huffman code was significantly revamped in this release. Both encoding and decoding speed were improved. Additionally, encoding speed for small inputs was improved even further. Speed is measured on the Silesia corpus by compressing with level 1 and extracting the literals left over after compression. Then compressing and decompressing the literals from each block. Measurements are done on an Intel i9-9900K @ 3.6 GHz.
Compiler | Scenario | v1.5.0 Speed | v1.5.1 Speed | Delta |
---|---|---|---|---|
gcc-11 | Literal compression - 128KB block | 748 MB/s | 927 MB/s | +23.9% |
clang-13 | Literal compression - 128KB block | 810 MB/s | 927 MB/s | +14.4% |
gcc-11 | Literal compression - 4KB block | 223 MB/s | 321 MB/s | +44.0% |
clang-13 | Literal compression - 4KB block | 224 MB/s | 310 MB/s | +38.2% |
gcc-11 | Literal decompression - 128KB block | 1164 MB/s | 1500 MB/s | +28.8% |
clang-13 | Literal decompression - 128KB block | 1006 MB/s | 1504 MB/s | +49.5% |
Overall impact on (de)compression speed depends on the compressibility of the data. Compression speed improves from 1-4%, and decompression speed improves from 5-15%.
PR #2722 implements the Huffman decoder in assembly for x86-64 with BMI2 enabled. We detect BMI2 support at runtime, so this speedup applies to all x86-64 builds running on CPUs that support BMI2. This improves Huffman decoding speed by about 40%, depending on the scenario. PR #2733 improves Huffman encoding speed by 10% for clang and 20% for gcc. PR #2732 drastically speeds up the HUF_sort()
function, which speeds up Huffman tree building for compression. This is a significant speed boost for small inputs, measuring in at a 40% improvement for 4K inputs.
zstd
binary size grew significantly in v1.5.0
due to the new code added for middle compression level speed optimizations. In this release we recover the binary size, and in the process also significantly speed up builds, especially with sanitizers enabled.
Measured on x86-64 compiled with -O3
we measure libzstd.a
size. We regained 161 KB of binary size on gcc, and 293 KB of binary size on clang. Note that these binary sizes are listed for the whole library, optimized for speed over size. The decoder only, with size saving options enabled, and compiled with -Os
or -Oz
can be much smaller.
Version | gcc-11 size | clang-13 size |
---|---|---|
v1.5.1 | 1177 KB | 1167 KB |
v1.5.0 | 1338 KB | 1460 KB |
v1.4.9 | 1137 KB | 1151 KB |
x64
assembly, by @terrellnlazy2
levels 11-12, by @senhuang42ZSTD_DCtx
state size, using build macro ZSTD_DECODER_INTERNAL_BUFFER
, at a small cost of performance, by @bindhvom68k
(Motorola 68000's), by @cyan4973AIX
support, by @Helflymmtime
to output file, by @felixhandte-vv
), by @Svetlitski-FB--long-param=
can accept negative value arguments, by @binhdvoZSTD_VecMask_next
: fix incorrect variable name in fallback code path by @dnelson-1901 in https://github.com/facebook/zstd/pull/2657
publish-github-release
Job by @felixhandte in https://github.com/facebook/zstd/pull/2649
ZSTD_row_getMatchMask
by @aqrit in https://github.com/facebook/zstd/pull/2681
README
for Travis CI Badge by @gauthamkrishna9991 in https://github.com/facebook/zstd/pull/2700
S390x
(big endian) by @senhuang42 in https://github.com/facebook/zstd/pull/2678
--progress
flag to properly control progress display and default … by @binhdvo in https://github.com/facebook/zstd/pull/2698
ZSTD_FORCE_DECOMPRESS_*
build macros by @luisdallos in https://github.com/facebook/zstd/pull/2714
ZSTD_Sequence
into seqStore
by @senhuang42 in https://github.com/facebook/zstd/pull/2715
zstd_ldm_geartab
by @danlark1 in https://github.com/facebook/zstd/pull/2719
Z_PREFIX zError
function by @koalabearguo in https://github.com/facebook/zstd/pull/2707
pzstd
: fix linking for static builds by @jonringer in https://github.com/facebook/zstd/pull/2724
mtime
on Output Files by @felixhandte in https://github.com/facebook/zstd/pull/2742
ZSTD_HASHLOG3_MAX
private by @Cyan4973 in https://github.com/facebook/zstd/pull/2752
qemu
tests by @Cyan4973 in https://github.com/facebook/zstd/pull/2758
opt
by @senhuang42 in https://github.com/facebook/zstd/pull/2763
ZSTD_fast
(~+5% Speed) by @felixhandte in https://github.com/facebook/zstd/pull/2749
ZSTD_compressBound()
is respected by @terrelln in https://github.com/facebook/zstd/pull/2776
Make
+ CMake
by @terrelln in https://github.com/facebook/zstd/pull/2783
msvc2019
to build.generic.cmd by @animalize in https://github.com/facebook/zstd/pull/2787
DYNAMIC_BMI2=0 && BMI2
is supported by @terrelln in https://github.com/facebook/zstd/pull/2791
NCountWriteBound
by @senhuang42 in https://github.com/facebook/zstd/pull/2779
MatchState*
parameter when possible by @Cyan4973 in https://github.com/facebook/zstd/pull/2797
-Wfall-through
excepting legacy by @terrelln in https://github.com/facebook/zstd/pull/2796
-Wundef
inside Linux kernel tree by @solbjorn in https://github.com/facebook/zstd/pull/2802
-Werror
to CI by @terrelln in https://github.com/facebook/zstd/pull/2803
NULL
checks by @terrelln in https://github.com/facebook/zstd/pull/2817
ZSTD_c_ldmHashRateLog
bounds check by @terrelln in https://github.com/facebook/zstd/pull/2819
nbCompares
by @terrelln in https://github.com/facebook/zstd/pull/2820
ZSTD_dfast
by @felixhandte in https://github.com/facebook/zstd/pull/2774
dctx
by reutilizing dst buffer by @binhdvo in https://github.com/facebook/zstd/pull/2751
ZSTD_DECODER_INTERNAL_BUFFER
by @Cyan4973 in https://github.com/facebook/zstd/pull/2829
ZSTD_copy16()
uses ZSTD_memcpy() by @animalize in https://github.com/facebook/zstd/pull/2836
zstd_opt.c
by @terrelln in https://github.com/facebook/zstd/pull/2864
-O3
to CFLAGS
by @terrelln in https://github.com/facebook/zstd/pull/2866
-c
by @binhdvo in https://github.com/facebook/zstd/pull/2883
lzcnt
and bmi
target attributes by @terrelln in https://github.com/facebook/zstd/pull/2888
ZDICT_CONTENTSIZE_MIN
restriction for ZDICT_finalizeDictionary
by @terrelln in https://github.com/facebook/zstd/pull/2887
m68k
cpus by @Cyan4973 in https://github.com/facebook/zstd/pull/2896
cmake
windows build by @terrelln in https://github.com/facebook/zstd/pull/2900
FORCE_MEMORY_ACCESS=1
to legacy by @Hello71 in https://github.com/facebook/zstd/pull/2907
-q
when stderr is not a tty by @binhdvo in https://github.com/facebook/zstd/pull/2884
libzstd.pc
Whenever Building the Lib on Unix by @felixhandte in https://github.com/facebook/zstd/pull/2912
NULL
pointer addition by @terrelln in https://github.com/facebook/zstd/pull/2916
xxHash
to latest v0.8.1
by @Cyan4973 in https://github.com/facebook/zstd/pull/2914
x32
compatibility by @Cyan4973 in https://github.com/facebook/zstd/pull/2922
-m32
by @binhdvo in https://github.com/facebook/zstd/pull/2926
Full Changelog: https://github.com/facebook/zstd/compare/v1.5.0...v1.5.1
v1.5.0
is a major release featuring large performance improvements as well as API changes.
1.5.0 introduces a new default match finder for the compression strategies greedy
, lazy
, and lazy2
, (which map to levels 5-12 for inputs larger than 256K). The optimization brings a massive improvement in compression speed with slight perturbations in compression ratio (< 0.5%) and equal or decreased memory usage.
Benchmarked with gcc, on an i9-9900K:
level | silesia.tar speed delta |
enwik7 speed delta |
---|---|---|
5 | +25% | +25% |
6 | +50% | +50% |
7 | +40% | +40% |
8 | +40% | +50% |
9 | +50% | +65% |
10 | +65% | +80% |
11 | +85% | +105% |
12 | +110% | +140% |
On heavily loaded machines with significant cache contention, we have internally measured even larger gains: 2-3x+ speed at levels 5-7. 🚀
The biggest gains are achieved on files typically larger than 128KB. On files smaller than 16KB, by default we revert back to the legacy match finder which becomes the faster one. This default policy can be overriden manually: the new match finder can be forcibly enabled with the advanced parameter ZSTD_c_useRowMatchFinder
, or through the CLI option --[no-]row-match-finder
.
Note: only CPUs that support SSE2
realize the full extent of this improvement.
Improving compression ratio via block splitting is now enabled by default for high compression levels (16+). The amount of benefit varies depending on the workload. Compressing archives comprised of heavily differing files will see more improvement than compression of single files that don’t vary much entropically (like text files/enwik). At levels 16+, we observe no measurable regression to compression speed.
level 22 compression
file | ratio 1.4.9 | ratio 1.5.0 | ratio % delta |
---|---|---|---|
silesia.tar | 4.021 | 4.041 | +0.49% |
calgary.tar | 3.646 | 3.672 | +0.71% |
enwik7 | 3.579 | 3.579 | +0.0% |
The block splitter can be forcibly enabled on lower compression levels as well with the advanced parameter ZSTD_c_splitBlocks
. When forcibly enabled at lower levels, speed regressions can become more notable. Additionally, since more compressed blocks may be produced, decompression speed on these blobs may also see small regressions.
The decompression speed of data compressed with large window settings (such as --long
or --ultra
) has been significantly improved in this version. The gains vary depending on compiler brand and version, with clang
generally benefiting the most.
The following benchmark was measured by compressing enwik9
at level --ultra -22
(with a 128 MB window size) on a core i7-9700K.
Compiler version | D. Speed improvement |
---|---|
gcc-7 | +15% |
gcc-8 | +10 % |
gcc-9 | +5% |
gcc-10 | +1% |
clang-6 | +21% |
clang-7 | +16% |
clang-8 | +16% |
clang-9 | +18% |
clang-10 | +16% |
clang-11 | +15% |
Average decompression speed for “normal” payload is slightly improved too, though the impact is less impressive. Once again, mileage varies depending on exact compiler version, payload, and even compression level. In general, a majority of scenarios see benefits ranging from +1 to +9%. There are also a few outliers here and there, from -4% to +13%. The average gain across all these scenarios stands at ~+4%.
It was already possible to compile libzstd
with multithreading support. But it was an active operation. By default, the make
build script would build libzstd
as a single-thread-only library.
This changes in v1.5.0
.
Now the dynamic library (typically libzstd.so.1
on Linux) supports multi-threaded compression by default.
Note that this property is not extended to the static library (typically libzstd.a
on Linux) because doing so would have impacted the build script of existing client applications (requiring them to add -pthread
to their recipe), thus potentially breaking their build. In order to avoid this disruption, the static library remains single-threaded by default.
Luckily, this build disruption does not extend to the dynamic library, which can be built with multi-threading support while existing applications linking to libzstd.so
and expecting only single-thread capabilities will be none the wiser, and remain completely unaffected.
The idea is that starting from v1.5.0
, applications can expect the dynamic library to support multi-threading should they need it, which will progressively lead to increased adoption of this capability overtime.
That being said, since the locally deployed dynamic library may, or may not, support multi-threading compression, depending on local build configuration, it’s always better to check this capability at runtime. For this goal, it’s enough to check the return value when changing parameter ZSTD_c_nbWorkers
, and if it results in an error, then multi-threading is not supported.
Q: What if I prefer to keep the libraries in single-thread mode only ?
The target make lib-nomt
will ensure this outcome.
Q: Actually, I want both static and dynamic library versions to support multi-threading !
The target make lib-mt
will generate this outcome.
Moving up to the higher digit 1.5
signals an opportunity to extend the stable portion of zstd
public API.
This update is relatively minor, featuring only a few non-controversial newcomers.
ZSTD_defaultCLevel()
indicates which level is default (applied when selecting level 0
). It completes existing
ZSTD_minCLevel()
and ZSTD_maxCLevel()
.
Similarly, ZSTD_getDictID_fromCDict()
is a straightforward equivalent to already promoted ZSTD_getDictID_fromDDict()
.
Zstd-1.4.0 stabilized a new advanced API which allows users to pass advanced parameters to zstd. We’re now deprecating all the old experimental APIs that are subsumed by the new advanced API. They will be considered for removal in the next Zstd major release zstd-1.6.0. Note that only experimental symbols are impacted. Stable functions, like ZSTD_initCStream()
, remain fully supported.
The deprecated functions are listed below, together with the migration. All the suggested migrations are stable APIs, meaning that once you migrate, the API will be supported forever. See the documentation for the deprecated functions for more details on how to migrate.
ZSTD_compress2()
with parameter setters:
ZSTD_compress_advanced()
: Use ZSTD_CCtx_setParameter()
.ZSTD_compress_usingCDict_advanced()
: Use ZSTD_CCtx_setParameter()
and ZSTD_CCtx_refCDict()
.ZSTD_compressStream()
or ZSTD_compressStream2()
with parameter setters:
ZSTD_initCStream_srcSize()
: Use ZSTD_CCtx_setPledgedSrcSize()
.ZSTD_initCStream_usingDict()
: Use ZSTD_CCtx_loadDictionary()
.ZSTD_initCStream_usingCDict()
: Use ZSTD_CCtx_refCDict()
.ZSTD_initCStream_advanced()
: Use ZSTD_CCtx_setParameter()
.ZSTD_initCStream_usingCDict_advanced()
: Use ZSTD_CCtx_setParameter()
and ZSTD_CCtx_refCDict()
.ZSTD_resetCStream()
: Use ZSTD_CCtx_reset()
and ZSTD_CCtx_setPledgedSrcSize()
.ZSTD_compressBegin_advanced()
ZSTD_compressBegin_usingCDict_advanced()
Zstd has slightly re-organized the library layout to move all public headers to the top level lib/
directory. This is for consistency, so all public headers are in lib/
and all private headers are in a sub-directory. If you build zstd from source, this may affect your build system.
lib/common/zstd_errors.h
has moved to lib/zstd_errors.h
.lib/dictBuilder/zdict.h
has moved to lib/zdict.h
.We have moved the scripts in contrib/single_file_libs
to build/single_file_libs
. These scripts, originally contributed by @cwoffenden, produce a single compilation-unit amalgamation of the zstd library, which can be convenient for integrating Zstandard into other source trees. This move reflects a commitment on our part to support this tool and this pattern of using zstd going forward.
We are slightly changing the format of the Windows release .zip
files, to match our other release artifacts. The .zip
files now bundle everything in a single folder whose name matches the archive name. The contents of that folder exactly match what was previously included in the root of the archive.
We have created a signing key for the Zstandard project. This release and all future releases will be signed by this key. See #2520 for discussion.
ZSTD_defaultCLevel()
ZSTD_getDictID_fromCDict()
ZSTD_compress_advanced()
ZSTD_compress_usingCDict_advanced()
ZSTD_compressBegin_advanced()
ZSTD_compressBegin_usingCDict_advanced()
ZSTD_initCStream_srcSize()
ZSTD_initCStream_usingDict()
ZSTD_initCStream_usingCDict()
ZSTD_initCStream_advanced()
ZSTD_initCStream_usingCDict_advanced()
ZSTD_resetCStream()
ZSTDMT_NBWORKERS_MAX
reduced to 64 for 32-bit environments (#2643, @Cyan4973)clang
and for --long
modes (#2614 #2630, @Cyan4973)ZSTD_estimateCCtxSize*()
monotonically increases with compression level (#2538, @senhuang42)zdict.h
dictionary training API documentation (#2622, @terrelln)ZSTD_free*()
functions accept NULL pointers (#2521, @animalize)zstd_errors.h
and zdict.h
to lib/
root (#2597, @terrelln)build/
directory (#2618, @felixhandte)ZSTDMT_JOBSIZE_MIN
to be configured at compile-time, reduce default to 512KB (#2611, @Cyan4973)ZBUFF_*()
is no longer built by default (#2583, @senhuang42)md5
on Darwin (#2609, @felixhandte)--progress
flag added to always display progress bar (#2595, @senhuang42)--force
(#2613, @felixhandte)--filelist
end-of-line bug (#2620, @Cyan4973)This is an incremental release which includes various improvements and bug-fixes.
Long Distance Mode (LDM) --long
just got a whole lot faster thanks to optimizations by @mpu in #2483! These optimizations preserve the compression ratio but drastically speed up compression. It is especially noticeable in multithreaded mode, because the long distance match finder is not parallelized. Benchmarking with zstd -T0 -1 --long=31
on an Intel I9-9900K at 3.2 GHz we see:
File | v1.4.8 MB/s | v1.4.9 MB/s | Improvement |
---|---|---|---|
silesia.tar | 308 | 692 | 125% |
linux-versions* | 312 | 667 | 114% |
enwik9 | 294 | 747 | 154% |
* linux-versions
is a concatenation of the linux 4.0, 5.0, and 5.10 git archives.
ZSTD_d_refMultipleDDicts
If the advanced parameter ZSTD_d_refMultipleDDicts
is enabled, then multiple calls to ZSTD_refDDict()
will be honored in the corresponding DCtx
. Example usage:
ZSTD_DCtx* dctx = ZSTD_createDCtx();
ZSTD_DCtx_setParameter(dctx, ZSTD_d_refMultipleDDicts, ZSTD_rmd_refMultipleDDicts);
ZSTD_DCtx_refDDict(dctx, ddict1);
ZSTD_DCtx_refDDict(dctx, ddict2);
ZSTD_DCtx_refDDict(dctx, ddict3);
...
ZSTD_decompress...
Decompression of multiple frames, each with their own dictID
, is now possible with a single ZSTD_decompress
call. As long as the dictID
from each frame header references one of the dictID
s within the DCtx
, then the corresponding dictionary will be used to decompress that particular frame. Note that this feature is disabled with a statically-allocated DCtx
.
umask()
to Constrain Created File Permissions (#2495, @felixhandte)--output-dir-mirror
's Rejection of ..
-Containing Paths (#2512, @felixhandte)-f
/--force
is Passed (#2466, @felixhandte)stat -c
on NetBSD (#2513, @felixhandte)libzstd
(#2457, #2454 @Cyan4973)configure_file()
to Create the .pc
File (#2462, @lazka)_LARGEFILE_SOURCE
and _LARGEFILE64_SOURCE
(#2444, @indygreg)zlibwrapper
Makefile (#2437, @Cyan4973)recover_directory
Program (#2473, @terrelln)This is a minor hotfix for v1.4.7
,
where an internal buffer unalignment bug was detected by @bmwiedemann .
The issue is of no consequence for x64
and arm64
targets,
but could become a problem for cpus relying on strict alignment, such as mips
or older arm
designs.
Additionally, some targets, like 32-bit x86
cpus, do not care much about alignment, but the code does, and will detect the misalignment and return an error code. Some other less common platforms, such as s390x
, also seem to trigger the same issue.
While it's a minor fix, this update is nonetheless recommended.
Note : this version features a minor bug, which can be present on systems others than x64
and arm64
. Update v1.4.8
is recommended for all other platforms.
v1.4.7
unleashes several months of improvements across many axis, from performance to various fixes, to new capabilities, of which a few are highlighted below. It’s a recommended upgrade.
(Note: if you ever wondered what happened to v1.4.6
, it’s an internal release number reserved for synchronization with Linux Kernel)
--long
mode--long
mode makes it possible to analyze vast quantities of data in reasonable time and memory budget. The --long
mode algorithm runs on top of the regular match finder, and both contribute to the final compressed outcome.
However, the fact that these 2 stages were working independently resulted in minor discrepancies at highest compression levels, where the cost of each decision must be carefully monitored. For this reason, in situations where the input is not a good fit for --long
mode (no large repetition at long distance), enabling it could reduce compression performance, even if by very little, compared to not enabling it (at high compression levels). This situation made it more difficult to "just always enable" the --long
mode by default.
This is fixed in this version. For compression levels 16 and up, usage of --long
will now never regress compared to compression without --long
. This property made it possible to ramp up --long
mode contribution to the compression mix, improving its effectiveness.
The compression ratio improvements are most notable when --long
mode is actually useful. In particular, --patch-from
(which implicitly relies on --long
) shows excellent gains from the improvements. We present some brief results here (tested on Macbook Pro 16“, i9).
Since --long
mode is now always beneficial at high compression levels, it’s now automatically enabled for any window size >= 128MB and up.
This release includes optimizations that significantly speed up decompression of small blocks and small data. The decompression speed gains will vary based on the block size according to the table below:
Block Size | Decompression Speed Improvement |
---|---|
1 KB | ~+30% |
2 KB | ~+30% |
4 KB | ~+25% |
8 KB | ~+15% |
16 KB | ~+10% |
32 KB | ~+5% |
These optimizations come from improving the process of reading the block header, and building the Huffman and FSE decoding tables. zstd
’s default block size is 128 KB, and at this block size the time spent decompressing the data dominates the time spent reading the block header and building the decoding tables. But, as blocks become smaller, the cost of reading the block header and building decoding tables becomes more prominent.
The CLI received several noticeable upgrades with this version.
To begin with, zstd
can accept a new parameter through environment variable, ZSTD_NBTHREADS
. It’s useful when zstd
is called behind an application (tar
, or a python script for example). Also, users which prefer multithreaded compression by default can now set a desired nb of threads with their environment. This setting can still be overridden on demand via command line.
A new command --output-dir-mirror
makes it possible to compress a directory containing subdirectories (typically with -r
command) producing one compressed file per source file, and reproduce the arborescence into a selected destination directory.
There are other various improvements, such as more accurate warning and error messages, full equivalence between conventions --long-command=FILE
and --long-command FILE
, fixed confusion risks between stdin
and user prompt, or between console output and status message, as well as a new short execution summary when processing multiple files, cumulatively contributing to a nicer command line experience.
By default, each compression context can be set to use a maximum nb of threads. In complex scenarios, there might be multiple compression contexts, working in parallel, and each using some nb of threads. In such cases, it might be desirable to control the total nb of threads used by all these compression contexts altogether.
This is now possible, by making all these compression contexts share the same threadpool. This capability is expressed thanks to a new advanced compression parameter, ZSTD_CCtx_refThreadPool()
, contributed by @marxin. See its documentation for more details.
This release introduces a new experimental dictionary compression algorithm, applicable to mid-range compression levels, employing strategies such as ZSTD_greedy
, ZSTD_lazy
, and ZSTD_lazy2
. This new algorithm can be triggered by selecting the compression parameter ZSTD_c_enableDedicatedDictSearch
during ZSTD_CDict
creation (experimental section).
Benchmarks show the new algorithm providing significant compression speed gains :
Level | Hot Dict | Cold Dict |
---|---|---|
5 | ~+17% | ~+30% |
6 | ~+12% | ~+45% |
7 | ~+13% | ~+40% |
8 | ~+16% | ~+50% |
9 | ~+19% | ~+65% |
10 | ~+24% | ~+70% |
We hope it will help making mid-levels compression more attractive for dictionary scenarios. See the documentation for more details. Feedback is welcome!
We introduce a new entry point, ZSTD_compressSequences()
, which makes it possible for users to define their own sequences, by whatever mechanism they prefer, and present them to this new entry point, which will generate a single zstd
-compressed frame, based on provided sequences.
So for example, users can now feed to the function an array of externally generated ZSTD_Sequence
:
[(offset: 5, matchLength: 4, litLength: 10), (offset: 7, matchLength: 6, litLength: 3), ...]
and the function will output a zstd compressed frame based on these sequences.
This experimental API has currently several limitations (and its relevant params exist in the “experimental” section). Notably, this API currently ignores any repeat offsets provided, instead always recalculating them on the fly. Additionally, there is no way to forcibly specify existence of certain zstd features, such as RLE or raw blocks.
If you are interested in this new entry point, please refer to zstd.h
for more detailed usage instructions.
There are many other features and improvements in this release, and since we can’t highlight them all, they are listed below:
--long
mode at high compression levels, by @senhuang42--patch-from
at high compression levels, thanks to --long
improvementsZSTD_compress2()
, by @terrellnZSTD_NBTHREADS
, by @senhuang42--output-dir-mirror DIR
command, by @xxie24 (#2219)*.zstd
suffix--patch-from
can compress stdin
when used with --stream-size
, by @bimbashrestha (#2206)stdin
input can no longer be confused with user promptstdout
and status messagesZSTD_getDictID_fromCDict()
, by @LuAPidstSize_tooSmall
error whenever appropriateZSTD_initCStream_advanced()
with static allocation and no dictionaryZSTD_NO_INTRINSICS
macro to avoid explicit intrinsicsSTATIC_BMI2
macro for compile time detection of BMI2 on MSVC, by @Niadb (#2258)contrib/freestanding_lib
, by @terrellnThis is a fairly important release which includes performance improvements and new major CLI features. It also fixes a few corner cases, making it a recommended upgrade.
Decompression speed has been improved again, thanks to great contributions from @terrelln.
As usual, exact mileage varies depending on files and compilers.
For x64
cpus, expect a speed bump of at least +5%, and up to +10% in favorable cases.
ARM
cpus receive more benefit, with speed improvements ranging from +15% vicinity, and up to +50% for certain SoCs and scenarios (ARM
‘s situation is more complex due to larger differences in SoC designs).
For illustration, some benchmarks run on a modern x64
platform using zstd -b
compiled with gcc
v9.3.0 :
v1.4.4 | v1.4.5 | |
---|---|---|
silesia.tar | 1568 MB/s | 1653 MB/s |
--- | --- | --- |
enwik8 | 1374 MB/s | 1469 MB/s |
calgary.tar | 1511 MB/s | 1610 MB/s |
Same platform, using clang
v10.0.0 compiler :
v1.4.4 | v1.4.5 | |
---|---|---|
silesia.tar | 1439 MB/s | 1496 MB/s |
--- | --- | --- |
enwik8 | 1232 MB/s | 1335 MB/s |
calgary.tar | 1361 MB/s | 1457 MB/s |
Presuming a project needs to integrate libzstd
's source code (as opposed to linking a pre-compiled library), the /lib
source directory can be copy/pasted into target project. Then the local build system must setup a few include directories. Some setups are automatically provided in prepared build scripts, such as Makefile
, but any other 3rd party build system must do it on its own.
This integration is now simplified, thanks to @felixhandte, by making all dependencies within /lib
relative, meaning it’s only necessary to setup include directories for the *.h
header files that are directly included into target project (typically zstd.h
). Even that task can be circumvented by copy/pasting the *.h
into already established include directories.
Alternatively, if you are a fan of one-file integration strategy, @cwoffenden has extended his one-file decoder script into a full feature one-file compression library. The script create_single_file_library.sh
will generate a file zstd.c
, which contains all selected elements from the library (by default, compression and decompression). It’s then enough to import just zstd.h
and the generated zstd.c
into target project to access all included capabilities.
--patch-from
Zstandard CLI is introducing a new command line option --patch-from
, which leverages existing compressors, dictionaries and long range match finder to deliver a high speed engine for producing and applying patches to files.
--patch-from
is based on dictionary compression. It will consider a previous version of a file as a dictionary, to better compress a new version of same file. This operation preserves fast zstd
speeds at lower compression levels. To this ends, it also increases the previous maximum limit for dictionaries from 32 MB to 2 GB, and automatically uses the long range match finder when needed (though it can also be manually overruled).
--patch-from
can also be combined with multi-threading mode at a very minimal compression ratio loss.
Example usage:
# create the patch
zstd --patch-from=<oldfile> <newfile> -o <patchfile>
# apply the patch
zstd -d --patch-from=<oldfile> <patchfile> -o <newfile>`
Benchmarks:
We compared zstd
to bsdiff
, a popular industry grade diff engine. Our test corpus were tarballs of different versions of source code from popular GitHub repositories. Specifically:
`repos = {
# ~31mb (small file)
"zstd": {"url": "https://github.com/facebook/zstd", "dict-branch": "refs/tags/v1.4.2", "src-branch": "refs/tags/v1.4.3"},
# ~273mb (medium file)
"wordpress": {"url": "https://github.com/WordPress/WordPress", "dict-branch": "refs/tags/5.3.1", "src-branch": "refs/tags/5.3.2"},
# ~1.66gb (large file)
"llvm": {"url": "https://github.com/llvm/llvm-project", "dict-branch": "refs/tags/llvmorg-9.0.0", "src-branch": "refs/tags/llvmorg-9.0.1"}
}`
--patch-from
on level 19 (with chainLog=30 and targetLength=4kb) is comparable with bsdiff
when comparing patch sizes.
--patch-from
greatly outperforms bsdiff
in speed even on its slowest setting of level 19 boasting an average speedup of ~7X. --patch-from
is >200X faster on level 1 and >100X faster (shown below) on level 3 vs bsdiff
while still delivering patch sizes less than 0.5% of the original file size.
And of course, there is no change to the fast zstd decompression speed.
Addendum :
After releasing --patch-from
, we were made aware of two other popular diff engines by the community: SmartVersion and Xdelta. We ran some additional benchmarks for them and here are our primary takeaways. All three tools are excellent diff engines with clear advantages (especially in speed) over the popular bsdiff. Patch sizes for both binary and text data produced by all three are pretty comparable with Xdelta underperforming Zstd and SmartVersion only slightly [1]. For patch creation speed, Xdelta is the clear winner for text data and Zstd is the clear winner for binary data [2]. And for Patch Extraction Speed (ie. decompression), Zstd is fastest in all scenarios [3]. See wiki for details.
--filelist=
Finally, --filelist=
is a new CLI capability, which makes it possible to pass a list of files to operate upon from a file,
as opposed to listing all target files solely on the command line.
This makes it possible to prepare a list offline, save it into a file, and then provide the prepared list to zstd
.
Another advantage is that this method circumvents command line size limitations, which can become a problem when operating on very large directories (such situation can typically happen with shell expansion).
In contrast, passing a very large list of filenames from within a file is free of such size limitation.
ZSTD_DCtx
when too large for too long (#2069, by @bimbashrestha)aarch64
(#2040, ~+3%, by @caoyzh)--ultra
) and multithreading, by @terrellnZDICT_finalizeDictionary()
is promoted to stable (#2111)ZSTD_d_stableOutBuffer
(#2094)libzstd
library (#2065, by @cwoffenden)-I
flags for zstd
lib subdirs (#2103, by @felixhandte)zstd
now compiles cleanly under -pedantic
(#2099)zstd
now compiles with make-4.3mingw
cross-compilation from Linux, by @Ericson2314icc
fixes backed by new ci test on travis--patch-from
command, create and apply patches from files, by @bimbashrestha--filelist=
: Provide a list of files to operate upon from a file-b
can now benchmark multiple files in decompression mode--no-content-size
command--show-default-cparams
commandchecked_flipped_bits
, in contrib/
, by @felixhandtecontrib/
CONTRIBUTING.md
docs