Python bindings to the Zstandard (zstd) compression library
PyType_Spec
and corresponding APIs. (#187) Contributed by
Mike Hommey.ZstdDecompressor.decompressobj()
will change read_across_frames
to
default to True
in a future release. If you depend on the current
functionality of stopping at frame boundaries, start explicitly passing
read_across_frames=False
to preserve the current behavior.manylinux2010
wheels are no longer published since this wheel format
is no longer supported by the pypa/manylinux project.setup.py
will likely yield a working install. However, this is no officially supported.ZstdDecompressor.decompressobj()
now accepts a read_across_frames
boolean named argument to control whether to transparently read across
multiple zstd frames. It defaults to False
to preserve existing
behavior.pyproject.toml
now declares a [build-system]
section saying to build
with setuptools.setup.py
directly.ZSTD_copyDCtx()
was removed from the C and
Rust backends.ZstdDecompressionObj.decompress()
could
have raised an assertion in cases where the function was called multiple
times on an instance. In non-debug builds, calls to this method could have
leaked memory.compiler.preprocessor
if it
is set. (#179)ZstdDecompressionObj.decompress()
was refactored.
This may have fixed unconfirmed issues where unused_data
was set
prematurely. The new logic will also avoid an extra call to
ZSTD_decompressStream()
in some scenarios, possibly improving performance.ZstdDecompressor.decompress()
how has a read_across_frames
keyword
argument. It defaults to False. True is not yet implemented and will raise an
exception if used. The new argument will default to True in a future release
and is provided now so callers can start passing read_across_frames=False
to preserve the existing functionality during a future upgrade.ZstdDecompressor.decompress()
now has an allow_extra_data
keyword
argument to control whether an exception is raised if input contains extra
data. It defaults to True, preserving existing behavior of ignoring extra
data. It will likely default to False in a future release. Callers desiring
the current behavior are encouraged to explicitly pass
allow_extra_data=True
so behavior won't change during a future upgrade.manylinux2014_aarch64
wheels are now being produced for CPython 3.6+. (#145).ZstdCompressor()
and ZstdDecompressor()
are now all optional in the C backend and an explicit None
value is accepted. Before, the C backend wouldn't accept an explicit None
value (but the CFFI backend would). The new behavior should be consistent between the backends. (#153)ZstdCompressor.multi_compress_to_buffer()
and
ZstdDecompressor.multi_decompress_to_buffer()
are no longer
available when linking against a system zstd library. These
experimental features are only available when building against the
bundled single file zstd C source file distribution. (#106)setup.py
now recognizes a ZSTD_EXTRA_COMPILER_ARGS
environment variable to specify additional compiler arguments
to use when compiling the C backend..pyi
type annotations file has replaced various default argument
values with ...
.setup.py
no longer attempts to build the C backend on PyPy. (#130)<sys/types.h>
is now included before <sys/sysctl.h>
. This was
the case in releases prior to 0.15.0 and the include order was reversed
as part of running clang-format
. The old/working order has been
restored. (#128)train_dictionary()
now uses the fastcover
training mechanism
(as opposed to cover
). Some parameter values that worked with the old
mechanism may not work with the new one. e.g. d
must be 6
or 8
if it is defined.train_dictionary()
now always calls
ZDICT_optimizeTrainFromBuffer_fastCover()
instead of different APIs
depending on which arguments were passed.zstandard.backend_c
instead of zstd
. The
CFFI extension module is now built as zstandard._cffi
instead of
_zstd_cffi
. The CFFI backend is now zstandard.backend_cffi
instead
of zstandard.cffi
.ZstdDecompressionReader.seekable()
now returns False
instead of
True
because not all seek operations are supported and some Python
code in the wild keys off this value to determine if seek()
can be
called for all scenarios.ZstdDecompressionReader.seek()
now raises OSError
instead of
ValueError
when the seek cannot be fulfilled. (#107)ZstdDecompressionReader.readline()
and
ZstdDecompressionReader.readlines()
now accept an integer argument.
This makes them conform with the IO interface. The methods still raise
io.UnsupportedOperation
.ZstdCompressionReader.__enter__
and ZstdDecompressionReader.__enter__
now raise ValueError
if the instance was already closed.overlap_size_log
attribute on ZstdCompressionParameters
instances has been removed. The overlap_log
attribute should be used
instead.overlap_size_log
argument to ZstdCompressionParameters
has been removed. The overlap_log
argument should be used instead.ldm_hash_every_log
attribute on
ZstdCompressionParameters
instances has been removed. The
ldm_hash_rate_log
attribute should be used instead.ldm_hash_every_log
argument to
ZstdCompressionParameters
has been removed. The ldm_hash_rate_log
argument should be used instead.CompressionParameters
type alias to
ZstdCompressionParamaters
has been removed. Use
ZstdCompressionParameters
.ZstdCompressor.read_from()
and
ZstdDecompressor.read_from()
have been removed. Use the corresponding
read_to_iter()
methods instead.ZstdCompressor.write_to()
and
ZstdDecompressor.write_to()
have been removed. Use the corresponding
stream_writer()
methods instead.ZstdCompressor.copy_stream()
, ZstdCompressorIterator.__next__()
,
and ZstdDecompressor.copy_stream()
now raise the original exception
on error calling the source stream's read()
instead of raising
ZstdError
. This only affects the C backend.ZstdDecompressionObj.flush()
now returns bytes
instead of
None
. This makes it behave more similarly to flush()
methods
for similar types in the Python standard library. (#78)ZstdCompressionWriter.__exit__()
now always calls close()
.
Previously, close()
would not be called if the context manager
raised an exception. The old behavior was inconsistent with other
stream types in this package and with the behavior of Python's
standard library IO types. (#86)cffi
as an install_requires
except when running on PyPy. Instead, cffi
is listed as an
extras_require
.ZstdCompressor.stream_reader()
and ZstdDecompressor.stream_reader()
now default to closing the source stream when the instance is itself
closed. To change this behavior, pass closefd=False
. (#76)CFFI
backend now defines
ZstdCompressor.multi_compress_to_buffer()
and
ZstdDecompressor.multi_decompress_to_buffer()
. However, they
raise NotImplementedError
, as they are not yet implemented.CFFI
backend now exposes the types ZstdCompressionChunker
,
ZstdCompressionObj
, ZstdCompressionReader
,
ZstdCompressionWriter
, ZstdDecompressionObj
,
ZstdDecompressionReader
, and ZstdDecompressionWriter
as
symbols on the zstandard
module.CFFI
backend now exposes the types BufferSegment
,
BufferSegments
, BufferWithSegments
, and
BufferWithSegmentsCollection
. However, they are not implemented.ZstdCompressionWriter.flush()
now calls flush()
on the inner stream
if such a method exists. However, when close()
itself calls
self.flush()
, flush()
is not called on the inner stream.ZstdDecompressionWriter.close()
no longer calls flush()
on
the inner stream. However, ZstdDecompressionWriter.flush()
still
calls flush()
on the inner stream.ZstdCompressor.stream_writer()
and ZstdDecompressor.stream_writer()
now have their write_return_read
argument default to True
.
This brings the behavior of write()
in compliance with the
io.RawIOBase
interface by default. The argument may be removed
in a future release.ZstdCompressionParameters
no longer exposes a compression_strategy
property. Its constructor no longer accepts a compression_strategy
argument. Use the strategy
property/argument instead.stream_reader
decompressor when reader is closed
before reading everything. (Patch by Pierre Fersing.)ZstdCompressionWriter.write()
now catches exceptions when calling the inner stream's write()
.
ZstdCompressionWriter.flush()
on inner stream's write()
.
ZstdCompressor.copy_stream()
on dest stream's write()
.
ZstdDecompressionWriter.write()
on inner stream's write()
.
ZstdDecompressor.copy_stream()
on dest stream's write()
. (#102)PyBuffer
instances are no longer checked to be C contiguous and
have a single dimension. The former was redundant with what
PyArg_ParseTuple()
already did and the latter is not necessary
in practice because very few extension modules create buffers with
more than 1 dimension. (#124)zstandard
module. (#120)make_cffi.py
script should now respect the CC
environment
variable for locating the compiler. (#103)cffi
backend when running all tests.train_dictionary()
has been rewritten to use the fastcover
APIs
and to consistently call ZDICT_optimizeTrainFromBuffer_fastCover()
instead of different C APIs depending on what arguments were passed.
The function also now accepts arguments f
, split_point
, and
accel
, which are parameters unique to fastcover
.zstd.c
file renamed to c-ext/backend_c.c
.zstandard
package. Previously, there were modules in other packages. (#115)clang-format
.ZstdCompressor.stream_writer()
, ZstdCompressor.stream_reader()
,
ZstdDecompressor.stream_writer()
, and
ZstdDecompressor.stream_reader()
now accept a closefd
argument to control whether the underlying stream should be closed
when the ZstdCompressionWriter
, ZstdCompressReader
,
ZstdDecompressionWriter
, or ZstdDecompressionReader
is closed.
(#76)zstandard.open()
function for returning a file
object with zstd (de)compression. (#64)zstandard
module now exposes a backend_features
attribute containing a set of strings denoting optional features
present in that backend. This can be used to sniff feature support
by performing a string lookup instead of sniffing for API presence
or behavior.ZstdCompressionParameters
now exposes a strategy
property.compress()
and decompress()
convenience functions
on the zstandard
module. These are simply wrappers around the
corresponding APIs on ZstdCompressor
and ZstdDecompressor
.This release should contain no functional changes compared to 0.14.0. The only expected difference with this version is Python 3.9 wheels are now being produced.
make_cffi.py
script used to build the CFFI bindings now
calls distutils.sysconfig.customize_compiler()
so compiler
customizations (such as honoring the CC
environment variable)
are performed. Patch by @Arfrever. (#103)make_cffi.py
script now sets LC_ALL=C
when invoking
the preprocessor in an attempt to normalize output to ASCII. (#95)setup.py
is now executable.