Introduction of a compression package

Both the zstd and zstandard import names are claimed by projects on PyPI. To avoid breaking users of one of the existing bindings, this PEP proposes introducing a new namespace for compression libraries, compression. This name is already reserved on PyPI for use in the standard library. The new Zstandard module will be named compression.zstd. Other compression modules will be re-exported in the new compression package.

Providing a common namespace for compression modules has several advantages. First, it reduces user confusion about where to find compression modules. Second, the top level compression module could provide information on which compression formats are available, similar to hashlib’s algorithms_available. If PEP 775 is accepted, a compression.algorithms_guaranteed could be provided as well, listing zlib. Finally, a compression namespace prevents future issues with merging other compression formats into the standard library. New compression formats will likely be published to PyPI prior to integration into CPython. Therefore, any new compression format import name will likely already be claimed by the time a module would be considered for inclusion in CPython. Putting compression modules under a package prefix prevents issues with potential future name clashes.

Code that would like to remain compatible across Python versions may use the following pattern to ensure compatibility:

try:
    from compression.lzma import LZMAFile
except ImportError:
    from lzma import LZMAFile

This will use the newer import name when available and fall back to the old name otherwise.

Implementation based on pyzstd

The implementation for this PEP is based on the pyzstd project. This project was chosen as the code was originally written to be upstreamed to CPython by Ma Lin, who also wrote the output buffer implementation used in the standard library today. The project has since been taken over by Rogdham and is published to PyPI. The APIs in pyzstd are similar to the APIs for other compression modules in the standard library such as bz2 and lzma.

Minimum supported Zstandard version

The minimum supported Zstandard was chosen as v1.4.5, released in May of 2020. This version was chosen as a minimum based on reviewing the versions of Zstandard available in a number of Linux distribution package repositories, including LTS releases. This version choice is rather conservative to maximize compatibility with existing LTS Linux distributions, but a newer Zstandard version could likely be chosen given that newer Python releases are generally packaged as part of newer distribution releases.

The compression namespace

A new namespace for compression modules will be introduced named compression. The top-level module for this package will be empty to begin with, but a standard API for interacting with compression routines may be added in the future to the toplevel.

The compression.zstd module

A new module, compression.zstd will be introduced with Zstandard compression APIs that match other compression modules in the standard library, namely

• compress() / decompress() - APIs for one-shot compression or decompression
• ZstdFile / open() - APIs for interacting with streams and file-like objects
• ZstdCompressor / ZstdDecompressor - APIs for incremental compression or decompression

It will also contain some Zstandard-specific functionality:

• ZstdDict / train_dict() / finalize_dict() - APIs for interacting with Zstandard dictionaries, which are useful for compressing many small chunks of similar data

libzstd optional dependency

The libzstd library will become an optional dependency of CPython. If the library is not available, the compression.zstd module will be unavailable. This is handled automatically on Unix platforms as part of the normal build environment detection.

On Windows, libzstd will be added to the source dependencies used to build libraries CPython depends on for Windows.

Other compression modules

New import names compression.lzma, compression.bz2, compression.gzip and compression.zlib will be introduced in Python 3.14 re-exporting the contents of the existing lzma, bz2, gzip and zlib modules respectively. The compression sub-modules will become the canonical import names going forward. The use of the new compression names will be promoted over the original top level module names in the Python documentation when the minimum supported Python version requirements make that feasible.

The _compression module, given that it is marked private, will be immediately renamed to compression._common._streams. The new name was selected due to the current contents of the module being I/O related helpers for stream APIs (e.g. LZMAFile) in standard library compression modules.

Name the module zstdlib and do not make a new compression namespace

One option instead of making a new compression namespace would be to find a different name, such as zstdlib, as the import name. Several other names, such as zst, libzstd, and zstdcomp were proposed as well. In discussion, the names were found to either be too easy to typo, or unintuitive. Furthermore, the issue of existing import names is likely to persist for future compression formats added to the standard library. LZ4, a common high speed compression format, has a package on PyPI, lz4, with the import name lz4. Instead of solving this issue for each compression format, it is better to solve it once and for all by using the already-claimed compression namespace.

Introduce an experimental _zstd package in Python 3.14

Since this PEP was published close to the beta cutoff for new features for Python 3.14, one proposal was to name the package a private module _zstd so that packaging tools could use it sooner, but not deciding on a name. This would allow more time for discussion of the final module name during the 3.15 development window. However, introducing a private module was not popular. The expectations and contract for external usage of a private module in the standard library are unclear.

Introduce a standard library namespace instead of compression

One alternative to a compression namespace would be to introduce a std namespace for the entire standard library. However, this was seen as too significant a change for 3.14, with no agreed upon semantics, migration path, or name for the package. Furthermore, a future PEP introducing a std namespace could always define that the compression sub-modules be flattened into the std namespace.

Include zipfile and tarfile in compression

Compression is often used with archiving tools, so putting both zipfile and tarfile under the compression namespace is appealing. However, compression can be used beyond just archiving tools. For example, network requests can be gzip compressed. Furthermore, formats like tar do not include compression themselves, instead relying on external compression. Therefore, this PEP does not propose moving zipfile or tarfile under compression.

Do not include gzip under compression

The GZip format RFC defines a format which can include multiple blocks and metadata about its contents. In this way GZip is rather similar to archive formats like ZIP and tar. Despite that, in usage GZip is often treated as a compression format rather than an archive format. Looking at how different languages classify GZip, the prevailing trend is to classify it as a compression format and not an archiving format.

In addition, the gzip module in Python mostly focuses on single block content and has an API similar to other compression modules, making it a good fit for the compression namespace.