Description

emmatyping

opened on Apr 26, 2025

Issue body actions

Feature or enhancement

This is a tracking issue for implementing PEP 784. See the PEP text for more details.

Since the diff is significant (~10k lines) I wanted to split up the PRs a bit.

Implementation Plan:

• Add compression module just re-exporting existing compression modules. Move the _compression module.
• Add _zstd native module with Unix build config
• Add Windows build config for _zstd (blocked on adding libzstd to cpython-source-deps) and SBOM config.
• Add zstd Python module with tests
• add NEWS/What's New section
• Add documentation for zstd
• Decide on whether or not to check parameter types for options
• Verify style conformance of C and Python code
  • Python
  • C
• Refactor _train_dict and _finalize_dict to share common code
• Fix test_zstd failed on ubuntu with free-threading #133885
• Improve error message when options value can't be converted to int

Linked PRs

• gh-132983: Introduce compression package and move _compression module #133018
• gh-132983: Introduce _zstd bindings module #133027
• gh-132983: Install compression package contents #133063
• gh-132983: make compression._common a full-fledged package #133076
• gh-132983: Convert compression._common to a package #133086
• gh-132983: [DRAFT] Test _zstd code with Python test suite #133185
• gh-132983: [Draft] Run refleak tests against _zstd with tests #133282
• gh-132983: Add compression.zstd and Python tests #133365
• GH-132983: Build _zstd on Windows #133366
• GH-132983: Use a standard autoconf version check for zstd #133479
• gh-132983: Add What's New entry for PEP 784 implementation #133495
• GH-132983: Remove zstd version check in the header file #133502
• gh-132983: Remove pyzstd in identifiers #133535
• gh-132983: Style improvements for compression.zstd #133547
• GH-132983: Support finding libzstd without pkg-config #133550
• GH-132983: Restore libzstd fallback detection #133565
• [3.14] gh-132983: Remove pyzstd in identifiers (GH-133535) #133629
• gh-132983: Clean-ups for _zstd #133670
• gh-103092: Support subinterpreters in _zstd #133674
• GH-132983: Remove subclassing support from zstd types #133694
• [3.14] gh-103092: Support subinterpreters in _zstd (GH-133674) #133695
• gh-132983: Fix small issues with zstd support in zipfile #133723
• gh-132983: Don't allow trailer data in ZstdFile #133736
• [3.14] gh-132983: Clean-ups for _zstd (GH-133670) #133756
• [3.14] GH-132983: Restore libzstd fallback detection (GH-133565) #133757
• [3.14] GH-132983: Remove subclassing support from zstd types (GH-133694) #133762
• gh-132983: Simplify _zstd_exec() #133775
• gh-132983: Make zstd types immutable #133784
• GH-132983: remove empty_bytes from _zstd module state #133785
• [3.14] gh-132983: Simplify _zstd_exec() (GH-133775) #133786
• [3.14] GH-132983: remove empty_bytes from _zstd module state (GH-133785) #133788
• GH-132983: PEP 7 and Argument Clinic changes for zstd #133791
• [3.14] GH-132983: PEP 7 and Argument Clinic changes for zstd (GH-133791) #133792
• gh-132983: Reduce the size of _zstdmodule.h #133793
• [3.14] gh-132983: Don't allow trailer data in ZstdFile (GH-133736) #133799
• [3.14] gh-132983: Reduce the size of _zstdmodule.h (GH-133793) #133854
• gh-132983: Remove EndlessZstdDecompressor remains #133856
• [3.14] gh-132983: Make zstd types immutable (GH-133784) #133857
• [3.14] gh-132983: Remove leftovers from EndlessZstdDecompressor (GH-133856) #133859
• gh-132983: Convert zstd __new__ methods to Argument Clinic #133860
• gh-132983: Add documentation for compression.zstd #133911
• [3.14] gh-132983: Convert zstd __new__ methods to Argument Clinic (GH-133860) #133915
• gh-132983: Split _zstd_set_c_parameters #133921
• gh-132983: Convert dict_content to take Py_buffer #133924
• gh-132983: Fix zstd compiler warning about unused function mt_continue_should_break() #133947
• [3.14] gh-132983: Fix compiler warning about unused function mt_continue_should_break() (GH-133947) #133950
• gh-132983: Call Py_XDECREF rather than PyObject_GC_Del in failed __new__ #133962
• [3.14] gh-132983: Fix small issues with zstd support in zipfile (GH-133723) #133974
• [3.14] gh-132983: Style improvements for compression.zstd (GH-133547) #134001
• gh-132983: Add zstd version info to test.pythoninfo #134230
• [3.14] gh-132983: Call Py_XDECREF rather than PyObject_GC_Del in failed __new__ (GH-133962) #134305
• [3.14] gh-132983: Add documentation for compression.zstd (GH-133911) #134425
• gh-132983: Refactor shared code in train_dict and finalize_dict #134432
• [3.14] gh-132983: Refactor shared code in train_dict and finalize_dict (GH-134432) #134442
• gh-132983: Fix refleak in zstd dictionary functions #134459
• [3.14] gh-132983: Fix refleak in zstd dictionary functions (gh-134459) #134463
• gh-132983: Slightly tweak error messages for _zstd compressor/decompressor options dict #134601
• [3.14] gh-132983: Slightly tweak error messages for _zstd compressor/decompressor options dict (GH-134601) #134602
• gh-132983: Make _zstd C code PEP 7 compliant #134605
• [3.14] gh-132983: Make _zstd C code PEP 7 compliant (GH-134605) #134609
• [3.14] gh-132983: Convert dict_content to take Py_buffer in ZstdDict() (GH-133924) #134723
• [3.14] gh-132983: Split _zstd_set_c_parameters (GH-133921) #134838
• gh-132983: Minor fixes and clean up for the _zstd module #134930
• [3.14] gh-132983: Minor fixes and clean up for the _zstd module (GH-134930) #134998
• gh-132983: Add missing references to Zstandard in shutil docstrings #136617
• [3.14] gh-132983: Add missing references to Zstandard in shutil docstrings (GH-136617) #137052
• gh-132983: _zstd: use Py_UNREACHABLE in _zstd_load_impl #137320
• gh-132983: Fix docstrings in ZstdDict #137321
• [3.14] gh-132983: Fix docstrings in ZstdDict (GH-137321) #137343
• [3.14] gh-132983: Use Py_UNREACHABLE in _zstd_load_impl() (GH-137320) #137360

Reacted by Gregory P. Smith, Augustin Žídek, owocado and James Lamb

Reacted by Hugo van Kemenade, Prince Roshan, Stan Ulbrych, devdanzin, Brian Schubert, Rogdham, Gregory P. Smith, W. H. Wang, R1kaB3rN, Máté Gyöngyösi and 3 more

Reacted by Prince Roshan

Activity

1. emmatyping

   added

   extension-modulesC modules in the Modules dirC modules in the Modules dir

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   type-featureA feature request or enhancementA feature request or enhancement

   on Apr 26, 2025
2. emmatyping

   self-assigned this

   on Apr 26, 2025
3. picnixz commented on Apr 26, 2025

   

   picnixz

   on Apr 26, 2025

   Member

   More actions

   It's something I should have asked on discourse, but is compression the final name of the public module? I see in the PEP that we want to mimic hashlib.algorithms_guaranteed as well and one way to mitigate the name shadowing is to actually use compressionlib as the name (we have hashlib, annotationlib, and we could have compressionlib as well; I also think compressionlib is a less likely name to be chosen but it's not the best one as well). Feel free to ignore this question as the PEP has been accepted.
4. picnixz

   added this to Compression issues 🗜on Apr 26, 2025
5. emmatyping commented on Apr 26, 2025

   

   emmatyping

   on Apr 26, 2025

   MemberAuthor

   More actions

   It's something I should have asked on discourse, but is compression the final name of the public module?

   Yes, the decision was to use compression. A search of Github code showed that it was used very infrequently. Literally one in a million files https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377/66

   Reacted by Bénédikt Tran and Gregory P. Smith

   Reacted by stonebig
6. gpshead

   added a commit that references this issue on Apr 27, 2025

   gh-132983: Introduce `compression` package and move `_compression` mo…

   20be6ba
7. freakboy3742

   added a commit that references this issue on Apr 28, 2025

   gh-132983: Install compression package contents (#133063)

   632978f
8. vstinner

   added 2 commits that reference this issue on Apr 28, 2025

   pythongh-132983: Convert compression._common to a package

   0789ab8

   gh-132983: Convert compression._common to a package (#133086)

   fe8ad53
9. gpshead

   added a commit that references this issue on May 4, 2025

   gh-132983: Introduce `_zstd` bindings module (GH-133027)

   3b43335
10. AA-Turner commented on May 4, 2025

    

    AA-Turner

    on May 4, 2025

    Member

    More actions

    I've had a go at adding support for building on Windows in #133366, let me know thoughts.

    A
11. diegorusso

    added a commit that references this issue on May 4, 2025

    Merge remote-tracking branch 'origin/main' into call-len

    c03babc
12. ned-deily commented on May 5, 2025

    

    ned-deily

    on May 5, 2025

    Member

    More actions

    It looks like there are a couple of problems with the configure.ac changes to discover and check libzstd.
13. 172 remaining items

    Load more actions
14. hugovk

    added a commit that references this issue on Aug 7, 2025

    [3.14] gh-132983: Use ``Py_UNREACHABLE`` in ``_zstd_load_impl()`` (GH…

    0fc49be
15. Agent-Hellboy

    added 4 commits that reference this issue on Aug 19, 2025

    pythongh-132983: Minor fixes and clean up for the _zstd module (pytho…

    9acbfcc

    pythongh-132983: Add missing references to Zstandard in shutil docstr…

    e5ef0f4

    pythongh-132983: Fix docstrings in ``ZstdDict`` (python#137321)

    bff2573

    pythongh-132983: Use ``Py_UNREACHABLE`` in ``_zstd_load_impl()`` (pyt…

    4f35853
16. kumaraditya303

    added 3 commits that reference this issue on Sep 9, 2025

    [3.14] pythongh-132983: Add missing references to Zstandard in shutil…

    45a9174

    [3.14] pythongh-132983: Fix docstrings in ``ZstdDict`` (pythonGH-137321…

    f626b50

    [3.14] pythongh-132983: Use ``Py_UNREACHABLE`` in ``_zstd_load_impl()…

    301dcde
17. gostak-dd

    added 7 commits that reference this issue on Jun 2, 2026

    pythongh-132983: Add documentation for compression.zstd (pythonGH-133911

    5f28d73

    pythongh-132983: Refactor shared code in train_dict and finalize_dict (…

    8c702a3

    pythongh-132983: Fix refleak in zstd dictionary functions (pythongh-1…

    c231a62

    pythongh-132983: Slightly tweak error messages for _zstd compressor/d…

    cbaa5d9

    pythongh-132983: Make _zstd C code PEP 7 compliant (pythonGH-134605)

    9cb91b8

    pythongh-132983: Convert dict_content to take Py_buffer in ``ZstdDict…

    c94c79e

    pythongh-132983: Split ``_zstd_set_c_parameters`` (python#133921)

    82b28db

Issue actions