GeezerVison is showing github.com — Use zlib-ng (fast!) rather than mainline stale zlib in binary releases · Issue #91349 · python/cpython
Use zlib-ng (fast!) rather than mainline stale zlib in binary releases #91349Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2022-04-01.19:19:24.173>
labels = ['3.11', 'OS-windows', 'performance']
title = 'Use zlib-ng rather than zlib in binary releases'
updated_at = <Date 2022-04-02.09:48:33.061>
user = 'https://github.com/gpshead' bugs.python.org fields:
activity = <Date 2022-04-02.09:48:33.061>
actor = 'corona10'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Windows']
creation = <Date 2022-04-01.19:19:24.173>
creator = 'gregory.p.smith'
dependencies = []
files = []
hgrepos = []
issue_num = 47193
keywords = []
message_count = 1.0
messages = ['416508']
nosy_count = 7.0
nosy_names = ['gregory.p.smith', 'paul.moore', 'tim.golden', 'zach.ware', 'steve.dower', 'corona10', 'arhadthedev']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue47193'
versions = ['Python 3.11'] Linked PRs
Activitygpshead commented on Apr 1, 2022 zlib-ng is an optimized zlib library with better performance on most architectures (with contributions from the likes of Google, Cloudflare, and Intel). It is API compatible with zlib. https://github.com/zlib-ng/zlib-ng
I believe the only platform we don't use the OS's own zlib on is Windows so I'm tagging this issue Windows. gpshead commented on Jun 17, 2022 if this hasn't happened in the 3.11 betas this is presumably bumped to 3.12. zooba commented on Jul 17, 2022 I started taking a look at this and it seems like we can build it without having to worry about their build system by renaming the zconf.h.in and zconf-ng.h.in files to remove the .in, and setting ZLIB_COMPAT preprocessor in pythoncore.vcxproj (as well as referencing the files).
Running test_zlib the only failures seem to be testing for certain failures that no longer fail, but I've got no idea how important they are. Also no indication of the performance impact, or anything else that may change (e.g. new DLL exports, etc.), but it certainly does seem like a feasible drop-in replacement. rhpvorderman commented on Aug 1, 2022 @zooba which failures are these? I have accumulated quite some experience with the zlib/gzip formats due to working on python-isal, bindings for the ISA-L library that speeds up zlib-compatible compression by rewriting the algorithms in x86 Assembly language. It is quite good, but not suitable for a drop-in replacement, hence the python-isal project.
Having said that I'd love to help out with anything zlib related in CPython. zooba commented on Aug 1, 2022 I knew I should've kept better track of the changed error messages 😄
Skimming through the tests, I'm pretty sure test_wbits failed in a few places because we expected some "invalid window size" errors that aren't errors with zlib-ng. I don't think any were critical, but I don't know the usage of the library well enough to be sure (e.g. do people expect these errors and change behaviour? or are they always just developer error and cause changes to just make things work?) morotti commented on Apr 17, 2025 quick update,
-
I have attempted to adjust the linux build to detect and use zlib-ng, to no success.
the .configure script eludes me.
there might only be 10 people in the world who understand any of it.
any help would be greatly appreciated.
-
liblzma just had a new release a few days ago (xz-utils package on linux). https://github.com/tukaani-project/xz/releases
the latest liblzma release 5.8.x is using hardware CRC instructions, it is now as fast as zlib-ng and libdeflate.
I have submitted a PR to expose crc32 from lzma.
after it is merged, I shall do more PR to pick the optimal crc function between zlib/zlibng/liblzma depending on what is available.
nmoinvaz commented on Apr 17, 2025 It would be great to see what the stats are compared to zlib-ng. It has optimizations for different architectures and even instruction sets. So it will perform well on older hardware if that is necessary for your use case. added a commit that references this issue on Jul 20, 2025gh-91349: Adjust default compression level to 6 (down from 9) in gzip… morotti commented on May 21, 2026 Reacted by Hugo van Kemenade and Steve Dower rhpvorderman commented on Jul 21, 2026 Thanks @morotti for the write up. Very interesting. It also seems that my side project of python-zlib-ng is now redundant for windows 3.14 and higher. Less maintenance for me in the future. I hope that zlib-ng gets shipped by distro's soon. added 2 commits that reference this issue on Oct 21, 2026gh-91349: Update `gzip` docs and docstrings for the default compressl… [3.15] gh-91349: Update `gzip` docs and docstrings for the default co…
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|
