Description

rhpvorderman

opened on Oct 17, 2022

Issue body actions

Documentation

The documentation mentions several PyPI packages such as numpy and requests as an alternative for standard library packages.

I would like to propose that python-isal gets a mention in the "see also" section of the zlib and gzip documentation. Simply: "python-isal, faster zlib and gzip decompression and compression".

Since the documentation cannot just recommend any random project out there here follows the argumentation why python-isal should get a mention.

1. Python-isal uses stdlib code and uses a PSF-2.0 license.
2. As a result the following improvements could be made to the stdlib code:

• bpo-43317: Use io.DEFAULT_BUFFER_SIZE instead of arbitrary number in gzip module #24645
• bpo-43316: gzip: CLI uses non-zero return code on error. #24647
• bpo-43612: Add wbits parameter to zlib.compress #25011
• bpo-43613: Faster implementation of gzip.compress and gzip.decompress #27941
• gh-95534: Improve gzip reading speed by 10% #97664
• Minor test fixes for zlib and gzip #22408
• bpo-45509: Check gzip headers for corrupted fields #29028

Python-isal is a "good citizen" of the python ecosystem. All the improvements have been ported back to CPython. The useful thing about python-isal is that it allows the gzip and zlib code of CPython to evolve and get tested by a smaller group of users before it lands in CPython itself. The PRs above were all suggested only after the changes were found to be stable in releases of python-isal.
Therefore more python-isal users is also beneficial to CPython itself. It is also beneficial for the users to be able to install a library that offers 2x faster decompression and 5x(!) faster compression. Hence a small one-liner in "see also" is warranted in my opinion.

The next thing for python-isal to tackle is this: #89550 . When a working solution is found this will be backported to CPython.

Disclosure: I am the python-isal maintainer.

Linked PRs

• gh-98347: Add links to python-isal in the documentation #98637
• [3.13] gh-98347: Add links to python-isal in the documentation (GH-98637) #132894

Activity

1. rhpvorderman

   added

   docsDocumentation in the Doc dirDocumentation in the Doc dir

   on Oct 17, 2022
2. encukou commented on Oct 20, 2022

   

   encukou

   on Oct 20, 2022

   Member

   More actions

   The documentation mentions several PyPI packages such as numpy and requests as an alternative for standard library packages.

   And we haven't exactly had the best experience with doing that. Requests is a good example: nowadays you could argue that httpx is a much better choice, but no one really wants to do the arguing.

   The useful thing about python-isal is that it allows the gzip and zlib code of CPython to evolve and get tested by a smaller group of users before it lands in CPython itself. The PRs above were all suggested only after the changes were found to be stable in releases of python-isal.

   That's a much better reason. Do you want to position the generic part of isal as an “upstream for gzip/zlip”, similarly to how importlib_resources powers importlib.resources or how tomli became tomllib?
   Is having gzip.open/GzipFile one day taking the compression library as an argument a possible goal?
   (Please don't take these as veiled promises, I'd just like to know your opinions.)

   The zlib experts are @Yhg1s & @gpshead. I think it's fine to add the mention if the module experts agree. But, both of them (and me) happen to be on the Steering Council, which should be able to figure out how to make policy decisions around mentioning other projects.
3. gpshead commented on Oct 20, 2022

   

   gpshead

   on Oct 20, 2022

   Member

   More actions

   Documentation is a living thing, we're free to change what it links to at any time (and backport that as far as desired into branch docs). So in that sense I think mentioning currently seen as good idea packages with a history of being useful that we'd like to see grow more community such as python-isal from our zlib and gzip module docs is reasonable. Even if we haven't worked out the desired long term role of the external thing we're linking to.

   Reacted by Petr Viktorin
4. gpshead

   self-assigned this

   on Oct 20, 2022
5. rhpvorderman commented on Oct 25, 2022

   

   rhpvorderman

   on Oct 25, 2022

   ContributorAuthor

   More actions

   That's a much better reason. Do you want to position the generic part of isal as an “upstream for gzip/zlip”, similarly to how importlib_resources powers importlib.resources or how tomli became tomllib?

   Since this is a C extension primarily, this is not possible. CPython uses argument clinic, but that does not work well on C-extensions because argument clinic is tuned for the specific version of CPython rather than for all supported versions of CPython. Furthermore I dislike the blocksoutputbuffer that is used in zlibmodule.c as it seems to me a much more convoluted solution than what was previously used, while it does not solve any actual problems that happen in the real world. (It is primarily useful for compressing/decompressing very large chunks in-memory, but the way computers work is that a streaming solution will always be faster in such cases, so why optimize for large in-memory blocks?). Also zlibmodule.c uses heap types. Which makes declaring types more difficult, so python-isal uses simple statically declared types. ISA-L and zlib also have some minor API differences. Mostly that ISA-L requires the developer to allocate memory.

   In conclusion, one-to-one copying is impossible. Having said that l did make accomodations so that code exchange is still very much possible. For example by separating the argument parsing code from the actual code, as is done in CPython's extension modules. The changes I have backported to CPython so far required minor rewrites, but were by no means difficult.

   Is having gzip.open/GzipFile one day taking the compression library as an argument a possible goal?

   I do not think this is necessary. Plain zlib is already quite fast. However, when you have data formats that use gzip/zlib compression and are measured in the order of gigabytes, zlib becomes a bottleneck. This is the case in bioinformatics, so that is why I created the python-isal bindings. In use cases where zlib is not the bottleneck, or the extra time spent is not really noticable, zlib is fine. Also, ISA-L does not have a zlib-compatible API. It is mostly similar, so it is quite easy for developers who know zlib, but it is going to be a pain to have the library as a generic argument. So I do not think it is worth it, there are implementation problems and the target audience is simply not big enough. Developers who know that compression/decompression is a bottleneck in their application can easily add python-isal as a dependency and that will solve the problem for most users as well.

   So in that sense I think mentioning currently seen as good idea packages with a history of being useful that we'd like to see grow more community such as python-isal from our zlib and gzip module docs is reasonable.

   Thank you for the compliment on the "good idea package". These ideas come from a lot of experimenting to make things faster which is more fun than video games sometimes.
6. serhiy-storchaka

   added this to Compression issues 🗜on Sep 10, 2023
7. AA-Turner

   added a commit that references this issue on Apr 25, 2025

   gh-98347: Add links to python-isal in the documentation (#98637)

   b1fc8b6
8. miss-islington

   added a commit that references this issue on Apr 25, 2025

   pythongh-98347: Add links to python-isal in the documentation (python…

   e38fc82
9. AA-Turner

   added a commit that references this issue on Apr 25, 2025

   [3.13] gh-98347: Add links to python-isal in the documentation (GH-98637

   293f289
10. ichard26 commented on Jul 8, 2025

    

    ichard26

    on Jul 8, 2025

    Contributor

    More actions

    This can be closed as #98637 has been merged.
11. gpshead

    closed this as completedon Jul 8, 2025
12. github-project-automation

    moved this to Done in Compression issues 🗜on Jul 8, 2025

Issue actions