GeezerVison

…pythonGH-156003)

Patch by @tonghuaroot.

zipfile.ZipExtFile._read1() bounds the output of each decompress() call
for DEFLATE members by passing a max_length to zlib, but for bzip2, LZMA,
and Zstandard members it called decompress() with no bound. A whole
compressed chunk was therefore expanded into a single allocation before
the data[:self._left] clip ran, so a consumer that deliberately reads in
small chunks to limit memory (for example zf.open(name).read(8192)) was
silently unprotected for non-DEFLATE members. A small, spec-conformant
archive member declaring a large uncompressed size could drive multi-GB
peak memory.

_read1() now passes a per-call bound to the non-DEFLATE decompress()
(mirroring the DEFLATE branch) and drains the decompressor's internal
buffer across calls by checking needs_input before reading more
compressed input. zipfile's LZMADecompressor wrapper forwards max_length
and exposes needs_input so the bound also holds for LZMA members.

Co-authored-by: tonghuaroot <[email protected]>

GH-156003) (#156362)

Patch by @tonghuaroot.

zipfile.ZipExtFile._read1() bounds the output of each decompress() call
for DEFLATE members by passing a max_length to zlib, but for bzip2, LZMA,
and Zstandard members it called decompress() with no bound. A whole
compressed chunk was therefore expanded into a single allocation before
the data[:self._left] clip ran, so a consumer that deliberately reads in
small chunks to limit memory (for example zf.open(name).read(8192)) was
silently unprotected for non-DEFLATE members. A small, spec-conformant
archive member declaring a large uncompressed size could drive multi-GB
peak memory.

_read1() now passes a per-call bound to the non-DEFLATE decompress()
(mirroring the DEFLATE branch) and drains the decompressor's internal
buffer across calls by checking needs_input before reading more
compressed input. zipfile's LZMADecompressor wrapper forwards max_length
and exposes needs_input so the bound also holds for LZMA members.


(cherry picked from commit f897dbf)

Co-authored-by: tonghuaroot <[email protected]>

GH-156003) (GH-156362) (GH-156737)

[3.15] gh-156002: Bound zipfile decompression for bzip2/LZMA/Zstandard (GH-156003) (GH-156362)

Patch by @tonghuaroot.

zipfile.ZipExtFile._read1() bounds the output of each decompress() call
for DEFLATE members by passing a max_length to zlib, but for bzip2, LZMA,
and Zstandard members it called decompress() with no bound. A whole
compressed chunk was therefore expanded into a single allocation before
the data[:self._left] clip ran, so a consumer that deliberately reads in
small chunks to limit memory (for example zf.open(name).read(8192)) was
silently unprotected for non-DEFLATE members. A small, spec-conformant
archive member declaring a large uncompressed size could drive multi-GB
peak memory.

_read1() now passes a per-call bound to the non-DEFLATE decompress()
(mirroring the DEFLATE branch) and drains the decompressor's internal
buffer across calls by checking needs_input before reading more
compressed input. zipfile's LZMADecompressor wrapper forwards max_length
and exposes needs_input so the bound also holds for LZMA members.

(cherry picked from commit f897dbf)
(cherry picked from commit 1b424c0)

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: tonghuaroot <[email protected]>

…H-157180)

GH-156003 made ZipExtFile._read1() call decompress(data, max_length) on
non-deflate decompressors and consult needs_input before reading more. A
decompressor installed by monkey-patching _get_decompressor() (as projects
like zipfile-zstd, zipfile-deflate64, ... do) may support neither, and every read
through it then failed with AttributeError.

- Make LZMADecompressor.needs_input public to simplify implementation.
- Make the needs_input attribute optional.
- If `decompress()` fails with TypeError, try again with one argument.
- Since the fallback to one-argument call is a maintenance burden, raise
  DeprecationWarning.
- Add tests for future changes, so we can make informed decisions
  about breaking monkey-patchers.

Co-authored-by: Petr Viktorin <[email protected]>

…ssors (pythonGH-157180)

pythonGH-156003 made ZipExtFile._read1() call decompress(data, max_length) on
non-deflate decompressors and consult needs_input before reading more. A
decompressor installed by monkey-patching _get_decompressor() (as projects
like zipfile-zstd, zipfile-deflate64, ... do) may support neither, and every read
through it then failed with AttributeError.

- Make LZMADecompressor.needs_input public to simplify implementation.
- Make the needs_input attribute optional.
- If `decompress()` fails with TypeError, try again with one argument.
- Since the fallback to one-argument call is a maintenance burden, raise
  DeprecationWarning.
- Add tests for future changes, so we can make informed decisions
  about breaking monkey-patchers.

Co-authored-by: Petr Viktorin <[email protected]>

…ssors (pythonGH-157180)

pythonGH-156003 made ZipExtFile._read1() call decompress(data, max_length) on
non-deflate decompressors and consult needs_input before reading more. A
decompressor installed by monkey-patching _get_decompressor() (as projects
like zipfile-zstd, zipfile-deflate64, ... do) may support neither, and every read
through it then failed with AttributeError.

- Make LZMADecompressor.needs_input public to simplify implementation.
- Make the needs_input attribute optional.
- If `decompress()` fails with TypeError, try again with one argument.
- Since the fallback to one-argument call is a maintenance burden, raise
  DeprecationWarning.
- Add tests for future changes, so we can make informed decisions
  about breaking monkey-patchers.

Co-authored-by: Petr Viktorin <[email protected]>

…) (GH-156362) (#156738)

* [3.15] gh-156002: Bound zipfile decompression for bzip2/LZMA/Zstandard (GH-156003) (GH-156362)

Patch by @tonghuaroot.

zipfile.ZipExtFile._read1() bounds the output of each decompress() call
for DEFLATE members by passing a max_length to zlib, but for bzip2, LZMA,
and Zstandard members it called decompress() with no bound. A whole
compressed chunk was therefore expanded into a single allocation before
the data[:self._left] clip ran, so a consumer that deliberately reads in
small chunks to limit memory (for example zf.open(name).read(8192)) was
silently unprotected for non-DEFLATE members. A small, spec-conformant
archive member declaring a large uncompressed size could drive multi-GB
peak memory.

_read1() now passes a per-call bound to the non-DEFLATE decompress()
(mirroring the DEFLATE branch) and drains the decompressor's internal
buffer across calls by checking needs_input before reading more
compressed input. zipfile's LZMADecompressor wrapper forwards max_length
and exposes needs_input so the bound also holds for LZMA members.

(cherry picked from commit f897dbf)
(cherry picked from commit 1b424c0)

* gh-156002: Keep reading through monkey-patched zipfile decompressors (GH-157180) (GH-157557)
(cherry picked from commit f507e69)


Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: tonghuaroot <[email protected]>
Co-authored-by: rasmusfaber <[email protected]>

GH-156003) (GH-156362) (#156739)

* gh-156002: Bound zipfile decompression for bzip2/LZMA/Zstandard (GH-156003) (GH-156362)

Patch by @tonghuaroot.

zipfile.ZipExtFile._read1() bounds the output of each decompress() call
for DEFLATE members by passing a max_length to zlib, but for bzip2, LZMA,
and Zstandard members it called decompress() with no bound. A whole
compressed chunk was therefore expanded into a single allocation before
the data[:self._left] clip ran, so a consumer that deliberately reads in
small chunks to limit memory (for example zf.open(name).read(8192)) was
silently unprotected for non-DEFLATE members. A small, spec-conformant
archive member declaring a large uncompressed size could drive multi-GB
peak memory.

_read1() now passes a per-call bound to the non-DEFLATE decompress()
(mirroring the DEFLATE branch) and drains the decompressor's internal
buffer across calls by checking needs_input before reading more
compressed input. zipfile's LZMADecompressor wrapper forwards max_length
and exposes needs_input so the bound also holds for LZMA members.

(cherry picked from commit f897dbf)
(cherry picked from commit 1b424c0)

* gh-156002: Keep reading through monkey-patched zipfile decompressors (GH-157180) (GH-157557)

(cherry picked from commit f507e69)

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: tonghuaroot <[email protected]>
Co-authored-by: rasmusfaber <[email protected]>

GH-156003) (GH-156362) (#156741)

* [3.15] gh-156002: Bound zipfile decompression for bzip2/LZMA/Zstandard (GH-156003) (GH-156362)

Patch by @tonghuaroot.

zipfile.ZipExtFile._read1() bounds the output of each decompress() call
for DEFLATE members by passing a max_length to zlib, but for bzip2, LZMA,
and Zstandard members it called decompress() with no bound. A whole
compressed chunk was therefore expanded into a single allocation before
the data[:self._left] clip ran, so a consumer that deliberately reads in
small chunks to limit memory (for example zf.open(name).read(8192)) was
silently unprotected for non-DEFLATE members. A small, spec-conformant
archive member declaring a large uncompressed size could drive multi-GB
peak memory.

_read1() now passes a per-call bound to the non-DEFLATE decompress()
(mirroring the DEFLATE branch) and drains the decompressor's internal
buffer across calls by checking needs_input before reading more
compressed input. zipfile's LZMADecompressor wrapper forwards max_length
and exposes needs_input so the bound also holds for LZMA members.

(cherry picked from commit f897dbf)
(cherry picked from commit 1b424c0)

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: tonghuaroot <[email protected]>

* Remove zstd test (3.14+)

* gh-156002: Keep reading through monkey-patched zipfile decompressors (GH-157180) (GH-157557)

(cherry picked from commit f507e69)

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: rasmusfaber <[email protected]>

* Avoid unittest's enterContext; it doesn't exist yet

* Don't use the :cve: RST role, it doesn't exist yet

---------

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: tonghuaroot <[email protected]>
Co-authored-by: rasmusfaber <[email protected]>

GH-156003) (GH-156362) (#156740)

* [3.15] gh-156002: Bound zipfile decompression for bzip2/LZMA/Zstandard (GH-156003) (GH-156362)

Patch by @tonghuaroot.

zipfile.ZipExtFile._read1() bounds the output of each decompress() call
for DEFLATE members by passing a max_length to zlib, but for bzip2, LZMA,
and Zstandard members it called decompress() with no bound. A whole
compressed chunk was therefore expanded into a single allocation before
the data[:self._left] clip ran, so a consumer that deliberately reads in
small chunks to limit memory (for example zf.open(name).read(8192)) was
silently unprotected for non-DEFLATE members. A small, spec-conformant
archive member declaring a large uncompressed size could drive multi-GB
peak memory.

_read1() now passes a per-call bound to the non-DEFLATE decompress()
(mirroring the DEFLATE branch) and drains the decompressor's internal
buffer across calls by checking needs_input before reading more
compressed input. zipfile's LZMADecompressor wrapper forwards max_length
and exposes needs_input so the bound also holds for LZMA members.

(cherry picked from commit f897dbf)
(cherry picked from commit 1b424c0)

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: tonghuaroot <[email protected]>

* Remove zstd test (3.14+)

* gh-156002: Keep reading through monkey-patched zipfile decompressors (GH-157180) (GH-157557)

(cherry picked from commit f507e69)

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: rasmusfaber <[email protected]>

* Don't use the :cve: RST role, it doesn't exist yet

---------

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: tonghuaroot <[email protected]>
Co-authored-by: rasmusfaber <[email protected]>

…tandard (pythonGH-156003) (pythonGH-156362) (python#156741)

* [3.15] pythongh-156002: Bound zipfile decompression for bzip2/LZMA/Zstandard (pythonGH-156003) (pythonGH-156362)

Patch by @tonghuaroot.

zipfile.ZipExtFile._read1() bounds the output of each decompress() call
for DEFLATE members by passing a max_length to zlib, but for bzip2, LZMA,
and Zstandard members it called decompress() with no bound. A whole
compressed chunk was therefore expanded into a single allocation before
the data[:self._left] clip ran, so a consumer that deliberately reads in
small chunks to limit memory (for example zf.open(name).read(8192)) was
silently unprotected for non-DEFLATE members. A small, spec-conformant
archive member declaring a large uncompressed size could drive multi-GB
peak memory.

_read1() now passes a per-call bound to the non-DEFLATE decompress()
(mirroring the DEFLATE branch) and drains the decompressor's internal
buffer across calls by checking needs_input before reading more
compressed input. zipfile's LZMADecompressor wrapper forwards max_length
and exposes needs_input so the bound also holds for LZMA members.

(cherry picked from commit f897dbf)
(cherry picked from commit 1b424c0)

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: tonghuaroot <[email protected]>

* Remove zstd test (3.14+)

* pythongh-156002: Keep reading through monkey-patched zipfile decompressors (pythonGH-157180) (pythonGH-157557)

(cherry picked from commit f507e69)

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: rasmusfaber <[email protected]>

* Avoid unittest's enterContext; it doesn't exist yet

* Don't use the :cve: RST role, it doesn't exist yet

---------

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: tonghuaroot <[email protected]>
Co-authored-by: rasmusfaber <[email protected]>
(cherry picked from commit c2bfbcd)