GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | |
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | |
GeezerVison is showing github.com — gh-155460: Pre-allocate the output buffer in compression.zstd decompression by ThomasWaldmann · Pull Request #155464 · python/cpython Conversation
|
|
Tested on Windows x64 (MSVC 14.41, Release): cherry-picks cleanly onto current main |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
…ecompression Frames produced by the one-shot compression APIs record the decompressed size in the frame header. When the input starts at such a frame header, read that size with ZSTD_getFrameContentSize() and allocate the output buffer at its exact size, instead of growing it progressively and shrinking it on finish. An exactly filled buffer is then returned without a copy. This is only a sizing hint: decompression does not rely on it, so a hand-crafted header recording a wrong size produces exactly the same results and exceptions as before. Two guards bound the allocation: the recorded size is trusted only up to 1 GiB, and recorded sizes claiming more than a 32768x expansion of the available input -- more than the format can produce -- are ignored. Co-Authored-By: Claude Opus 5 <[email protected]>
On top of the pre-allocation in the previous commit: - Pre-size only when the first input of a frame holds the whole zstd1 frame, and check the recorded size against that frame's compressed size instead of all of the input, so that neither a frame fed in pieces nor the data following a frame leads to a pre-allocation. The 1 GiB limit stays. - Check the frame's window size against window_log_max before pre-sizing: libzstd's single-pass mode, which a pre-sized output enables, doesn't check it. - zstd.decompress() feeds each frame whole, sized with get_frame_size(), so every frame of multi-frame input is pre-sized, not only the first. Co-authored-by: Maik Schulze <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]>
… feeding test_output_allocated_once_each_frame passed even without the whole-frame feeding in zstd.decompress(): its 2 MiB frame compresses to 459 bytes, so it fits whole into the 1 KiB that _decompress_all() feeds a later frame first. Use 2 MiB of zeros followed by 4 KiB of random bytes instead: the frame compresses to more than 1 KiB, and most of its output is decoded from its first KiB, which grows the output buffer unless the frame is fed whole. Co-Authored-By: Claude Opus 5.5 <[email protected]>
e051e36 to
d81eed6
Compare
There was an error while loading. Please reload this page.
Successfully merging this pull request may close these issues.
There was an error while loading. Please reload this page.