GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-155460: Pre-allocate the output buffer in compression.zstd decompression by ThomasWaldmann · Pull Request #155464 · python/cpython

Conversation

@ThomasWaldmann

@ThomasWaldmann ThomasWaldmann commented Aug 10, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor
  • Issue: Pre-allocate the output buffer in compression.zstd decompression from the frame's recorded content size #155460

compression.zstd decompression grows its output buffer step by step, even though most frames record their decompressed size in the frame header (zstd.compress() writes it by default).

With this PR, when the first input of a frame holds the whole frame and the frame records its content size, the output buffer is allocated once, at that size. libzstd then decodes straight into it in a single pass, instead of through its internal window buffer.

The recorded size is untrusted, so it is used only if:

  • it is at most 1 GiB (OUTPUT_PREALLOC_MAX);
  • the frame is at least 1/32768 of it (OUTPUT_MAX_EXPANSION): a block decodes to at most 128 KiB and takes at least 4 bytes, so no valid frame expands more;
  • the frame's window size fits window_log_max, which libzstd's single-pass mode doesn't check.

Otherwise the buffer grows as before. libzstd fails a frame that decodes to another size than recorded, whether the output was pre-allocated or not, and max_length still limits the allocation.

zstd.decompress() feeds each frame of multi-frame input whole (sized with get_frame_size()), so each frame is pre-allocated, not only the first.

Throughput at level 3, measured with pyperf (table): on Linux/x86-64 about 2x for 256 KiB to 16 MiB frames and unchanged from 32 MiB up; on macOS/arm64 7-29% higher at all sizes. For 1000 frames of 512 KiB to 8 MiB it is 14% higher on Linux and 18% higher on macOS.

Co-authored-by: Maik Schulze [email protected]

🤖 Generated with Claude Code

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

maikschulze reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@ThomasWaldmann
ThomasWaldmann requested review from AA-Turner and emmatyping as code owners August 10, 2026 06:50
@bedevere-app bedevere-app Bot added the awaiting review label Aug 10, 2026
@bedevere-app bedevere-app Bot mentioned this pull request Aug 10, 2026
Pre-allocate the output buffer in compression.zstd decompression from the frame's recorded content size #155460
Open
@maikschulze

maikschulze commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Tested on Windows x64 (MSVC 14.41, Release): cherry-picks cleanly onto current main
(f88565b) and test_zstd passes. Multi-threaded numbers and a note on the cap in the
issue: #155460 (comment)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@maikschulze maikschulze mentioned this pull request Nov 6, 2026
zlib, gzip, lzma and zstd in 3.15: decompressing large outputs is slower on macOS and Windows, and the process footprint grows on macOS (bisected to f262297) #161478
Open
ThomasWaldmann and others added 3 commits November 6, 2026 18:32
@ThomasWaldmann @claude
pythongh-155460: Pre-allocate the output buffer in compression.zstd d…
0814951
…ecompression

Frames produced by the one-shot compression APIs record the decompressed
size in the frame header.  When the input starts at such a frame header,
read that size with ZSTD_getFrameContentSize() and allocate the output
buffer at its exact size, instead of growing it progressively and
shrinking it on finish.  An exactly filled buffer is then returned
without a copy.

This is only a sizing hint: decompression does not rely on it, so a
hand-crafted header recording a wrong size produces exactly the same
results and exceptions as before.  Two guards bound the allocation: the
recorded size is trusted only up to 1 GiB, and recorded sizes claiming
more than a 32768x expansion of the available input -- more than the
format can produce -- are ignored.

Co-Authored-By: Claude Opus 5 <[email protected]>
@ThomasWaldmann @maikschulze @claude
pythongh-155460: Pre-size each whole zstd frame, respect window_log_max
431f0f1
On top of the pre-allocation in the previous commit:

- Pre-size only when the first input of a frame holds the whole zstd1
  frame, and check the recorded size against that frame's compressed
  size instead of all of the input, so that neither a frame fed in
  pieces nor the data following a frame leads to a pre-allocation.
  The 1 GiB limit stays.
- Check the frame's window size against window_log_max before
  pre-sizing: libzstd's single-pass mode, which a pre-sized output
  enables, doesn't check it.
- zstd.decompress() feeds each frame whole, sized with get_frame_size(),
  so every frame of multi-frame input is pre-sized, not only the first.

Co-authored-by: Maik Schulze <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
@ThomasWaldmann @claude
pythongh-155460: Make the multi-frame zstd test depend on whole-frame…
d81eed6
… feeding

test_output_allocated_once_each_frame passed even without the
whole-frame feeding in zstd.decompress(): its 2 MiB frame compresses to
459 bytes, so it fits whole into the 1 KiB that _decompress_all() feeds
a later frame first.  Use 2 MiB of zeros followed by 4 KiB of random
bytes instead: the frame compresses to more than 1 KiB, and most of its
output is decoded from its first KiB, which grows the output buffer
unless the frame is fed whole.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@ThomasWaldmann
ThomasWaldmann force-pushed the zstd-decompress-prealloc branch from e051e36 to d81eed6 Compare November 6, 2026 23:27
@bedevere-app bedevere-app Bot added the type-feature A feature request or enhancement label Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner is a code owner
@emmatyping emmatyping Awaiting requested review from emmatyping emmatyping is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting review type-feature A feature request or enhancement

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@ThomasWaldmann @maikschulze
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.