GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-139877: Use PyBytesWriter in pycore_blocks_output_buffer.h by emmatyping · Pull Request #139976 · python/cpython

Conversation

@emmatyping

@emmatyping emmatyping commented Oct 12, 2025 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Currently, the _BlocksOutputBuffer code creates a list of bytes objects to handle the output data from compression libraries. This ends up being slow due to the output buffer code needing to copy each bytes element of the list into the final bytes object buffer at the end of compression.

The new PyBytesWriter API introduced in PEP 782 is an ergonomic and fast method of writing data into a buffer that will later turn into a bytes object. Benchmarks show that using the PyBytesWriter API is 10-30% faster for decompression across a variety of settings. The performance gains are greatest when the decompressor is very performant, such as for Zstandard. Otherwise the decompressor can bottleneck decompression and the gains are more modest, but still sizable (e.g. zlib)!

  • Issue: Explore using PyBytesWriter API for compression libraries output buffers #139877

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
vstinner, chris-eibl, and cmaloney reacted with hooray emoji
All reactions
  • 🎉 3 reactions
@emmatyping
Use PyBytesWriter in pycore_blocks_output_buffer.h
086c79a 
Currently, the _BlocksOutputBuffer code creates a list of bytes
objects to handle the output data from compression libraries. This ends
up being slow due to the output buffer code needing to copy each bytes
element of the list into the final bytes object buffer at the end of
compression.

The new PyBytesWriter API introduced in PEP 782 is an ergonomic and fast
method of writing data into a buffer that will later turn into a bytes
object. Benchmarks show that using the PyBytesWriter API is 10-30%
faster for decompression across a variety of settings. The performance
gains are greatest when the decompressor is very performant, such as for
Zstandard. Otherwise the decompressor can bottleneck decompression and
the gains are more modest, but still sizable!
@emmatyping
emmatyping requested a review from AA-Turner as a code owner October 12, 2025 01:47
@emmatyping emmatyping added the skip news label Oct 12, 2025
@bedevere-app bedevere-app Bot added the awaiting core review label Oct 12, 2025
@bedevere-app bedevere-app Bot mentioned this pull request Oct 12, 2025
Explore using PyBytesWriter API for compression libraries output buffers #139877
Closed
emmatyping added 2 commits October 11, 2025 18:48
@emmatyping
Fix lint
29a8230
@emmatyping
Cast data buffer to char * for Windows
ffd48e4
@emmatyping emmatyping added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 12, 2025
@bedevere-bot

bedevere-bot commented Oct 12, 2025

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @emmatyping for commit ffd48e4 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F139976%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 12, 2025
picnixz
picnixz reviewed Oct 12, 2025
View reviewed changes
Comment thread Include/internal/pycore_blocks_output_buffer.h Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Include/internal/pycore_blocks_output_buffer.h Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Include/internal/pycore_blocks_output_buffer.h Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Include/internal/pycore_blocks_output_buffer.h Resolved

Uh oh!

There was an error while loading. Please reload this page.
@emmatyping
Respond to review from Bénédikt
2336d0e
vstinner
vstinner approved these changes Oct 13, 2025
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Nice optimization!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Oct 13, 2025
picnixz
picnixz reviewed Oct 13, 2025
View reviewed changes
Comment thread Include/internal/pycore_blocks_output_buffer.h Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Include/internal/pycore_blocks_output_buffer.h Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Include/internal/pycore_blocks_output_buffer.h Resolved

Uh oh!

There was an error while loading. Please reload this page.
emmatyping and others added 2 commits October 13, 2025 11:19
@emmatyping
Respond to more review from Bénédikt
dfd1c25
@emmatyping @picnixz
Update Include/internal/pycore_blocks_output_buffer.h
e5f3661 
Co-authored-by: Bénédikt Tran <[email protected]>
picnixz
picnixz reviewed Oct 13, 2025
View reviewed changes
Comment thread Include/internal/pycore_blocks_output_buffer.h Resolved

Uh oh!

There was an error while loading. Please reload this page.
@emmatyping
Refactor block size calculation
029c53c
picnixz
picnixz approved these changes Oct 14, 2025
View reviewed changes
@emmatyping
emmatyping merged commit f262297 into python:main Oct 14, 2025
4 checks passed

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot removed the awaiting merge label Oct 14, 2025
@emmatyping emmatyping mentioned this pull request Oct 16, 2025
BlocksOutputBuffer causes a performance regression in bz2, lzma and zlib modules #101260
Closed
@rhpvorderman

rhpvorderman commented Dec 1, 2025

Copy link
Copy Markdown
Contributor

Thanks @emmatyping! Looks like I can use this too for python-isal and python-zlib-ng. Awesome!
emmatyping reacted with rocket emoji
All reactions
  • 🚀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@rhpvorderman rhpvorderman mentioned this pull request Dec 1, 2025
Checkout the PyBytesWriter API instead of custom bytes resizing pycompression/python-isal#246
Open
@vstinner

vstinner commented Dec 2, 2025

Copy link
Copy Markdown
Member

Thanks @emmatyping! Looks like I can use this too for python-isal and python-zlib-ng. Awesome!

FYI pythoncapi-compat provides PyBytesWriter API on Python 3.14 and older.
rhpvorderman and emmatyping reacted with thumbs up emoji
All reactions
  • 👍 2 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@rhpvorderman

rhpvorderman commented Dec 3, 2025

Copy link
Copy Markdown
Contributor

@vstinner Amazing! Thanks. That will sure come in handy.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
StanFromIreland pushed a commit to StanFromIreland/cpython that referenced this pull request Dec 6, 2025
@emmatyping @picnixz @StanFromIreland
pythongh-139877: Use PyBytesWriter in pycore_blocks_output_buffer.h (p…
308551f 
…ython#139976)

Previously, the _BlocksOutputBuffer code creates a list of bytes objects to handle the output data from compression libraries. This ends up being slow due to the output buffer code needing to copy each bytes element of the list into the final bytes object buffer at the end of compression.

The new PyBytesWriter API introduced in PEP 782 is an ergonomic and fast method of writing data into a buffer that will later turn into a bytes object. Benchmarks show that using the PyBytesWriter API is 10-30% faster for decompression across a variety of settings. The performance gains are greatest when the decompressor is very performant, such as for Zstandard (and likely zlib-ng). Otherwise the decompressor can bottleneck decompression and the gains are more modest, but still sizable (e.g. 10% faster for zlib)!

Co-authored-by: Bénédikt Tran <[email protected]>
@vstinner vstinner mentioned this pull request Sep 12, 2026
gh-155742: Use PyBytesWriter in zlib #157348
Draft
@maikschulze maikschulze mentioned this pull request Nov 6, 2026
zlib, gzip, lzma and zstd in 3.15: decompressing large outputs is slower on macOS and Windows, and the process footprint grows on macOS (bisected to f262297) #161478
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner vstinner approved these changes
@picnixz picnixz picnixz approved these changes
@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@emmatyping @bedevere-bot @rhpvorderman @vstinner @picnixz
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.