GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-160308: Buffer input in the punycode incremental codec until final by OsmnvAslan · Pull Request #160599 · python/cpython

Conversation

@OsmnvAslan

@OsmnvAslan OsmnvAslan commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

The punycode IncrementalEncoder and IncrementalDecoder ran the whole-string transform on each chunk and ignored final, so chunked input produced wrong output (e.g. b'a-b-' instead of b'ab-'). Punycode encodes the positions of non-ASCII characters relative to the whole string, so a chunk cannot be transformed on its own.

They now derive from BufferedIncrementalEncoder / BufferedIncrementalDecoder and buffer the input until final is true, as suggested in the issue. codecs.iterencode()/iterdecode() now give correct results, and getstate()/setstate()/reset() work through the buffered base classes.

Behaviour note: a caller that uses the incremental classes directly with a whole label but never passes final=True now gets empty output, since nothing is emitted until the input is known to be complete. That is the documented contract of incremental codecs, but it is a visible change for such code.

punycode stays in broken_unicode_with_stateful in test_codecs, because the StreamReader/StreamWriter classes still use the stateless codec per chunk; this PR only fixes the incremental classes.

Closes gh-160308.

Prepared with AI assistance, then reviewed and tested before submission.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@OsmnvAslan
pythongh-160308: Buffer input in the punycode incremental codec until…
0c13737
… final

Punycode encodes the positions of non-ASCII characters relative to the whole string, so chunks cannot be transformed independently. Use BufferedIncrementalEncoder/Decoder and only transform the input once final is true.
@bedevere-app bedevere-app Bot mentioned this pull request Oct 24, 2026
The punycode incremental encoder and decoder transform each chunk separately #160308
Open
@OsmnvAslan OsmnvAslan closed this Oct 24, 2026
@OsmnvAslan
OsmnvAslan deleted the fix-punycode-incremental branch November 4, 2026 00:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

The punycode incremental encoder and decoder transform each chunk separately

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@OsmnvAslan
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.