Bug description

encodings.punycode registers an IncrementalEncoder and IncrementalDecoder, but they just run the whole-string transform on every chunk they are given, so anything fed in more than one piece comes out wrong:

>>> import codecs
>>> 'ab'.encode('punycode')
b'ab-'
>>> encoder = codecs.getincrementalencoder('punycode')()
>>> encoder.encode('a') + encoder.encode('b', final=True)
b'a-b-'
>>> b'a-b-'.decode('punycode')
'a-b'

Punycode encodes the positions of the non-ASCII characters relative to the whole label, so a chunk cannot be encoded without the rest of the string; the same goes for decoding. The classes are two lines each, Lib/encodings/punycode.py lines 226-234, and ignore final altogether. This is half known: test_codecs.py lists punycode in broken_unicode_with_stateful ("encodings that don't work in stateful mode") and skips the chunked part of test_basics for it, but nothing says so in the docs, and the classes are still registered and handed out by codecs.getincrementalencoder(), codecs.iterencode() and StreamWriter.

I think the right fix is to buffer until final (the BufferedIncrementalEncoder and BufferedIncrementalDecoder base classes make that a few lines: _buffer_encode returns (b'', 0) until final), since that keeps codecs.iterencode, StreamWriter and friends working for anyone who hands them the whole label at once, which is presumably every current user. The idna codec already does something like it, buffering up to each ..

Related: #64331 (many incremental codecs do not handle fragmented data) mentions punycode in passing, in a list taken from that same test_codecs.py entry, but does not go into it.

Found by the Hypothesis tests in #160155, where it is pinned as PunycodeIncrementalBug.test_chunked_encode under expectedFailure; the chunked tests there leave punycode out otherwise.

CPython versions tested on

CPython main (6e1bd24)

Operating systems tested on

Linux

Linked PRs

• gh-160308: Buffer input in the punycode incremental codec until final #160599