GeezerVison is showing github.com — CJK StreamWriter.reset() does not write the shift-back sequence · Issue #160185 · python/cpython
CJK StreamWriter.reset() does not write the shift-back sequence #160185DescriptionBug report
Bug description:
StreamWriter.reset() of the stateful CJK codecs (hz, iso2022_jp*, iso2022_kr) usually writes nothing. MultibyteStreamWriter.reset() returns early when self->pending is NULL, but pending only holds an unfinished combining character, not the shift state, so the codec's encreset (which writes ESC ( B, SI or ~}) is skipped and the writer stays in double-byte mode.
import codecs, io
out = io.BytesIO()
w = codecs.getwriter('iso2022_jp')(out)
w.write('こんにちは。')
w.reset()
print(out.getvalue()) # b'\x1b$B$3$s$K$A$O!#'
Expected b'\x1b$B$3$s$K$A$O!#\x1b(B', as str.encode() gives. Anything written after reset() is read in JIS X 0208 mode, and the next CJK text written by the same writer has no opening escape. hz gives b'~{Dc:C' for '你好' instead of b'~{Dc:C~}'.
This is a regression from the gh-67436 crash fix (2015, 3.4.4/3.5), which added the early return to avoid passing a NULL pending to the encoder; before that, reset() always ran the codec reset. The same function also leaks the result of stream.write().
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Linked PRs
Activityvstinner commented on Oct 22, 2026 Implementing correctly stateful codecs is hard. The io module should handle this case correctly, whereas the codecs module has multiple issues related to stateful codecs. In 2011, I proposed https://peps.python.org/pep-0400/ to deprecate codecs.StreamReader and codecs.StreamWriter. But this PEP was not accepted (but deferred). I still think that we should deprecate these APIs. If they are used, they should be replaced by io.TextIOWrapper instead. Reacted by Gregory P. Smith
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|