GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — CJK StreamWriter.reset() does not write the shift-back sequence · Issue #160185 · python/cpython

CJK StreamWriter.reset() does not write the shift-back sequence #160185

New issue
Copy link
New issue
Copy link
Closed
#160186
Closed
CJK StreamWriter.reset() does not write the shift-back sequence#160185
#160186
Copy link
Assignees
gpshead
Labels
topic-unicodetype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

Description

@gpshead
gpshead
opened on Oct 22, 2026
Issue body actions

Bug report

Bug description:

StreamWriter.reset() of the stateful CJK codecs (hz, iso2022_jp*, iso2022_kr) usually writes nothing. MultibyteStreamWriter.reset() returns early when self->pending is NULL, but pending only holds an unfinished combining character, not the shift state, so the codec's encreset (which writes ESC ( B, SI or ~}) is skipped and the writer stays in double-byte mode.

import codecs, io
out = io.BytesIO()
w = codecs.getwriter('iso2022_jp')(out)
w.write('こんにちは。')
w.reset()
print(out.getvalue())  # b'\x1b$B$3$s$K$A$O!#'

Expected b'\x1b$B$3$s$K$A$O!#\x1b(B', as str.encode() gives. Anything written after reset() is read in JIS X 0208 mode, and the next CJK text written by the same writer has no opening escape. hz gives b'~{Dc:C' for '你好' instead of b'~{Dc:C~}'.

This is a regression from the gh-67436 crash fix (2015, 3.4.4/3.5), which added the early return to avoid passing a NULL pending to the encoder; before that, reset() always ran the codec reset. The same function also leaks the result of stream.write().

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

  • gh-160185: Fix CJK StreamWriter.reset() not returning to ASCII #160186
  • [3.15] gh-160185: Fix CJK StreamWriter.reset() not returning to ASCII (GH-160186) #161067
  • [3.14] gh-160185: Fix CJK StreamWriter.reset() not returning to ASCII (GH-160186) #161068

Activity

  1. gpshead
    self-assigned this
    on Oct 22, 2026
  2. gpshead
    added
    type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
    topic-unicode
    on Oct 22, 2026
  3. gpshead
    added a commit that references this issue on Oct 22, 2026

    pythongh-160185: Fix CJK StreamWriter.reset() not returning to ASCII

    d86d637
  4. vstinner commented on Oct 22, 2026

    @vstinner
    vstinner
    on Oct 22, 2026
    Member
    More actions

    Implementing correctly stateful codecs is hard. The io module should handle this case correctly, whereas the codecs module has multiple issues related to stateful codecs. In 2011, I proposed https://peps.python.org/pep-0400/ to deprecate codecs.StreamReader and codecs.StreamWriter. But this PEP was not accepted (but deferred). I still think that we should deprecate these APIs. If they are used, they should be replaced by io.TextIOWrapper instead.

    Reacted by Gregory P. Smith
  5. gpshead
    added a commit that references this issue on Nov 1, 2026

    gh-160185: Fix CJK StreamWriter.reset() not returning to ASCII (GH-16…

    aa4f541
  6. gpshead
    closed this as completedin #160186on Nov 1, 2026
  7. gpshead
    added 2 commits that reference this issue on Nov 1, 2026

    [3.15] gh-160185: Fix CJK StreamWriter.reset() not returning to ASCII (…

    6d18fc9

    [3.14] gh-160185: Fix CJK StreamWriter.reset() not returning to ASCII (…

    f11ed23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

  • @gpshead
    gpshead

Labels

topic-unicodetype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

    You are visitor 01402503 — GeezerVison does not store the pages you look up.
    Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.