GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-160302: Don't encode ESC, SO and SI in the ISO-2022 codecs by aozmekik · Pull Request #161381 · python/cpython

Conversation

@aozmekik

@aozmekik aozmekik commented Nov 5, 2026 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

The ISO-2022 encoders took the ASCII fast path for every code point below 0x80, so U+001B came out as a bare ESC byte, and in iso2022_kr U+000E and U+000F as bare SO and SI. Decoders read those bytes as the start of an escape sequence or as shifts. This makes them unencodable, so strict raises UnicodeEncodeError and the error handlers work as usual (replace gives ?, backslashreplace gives \x1b). SO and SI are left alone in the iso2022_jp* codecs, where they aren't shifts.

This broke one caller in the stdlib. For euc-jp and shift_jis, which email sends as iso-2022-jp, Message.set_payload() encoded the text and set_charset() then encoded the result again, which only worked because the encoder let ESC through. set_payload() now leaves that conversion to set_charset(), so the payload is encoded once and comes out the same.

Tested on a debug build: test_codecencodings_iso2022, test_multibytecodec, test_codecs, the other CJK codec tests, test_email, test_mailbox, test_smtplib, test_imaplib and test_mimetypes pass, -R 3:3 shows no leaks for the codec tests, and the new test cases fail on main.

gh-160155 pins this as xfail in test_encode_decode and keeps these characters out of the generated text, so whichever of the two lands second has to drop that.

  • Issue: ISO-2022 encoders accept U+001B (and iso2022_kr U+000E/U+000F) although the encoding cannot represent them #160302

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
aozmekik added 2 commits November 5, 2026 08:46
@aozmekik
pythongh-160302: Don't encode ESC, SO and SI in the ISO-2022 codecs
e7e54fb
The ISO-2022 encoders wrote U+001B as a bare ESC byte, and iso2022_kr
wrote U+000E and U+000F as bare SO and SI. Decoders take these bytes as
the start of an escape sequence or as shifts, so the output was misread.
Treat these characters as unencodable instead.
@aozmekik
pythongh-160302: Don't encode the payload twice in Message.set_payload()
2b6f035
When the input and output charsets differ, as for euc-jp and shift_jis,
which are sent as iso-2022-jp, set_payload() encoded the text and
set_charset() then encoded the result again. That only worked because
the iso2022_jp encoder let ESC through. Leave the conversion to
set_charset().
@aozmekik
aozmekik requested review from a team, AA-Turner and corona10 as code owners November 5, 2026 06:19
@bedevere-app bedevere-app Bot added the awaiting review label Nov 5, 2026
@bedevere-app bedevere-app Bot mentioned this pull request Nov 5, 2026
ISO-2022 encoders accept U+001B (and iso2022_kr U+000E/U+000F) although the encoding cannot represent them #160302
Open
@read-the-docs-community

read-the-docs-community Bot commented Nov 5, 2026

Copy link
Copy Markdown

Documentation build overview

📚 cpython-previews | 🛠️ Build #35662925 | 📁 Comparing 2b6f035 against main (aaaf61c)

  🔍 Preview build  

2 files changed
± whatsnew/3.16.html
± whatsnew/changelog.html
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@corona10 corona10 Awaiting requested review from corona10 corona10 is a code owner
@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@aozmekik
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.