GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — GH-133711: Enable UTF-8 mode by default (PEP 686) by AA-Turner · Pull Request #133712 · python/cpython

Conversation

@AA-Turner

@AA-Turner AA-Turner commented May 8, 2025 •
edited by github-actions Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member
  • Issue: Enable UTF-8 mode by default (implement PEP 686) #133711

📚 Documentation preview 📚: https://cpython-previews--133712.org.readthedocs.build/

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@AA-Turner
Enable UTF-8 mode by default (PEP 686)
fc19177
@AA-Turner
AA-Turner requested a review from methane May 8, 2025 21:21
@AA-Turner
AA-Turner requested review from FFY00, ericsnowcurrently and gpshead as code owners May 8, 2025 21:21
@bedevere-app bedevere-app Bot mentioned this pull request May 8, 2025
Enable UTF-8 mode by default (implement PEP 686) #133711
Closed
@bedevere-app bedevere-app Bot added the awaiting core review label May 8, 2025
@AA-Turner
AA-Turner requested a review from vstinner May 8, 2025 21:21
@StanFromIreland

This comment was marked as outdated.

Sign in to view
@AA-Turner

This comment was marked as resolved.

Sign in to view
@methane

This comment was marked as resolved.

Sign in to view
@methane

This comment was marked as resolved.

Sign in to view
vstinner
vstinner reviewed May 9, 2025
View reviewed changes
Comment thread Doc/library/os.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Doc/library/os.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Doc/library/sys.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.
zooba
zooba reviewed May 9, 2025
View reviewed changes
Comment thread Doc/whatsnew/3.15.rst Outdated
* Python UTF-8 mode is now enabled by default.
It may be disabled with by setting :envvar:`PYTHONUTF8=0 <PYTHONUTF8>` as
an environment variable or by using the :option:`-X utf8=0 <-X>` flag.
See :pep:`686` for further details.

@zooba zooba May 9, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like we can probably put some more explanation in here, such as that it affects TextIOWrapper and hence open(). The current description doesn't sound as scary as it needs to, in my opinion.

Along the lines of: "Python UTF-8 mode is now enabled by default. This means that (files/console/etc.) will now use UTF-8 regardless of system settings, unless specifically overridden in code (typically with an encoding= argument)."

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
vstinner reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

@zooba zooba May 9, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be clear, it's nothing new. But we shouldn't assume that everyone already knows what UTF-8 mode implies. There are many more people out there who haven't ever thought about it than those who are waiting for it to be the default.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@vstinner vstinner May 9, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another effect of the UTF-8 Mode is that Python ignores the locale encoding.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@AA-Turner AA-Turner May 10, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've expanded the What's New entry, please see https://cpython-previews--133712.org.readthedocs.build/en/133712/whatsnew/3.15.html#other-language-changes

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
AA-Turner added 3 commits May 10, 2025 01:47
@AA-Turner
Victor's documentation comments
c706cfa
@AA-Turner
Fix test_cmd_line
f095ab0
@AA-Turner
Improve What's New entry
ff066b1
@AA-Turner
AA-Turner force-pushed the utf-8-mode branch from 036555d to ff066b1 Compare May 10, 2025 02:09
AA-Turner and others added 2 commits May 10, 2025 03:20
@AA-Turner
Add references to UTF-8 and Unicode
abfb864
@AA-Turner
Merge branch 'main' into utf-8-mode
f40b986
@FFY00
FFY00 removed their request for review May 10, 2025 02:28
vstinner
vstinner reviewed May 10, 2025
View reviewed changes
Comment thread Doc/using/windows.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.
vstinner
vstinner reviewed May 10, 2025
View reviewed changes
Comment thread Doc/whatsnew/3.15.rst
======================


* Python now uses UTF-8_ as the default encoding, independent of the system's

@vstinner vstinner May 10, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might mention the UTF-8 Mode earlier since it has other side effects documented in the UTF-8 Mode section, such as changing sys.stdout error handler and ignoring the locale encoding.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@methane methane Jun 16, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"default encoding" is UTF-8 since Python 3.0. (e.g. sys.getdefaultencoding())

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
methane
methane reviewed May 10, 2025
View reviewed changes
Comment thread Lib/test/test_cmd_line.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
AA-Turner added 2 commits May 10, 2025 23:50
@AA-Turner
Check stream buffer type in test_python_legacy_windows_stdio
3226e1d
@AA-Turner
Merge branch 'main' into utf-8-mode
b83c966 
# Conflicts:
#	Doc/whatsnew/3.15.rst
@AA-Turner

This comment was marked as resolved.

Sign in to view
@bedevere-bot

This comment was marked as resolved.

Sign in to view
methane
methane reviewed May 10, 2025
View reviewed changes
Comment thread Lib/test/test_cmd_line.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
methane
methane reviewed May 11, 2025
View reviewed changes
Comment thread Lib/test/test_cmd_line.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@vstinner

vstinner commented May 28, 2025

Copy link
Copy Markdown
Member

test_regrtest failed:

I wrote #134839 to fix test_regrtest when PYTHONUTF8=1 env var is set.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@vstinner

vstinner commented May 28, 2025

Copy link
Copy Markdown
Member

I can also reproduce the [test_readline] issue in the main branch using the fr_FR locale and the command:
PYTHONUTF8=1 LANG=fr_FR ./python -m test -v test_readline -u all

I wrote #134841 to fix test_readline for the Python UTF-8 Mode.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@AA-Turner
Merge branch 'main' into utf-8-mode
9b3ebf7
@AA-Turner AA-Turner added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 28, 2025
@bedevere-bot

This comment was marked as duplicate.

Sign in to view
@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 28, 2025
@vstinner

vstinner commented May 28, 2025

Copy link
Copy Markdown
Member

buildbot/wasm32-wasi Non-Debug PR — Build done.

Configure host Python fails with:

checking for /dev/ptmx... not set
configure: error: set ac_cv_file__dev_ptmx to yes/no in your CONFIG_SITE file when cross compiling

It seems to be unrelated to this PR.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@AA-Turner

AA-Turner commented May 29, 2025

Copy link
Copy Markdown
Member Author

buildbot/wasm32-wasi Non-Debug PR — Build done.

Configure host Python fails with:

checking for /dev/ptmx... not set
configure: error: set ac_cv_file__dev_ptmx to yes/no in your CONFIG_SITE file when cross compiling

It seems to be unrelated to this PR.

cc @brettcannon, this builder seems to be consistently failing since ~15/05.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
vstinner
vstinner approved these changes May 30, 2025
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Doc/library/os.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Include/cpython/initconfig.h Resolved

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels May 30, 2025
@vstinner

vstinner commented Jun 16, 2025

Copy link
Copy Markdown
Member

Ping @AA-Turner, I made two suggestions.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
vstinner added 3 commits July 13, 2025 19:01
@vstinner
Update Doc/library/os.rst
6ecea69
@vstinner
Update Include/cpython/initconfig.h
e87ce6c
@vstinner
Merge branch 'main' into utf-8-mode
3130318
@vstinner
vstinner enabled auto-merge (squash) July 13, 2025 17:04
@vstinner
vstinner disabled auto-merge July 13, 2025 17:04
@vstinner

vstinner commented Jul 13, 2025

Copy link
Copy Markdown
Member

vstinner disabled auto-merge

In fact, it would be safe to run this PR on buildbots before merging it. I will wait until GitHub CIs complete first.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@vstinner vstinner added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jul 13, 2025
@bedevere-bot

bedevere-bot commented Jul 13, 2025

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @vstinner for commit 3130318 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F133712%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jul 13, 2025
@AA-Turner

AA-Turner commented Jul 15, 2025

Copy link
Copy Markdown
Member Author

Thanks @vstinner!

A
vstinner reacted with hooray emoji
All reactions
  • 🎉 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@AA-Turner
AA-Turner merged commit c89a66f into python:main Jul 15, 2025
@bedevere-app bedevere-app Bot removed the awaiting merge label Jul 15, 2025
@AA-Turner
AA-Turner deleted the utf-8-mode branch July 15, 2025 09:45
taegyunkim pushed a commit to taegyunkim/cpython that referenced this pull request Aug 4, 2025
@AA-Turner @vstinner @taegyunkim
pythonGH-133711: Enable UTF-8 mode by default (PEP 686) (python#133712)
4c6acfb 
Co-authored-by: Victor Stinner <[email protected]>
Agent-Hellboy pushed a commit to Agent-Hellboy/cpython that referenced this pull request Aug 19, 2025
@AA-Turner @vstinner @Agent-Hellboy
pythonGH-133711: Enable UTF-8 mode by default (PEP 686) (python#133712)
e7da33e 
Co-authored-by: Victor Stinner <[email protected]>
@coderabbitai coderabbitai Bot mentioned this pull request May 6, 2026
Implement agent tool management CLI leynos/repo-local-tools#6
Merged
@coderabbitai coderabbitai Bot mentioned this pull request Aug 10, 2026
📖 Linguist: optimize agentic communication compiler timerloggedout-spec/termux-monorepo#126
Closed
@kuotunyu kuotunyu mentioned this pull request Sep 23, 2026
subprocess on Windows returns None instead of raising UnicodeDecodeError #158006
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@methane methane methane left review comments
@zooba zooba zooba left review comments
@vstinner vstinner vstinner approved these changes
@gpshead gpshead Awaiting requested review from gpshead gpshead is a code owner
@ericsnowcurrently ericsnowcurrently Awaiting requested review from ericsnowcurrently ericsnowcurrently is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@AA-Turner @StanFromIreland @methane @bedevere-bot @vstinner @zooba
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.