GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-136702: Deprecate passing non-ascii *encoding* (str) to `encodings.normalize_encoding` by StanFromIreland · Pull Request #140030 · python/cpython

Conversation

@StanFromIreland

@StanFromIreland StanFromIreland commented Oct 13, 2025 •
edited by github-actions Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Passing a non-ascii bytes already raises a ValueError. The requirement of non-ascii input has been documented, but not enforced.

  • Issue: Improve encodings.normalize_encoding behaviour or docs #136702

📚 Documentation preview 📚: https://cpython-previews--140030.org.readthedocs.build/

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@StanFromIreland
deprecate non-ascii
c8fc658
@StanFromIreland
StanFromIreland requested a review from malemburg October 13, 2025 10:28
@bedevere-app bedevere-app Bot added the awaiting review label Oct 13, 2025
@bedevere-app bedevere-app Bot mentioned this pull request Oct 13, 2025
Improve encodings.normalize_encoding behaviour or docs #136702
Open
StanFromIreland added 2 commits October 13, 2025 11:34
@StanFromIreland
Relocate import
5b50daa
@StanFromIreland
sanitize charset names in email
95f2e65
@StanFromIreland
StanFromIreland requested a review from a team as a code owner October 13, 2025 11:13
malemburg
malemburg approved these changes Oct 16, 2025
View reviewed changes

@malemburg malemburg left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check the performance of the helper function.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/email/_header_value_parser.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/email/utils.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting review labels Oct 16, 2025
@StanFromIreland
Use table, replace with 'ascii'
fad52cd
@StanFromIreland

StanFromIreland commented Oct 16, 2025

Copy link
Copy Markdown
Member Author

Thanks for the review! Using translate gives a nice performance bonus, changed the fallback to 'ascii' too.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@StanFromIreland
Merge branch 'main' into encodings/non-ascii
3ac0804
bitdancer
bitdancer reviewed Oct 31, 2025
View reviewed changes
Comment thread Lib/email/utils.py Outdated
return charset
sanitized = charset.translate(_SANITIZE_TABLE)
return sanitized if sanitized else fallback_charset

@bitdancer bitdancer Oct 31, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the trigger for this change? Do I actually have a test that uses a non-ascii charset name? If I did it should be an error case, since non-ascii is not permitted in charset names per the RFCs. I'm surprised I don't appear to be registering a defect for that, though I didn't go through the code enough to be sure I don't ;)

Regardless it isn't clear to me that 'sanitizing' is a useful operation. It isn't likely to produce a valid charset name, we should just be falling back to ascii at that point. What led you to choose this approach?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@StanFromIreland StanFromIreland Oct 31, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is currently done by normalize_encoding.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@bitdancer bitdancer Nov 1, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. emal doesn't call lookup directly and no tests fail without the changes.

I presume you did this to preserve backward compatibility. Unless I'm missing something, I don't think we should bother to do that. Given a non-ascii charset name, there are two possible outcomes from the current code: the name after sanitizing is not a valid codec name, or it is. If it is valid after sanitizing, there are two cases: the sanitized name results in successful decoding, or it does not. It is only the first of these second two cases that would be affected by the post-deprecation change.

How often would that case occur in reality? I would guess it would be a vanishingly small number of cases, if it ever occurs at all.

I think it will be better to remove the changes to the email package from this PR. If anyone sees the deprecation warning maybe they'll open an issue, but I'm betting nobody ever sees it from the email package. The behavior after the deprecation is over is the behavior we want: if the codec name contains non-ascii it is not a valid codec name, so any non-ascii in the text being decoded using that charset name will ultimately get turned into the 'unknown character' glyph when decoded by the email package.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@StanFromIreland StanFromIreland Nov 1, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I presume you did this to preserve backward compatibility.

Yes, I'm no email expert and I did not dig into the specifications, so I did this to not change any behaviour. I can remove it.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@malemburg malemburg Nov 9, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What't the conclusion here ? I still see the email package changes in place, but they look pretty harmless to me.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@StanFromIreland
StanFromIreland requested a review from bitdancer November 1, 2025 19:39
@StanFromIreland
Review
9d6f06e
StanFromIreland
StanFromIreland commented Nov 1, 2025
View reviewed changes
Comment thread Lib/test/test_email/test_email.py Outdated
Comment on lines +5720 to +5723
import warnings
with warnings.catch_warnings():
warnings.simplefilter("ignore", DeprecationWarning)
self.assertEqual(msg.get_filename(), 'myfile.txt')

@StanFromIreland StanFromIreland Nov 1, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bitdancer It was indeed tested.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@bitdancer bitdancer Nov 1, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I missed that. I guess maybe warnings weren't enabled the way I ran the tests, though I thought I was doing so...

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

StanFromIreland reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

@StanFromIreland StanFromIreland Nov 1, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you want to do, keep it like so or revert to my backwards-compatible approach?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@bitdancer bitdancer Nov 1, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe I've confirmed that the tests will still pass after lookup starts raising a ValueError, which is what I was hoping would happen. I wonder if we should actually assert the deprecation warning here, though, so that we are reminded to remove the check when the deprecation turns into reality. I'm fine with whatever is standard practice for such cases, though.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@StanFromIreland StanFromIreland Nov 1, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Converted to asserts.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
StanFromIreland added 2 commits November 1, 2025 20:22
@StanFromIreland
Fix second warning
16697dc
@StanFromIreland
Convert to asserts
e4036f8
bitdancer
bitdancer approved these changes Nov 1, 2025
View reviewed changes

@bitdancer bitdancer left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
StanFromIreland added 3 commits November 1, 2025 22:04
@StanFromIreland
Fix for platforms with ordered tests
b8fc5f4
@StanFromIreland
!fixup
7592af8
@StanFromIreland
Fix CI on Android and iOS
8c59899
@StanFromIreland

StanFromIreland commented Nov 8, 2025

Copy link
Copy Markdown
Member Author

@malemburg Anything else I should do here?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@malemburg

malemburg commented Nov 9, 2025

Copy link
Copy Markdown
Member

@malemburg Anything else I should do here?

The PR looks fine to, but I want to hear back from @bitdancer about the email package changes before merging it.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@StanFromIreland

StanFromIreland commented Nov 9, 2025

Copy link
Copy Markdown
Member Author

I thought we had reached agreement, he has said „LGTM” above.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@malemburg

malemburg commented Nov 9, 2025

Copy link
Copy Markdown
Member

Oh, ok, didn't see that.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@malemburg
malemburg merged commit 5ba0a1a into python:main Nov 9, 2025
4 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the awaiting merge label Nov 9, 2025
@malemburg

malemburg commented Nov 9, 2025

Copy link
Copy Markdown
Member

Thanks, @StanFromIreland and @bitdancer for the reviews.

StanFromIreland reacted with heart emoji
All reactions
  • ❤️ 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@StanFromIreland
StanFromIreland deleted the encodings/non-ascii branch November 9, 2025 12:45
@hugovk

hugovk commented Nov 10, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Looks like this is causing failures on refleaks buildbots:

  • https://buildbot.python.org/#/release_status
  • https://buildbot.python.org/#/builders/320/builds/3284
  • https://buildbot.python.org/#/builders/920/builds/2026
  • https://buildbot.python.org/#/builders/1610/builds/2391
  • etc
======================================================================
FAIL: test_codecs_lookup (test.test_codecs.CodecNameNormalizationTest.test_codecs_lookup)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.refleak/build/Lib/test/test_codecs.py", line 3891, in test_codecs_lookup
    with self.assertWarns(DeprecationWarning):
         ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
AssertionError: DeprecationWarning not triggered
======================================================================
FAIL: test_rfc2231_bad_character_in_encoding (test.test_email.test_email.TestRFC2231.test_rfc2231_bad_character_in_encoding)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.refleak/build/Lib/test/test_email/test_email.py", line 5741, in test_rfc2231_bad_character_in_encoding
    with self.assertWarns(DeprecationWarning):
         ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
AssertionError: DeprecationWarning not triggered

======================================================================
FAIL: test_value_rfc2231_nonascii_in_charset_of_charset_parameter_value (test.test_email.test_headerregistry.TestContentTypeHeader.test_value_rfc2231_nonascii_in_charset_of_charset_parameter_value)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.refleak/build/Lib/test/test_email/__init__.py", line 160, in <lambda>
    getattr(self, name)(*params))
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.refleak/build/Lib/test/test_email/test_headerregistry.py", line 255, in content_type_as_value
    with self.assertWarns(DeprecationWarning):
         ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
AssertionError: DeprecationWarning not triggered
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@hugovk

hugovk commented Nov 10, 2025

Copy link
Copy Markdown
Member

And I see the fix PR has just been merged :)

#141345

StanFromIreland reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

StanFromIreland added a commit to StanFromIreland/cpython that referenced this pull request Dec 6, 2025
@StanFromIreland
pythongh-136702: Deprecate passing non-ascii *encoding* (str) to `enc…
51b2f1e
…odings.normalize_encoding` (python#140030)

Closes python#136702
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@malemburg malemburg malemburg approved these changes
@bitdancer bitdancer bitdancer approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@StanFromIreland @malemburg @hugovk @bitdancer
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.