GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-83938, gh-122476: Stop incorrectly RFC 2047 encoding non-ASCII email addresses by medmunds · Pull Request #122540 · python/cpython

Conversation

@medmunds

@medmunds medmunds commented Aug 1, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

This PR fixes gh-83938 and fixes gh-122476, which have the same underlying issue.

Email generators had been incorrectly flattening non-ASCII email addresses to RFC 2047 encoded-word format, leaving them undeliverable. (RFC 2047 prohibits use of encoded-word in an addr-spec.) This change raises a ValueError when attempting to flatten an EmailMessage with a non-ASCII addr-spec and a policy with utf8=False. (Exception: If the non-ASCII address originated from parsing a message, it will be flattened as originally parsed, without error.)

Non-ASCII email addresses are supported when using a policy with utf8=True (such as email.policy.SMTPUTF8) under RFCs 6531 and 6532.

Non-ASCII email address domains (but not localparts) can also be used with non-SMTPUTF8 policies by encoding the domain as an IDNA A-label. (The email package does not perform this encoding, because it cannot know whether the caller wants IDNA 2003, IDNA 2008, or some other variant such as UTS-46.)

  • Issue: EmailMessage bad encoding for international domain #83938
  • Issue: EmailMessage bad encoding for non-ASCII localpart #122476

📚 Documentation preview 📚: https://cpython-previews--122540.org.readthedocs.build/

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@medmunds
medmunds requested a review from a team as a code owner August 1, 2024 00:35
@bedevere-app bedevere-app Bot mentioned this pull request Aug 1, 2024
EmailMessage bad encoding for international domain #83938
Closed
@bedevere-app bedevere-app Bot added the awaiting review label Aug 1, 2024
@medmunds
medmunds force-pushed the fix-issues-83938-122476 branch 2 times, most recently from d1f0bdc to 2e0696c Compare August 1, 2024 00:43
@medmunds

medmunds commented Aug 1, 2024

Copy link
Copy Markdown
Contributor Author

This is based on #81074 (comment):

we should probably be raising an error if the rendering policy does not have utf8=True and we don't have an "original source line" from parsing a message (which is the case here), rather than using the incorrect RFC2047 encoding.

Checking part.token_type == 'addr-spec' seemed like the simplest approach.

An alternative would be to introduce a new NonASCIIDomainLiteralDefect paralleling NonASCIILocalPartDefect and apply it in _header_value_parser.get_domain_literal(). And add NonASCIIAddrSpecDefect as a superclass of both. Then change _refold_parse_tree() to check any(isinstance(d, NonASCIIAddrSpecDefect) for d in part.all_defects) (and perhaps move it up with the other UnicodeEncodeError logic). (If we go this direction, PR #122477 will also need an update.)

Also, I think charset == 'unknown-8bit' is only possible in _refold_parse_tree() when the non-ASCII characters resulted from parsing an existing message: see the UndecodableBytesDefect logic just above the new code. (The added tests seem to confirm this.)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@medmunds
medmunds force-pushed the fix-issues-83938-122476 branch from 2e0696c to cbedf5d Compare August 1, 2024 01:11
medmunds added 2 commits July 31, 2024 18:35
@medmunds
Stop incorrectly RFC 2047 encoding non-ASCII email addresses
0abca3e
Email generators had been incorrectly flattening non-ASCII email
addresses to RFC 2047 encoded-word format, leaving them undeliverable.
(RFC 2047 prohibits use of encoded-word in an addr-spec.)
This change raises a ValueError when attempting to flatten an
EmailMessage with a non-ASCII addr-spec and a policy with utf8=False.
(Exception: If the non-ASCII address originated from parsing a message,
it will be flattened as originally parsed, without error.)

Non-ASCII email addresses are supported when using a policy with
utf8=True (such as email.policy.SMTPUTF8) under RFCs 6531 and 6532.

Non-ASCII email address domains (but not localparts) can also be used
with non-SMTPUTF8 policies by encoding the domain as an IDNA A-label.
(The email package does not perform this encoding, because it cannot
know whether the caller wants IDNA 2003, IDNA 2008, or some other
variant such as UTS python#46.)
@medmunds
Blurbs
faa4006
@medmunds
medmunds force-pushed the fix-issues-83938-122476 branch from cbedf5d to faa4006 Compare August 1, 2024 01:36
@medmunds medmunds mentioned this pull request Aug 21, 2024
Ticket 35581: Updated django.core.mail to Python's modern email API [WIP, DO-NOT-MERGE] medmunds/django#2
Closed
3 of 5 tasks
medmunds added 2 commits October 21, 2024 13:11
@medmunds
Merge branch 'main' into fix-issues-83938-122476
f8f5eef
@medmunds
Merge branch 'main' into fix-issues-83938-122476
99914e3
@picnixz picnixz changed the title gh-83938: Stop incorrectly RFC 2047 encoding non-ASCII email addresses gh-83938, gh-122476: Stop incorrectly RFC 2047 encoding non-ASCII email addresses Dec 3, 2024
@ZeroIntensity ZeroIntensity added the topic-email label Dec 6, 2024
bitdancer
bitdancer requested changes Mar 31, 2025
View reviewed changes

@bitdancer bitdancer left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your analysis is solid and the fix looks great. We'll need a follow on PR to have smtplib handle the new error, but that should be a trivial PR.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

medmunds reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
Comment thread Lib/test/test_email/test_generator.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/email/_header_value_parser.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Misc/NEWS.d/next/Library/2024-07-31-17-22-10.gh-issue-83938.TtUa-c.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the awaiting review label Mar 31, 2025
@bedevere-app

bedevere-app Bot commented Mar 31, 2025

Copy link
Copy Markdown

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

And if you don't make the requested changes, you will be poked with soft cushions!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot added the awaiting changes label Mar 31, 2025
medmunds added 2 commits April 1, 2025 11:45
@medmunds
Merge branch 'main' into fix-issues-83938-122476
819c0bc
@medmunds
fixup! Stop incorrectly RFC 2047 encoding non-ASCII email addresses
bd6845d
- Incorporate PR review feedback
- Improve docs
@medmunds
medmunds force-pushed the fix-issues-83938-122476 branch from 5d60c1c to bd6845d Compare April 1, 2025 20:14
@medmunds

medmunds commented Apr 1, 2025

Copy link
Copy Markdown
Contributor Author

I have made the requested changes; please review again

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot added awaiting change review and removed awaiting changes labels Apr 1, 2025
@bedevere-app

bedevere-app Bot commented Apr 1, 2025

Copy link
Copy Markdown

Thanks for making the requested changes!

@bitdancer: please review the changes made to this pull request.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app
bedevere-app Bot requested a review from bitdancer April 1, 2025 20:26
bitdancer
bitdancer requested changes May 18, 2025
View reviewed changes

@bitdancer bitdancer left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry it took me so long to get back to this.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/email/_header_value_parser.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Misc/NEWS.d/next/Library/2024-07-31-17-23-06.gh-issue-122476.TtUa-c.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot added awaiting changes and removed awaiting change review labels May 18, 2025
@bedevere-app bedevere-app Bot removed the awaiting merge label Apr 22, 2026
@bedevere-app

bedevere-app Bot commented Apr 22, 2026

Copy link
Copy Markdown

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot added the awaiting changes label Apr 22, 2026
@bitdancer

bitdancer commented Apr 23, 2026

Copy link
Copy Markdown
Member

Looking at the code more closely, it is supposed to already be handing this case. The fact that it isn't indicates there's a bug there, and there were certainly missing tests. I will poke at this.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bitdancer added 4 commits April 26, 2026 13:44
@bitdancer
Remove incomplete fix and temporarily disable new tests.
8f3f6fd
@bitdancer
Move mime-parameter folding to top of loop.
f47e029
The mime parameter folder doesn't make use of the encoding check
done be the code that is now below it, it does its own.  So it
makes more sense to take that branch first.  This will simplify
subsequent changes.
@bitdancer
Fix the bug, update the doc changes.
73c7b6b
This is a more complete fix, covering any syntax part where encoded
words are not permitted, and the doc changes are adjusted accordingly.
There is also no need for a new exception, since HeaderWriteError
already exists.

The fix itself is to use a separate code loop to fold parts that
may not have encoded words, guaranteeing that we do not do incorrect
encoding.  This opens a door to simplifying the main folding loop,
but that is a much bigger refactoring job better left for another time.
@bitdancer
Add some tests where the local part is folded.
0a259f4
Behavior when folding in parts versus rendering on one line takes
different code paths, so make sure both work.
@bitdancer

bitdancer commented Apr 27, 2026

Copy link
Copy Markdown
Member

I've crafted a more complete fix for the bug. If you could review it, @medmunds, that would be great.

Since I switched to using the existing HeaderWriteError, I'm wondering if we need the versionchanged at all at this point, if anyone has input on that.

medmunds reacted with heart emoji
All reactions
  • ❤️ 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bitdancer bitdancer mentioned this pull request Apr 27, 2026
gh-87112: Ensure that only digits convertible to integers are accepted as section number in MIME header parameter #136877
Open
@medmunds

This comment was marked as outdated.

Sign in to view
@medmunds
Update doc changes
6f5e5e1
Fix typo; bump versionchanged.
@read-the-docs-community

read-the-docs-community Bot commented Apr 29, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

Documentation build overview

📚 cpython-previews | 🛠️ Build #32489666 | 📁 Comparing 37f13db against main (76b3923)

  🔍 Preview build  

65 files changed · + 1 added · ± 64 modified

+ Added

  • c-api/sentinel.html

± Modified

  • contents.html
  • download.html
  • genindex-B.html
  • genindex-C.html
  • genindex-D.html
  • genindex-E.html
  • genindex-F.html
  • genindex-G.html
  • genindex-H.html
  • genindex-I.html
  • and 54 more...
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@medmunds
Fix reference in docs
6a6803e
medmunds
medmunds commented Apr 29, 2026
View reviewed changes

@medmunds medmunds left a comment

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bitdancer this LGTM. I pushed a couple of fixes to the docs.

I'd be inclined to keep the versionchanged note. (Unless there's some style guide policy against that for bug fixes.) There seems to be a popular misconception—in many email libraries and systems—that any MIME header with non-ASCII chars can just be run through rfc2047. The versionchanged note makes it clear dropping rfc2047 here was a deliberate fix in Python's email package, and might help discourage future requests to add it back in.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
bitdancer
bitdancer approved these changes Apr 30, 2026
View reviewed changes
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting changes labels Apr 30, 2026
@bitdancer
Merge branch 'main' into fix-issues-83938-122476
71636d3
@bitdancer

bitdancer commented Apr 30, 2026

Copy link
Copy Markdown
Member

Since this is a bug fix that isn't going to get backported because of backward compatibility concerns, we'll keep the versionchanged. We can always delete it if someone objects.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bitdancer

bitdancer commented Apr 30, 2026

Copy link
Copy Markdown
Member

It occurs to me that if it is worth a versionchanged there should also be a What's New entry. Do you want to write one?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@medmunds

medmunds commented Apr 30, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

It occurs to me that if it is worth a versionchanged there should also be a What's New entry. Do you want to write one?

Something like this? [edited]

Index: Doc/whatsnew/3.15.rst
<+>UTF-8
===================================================================
diff --git a/Doc/whatsnew/3.15.rst b/Doc/whatsnew/3.15.rst
--- a/Doc/whatsnew/3.15.rst	(revision 71636d347e37db41a11a21419137fd7eb3423f74)
+++ b/Doc/whatsnew/3.15.rst	(date 1777572899407)
@@ -914,6 +914,16 @@
   (Contributed by Eric Froemling in :gh:`149085`.)
 
 
+email
+-----
+
+* Email generators now raise an error when an :class:`.EmailMessage` cannot be
+  accurately flattened due to a non-ASCII email address (mailbox) in an address
+  header. :attr:`.EmailPolicy.utf8` offers options for supporting Email Address
+  Internationalization (EAI).
+  (Contributed by R David Murray and Mike Edmunds in :gh:`122540`.)
+
+
 functools
 ---------
 

[Edited: trying to capture the attention of users who might be affected and direct them to the docs that have complete details.]

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bitdancer

bitdancer commented Apr 30, 2026

Copy link
Copy Markdown
Member

Yeah, that looks good. Though I'd say "The documentation for...offers" just to make it clear that it isn't the policy option itself that provides the support.

medmunds reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bitdancer

bitdancer commented Apr 30, 2026

Copy link
Copy Markdown
Member

Although I guess technically it does...maybe your wording is correct.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

medmunds added 2 commits April 30, 2026 12:43
@medmunds
Fix typo in docs
fc31373
@medmunds
Add whatsnew item
37f13db
@medmunds
medmunds requested a review from AA-Turner as a code owner April 30, 2026 19:44
@bitdancer
bitdancer merged commit d96ffc1 into python:main May 1, 2026
53 of 54 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the awaiting merge label May 1, 2026
bitdancer pushed a commit that referenced this pull request May 1, 2026
@medmunds
gh-81074: Allow non-ASCII addr_spec in email.headerregistry.Address (#…
b605573
…122477)

The email.headerregistry.Address constructor raised an error if
addr_spec contained a non-ASCII character. (But it fully supports
non-ASCII in the separate username and domain args.) This change
removes the error for a non-ASCII addr_spec, as well as the
Defect that triggered it.  In the unicode era non-ascii is not a
defect, though it is an error when an attempt is made to serialize
it to ascii.  The serialization issue was handled in #122540.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bitdancer bitdancer bitdancer approved these changes
@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

stale Stale PR or inactive for long period of time. topic-email

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

EmailMessage bad encoding for non-ASCII localpart EmailMessage bad encoding for international domain

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@medmunds @bitdancer @ZeroIntensity
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.