GeezerVison is showing github.com — email.parser can insert extraneous spaces when parsing rfc2047 headers with policy.default · Issue #128110 · python/cpython
email.parser can insert extraneous spaces when parsing rfc2047 headers with policy.default #128110DescriptionBug report
Bug description:
When using email.parser with the modern email.policy.default, the parser incorrectly inserts a space between adjacent rfc2047 encoded-words that are separated by folding whitespace. This can result in splitting words or names in unexpected places in the parsed headers.
Example:
Python 3.13.1 (main, Dec 3 2024, 17:59:52) [Clang 16.0.0 (clang-1600.0.26.4)] on darwin
>>> from email import message_from_bytes, policy
>>> from email.message import EmailMessage
>>> address = "Bérénice-Amélie Rosemonde Dûbois-Bénard <[email protected]>"
>>> message = EmailMessage()
>>> message["From"] = address
>>> message_bytes = message.as_bytes()
>>> default_parsed = message_from_bytes(message_bytes, policy=policy.default)
>>> default_parsed_from = default_parsed["From"].addresses[0]
>>> assert default_parsed_from == address
Traceback (most recent call last):
File "<python-input-22>", line 1, in <module>
assert default_parsed_from == address
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
>>> print(default_parsed_from); print(address)
Bérénice-Amélie Rosemonde Dûbois-Béna rd <[email protected]>
Bérénice-Amélie Rosemonde Dûbois-Bénard <[email protected]>
>>> print(default_parsed["From"].addresses[0].display_name)
Bérénice-Amélie Rosemonde Dûbois-Béna rdNotice the unexpected space in the parsed version. (I have more-problematic examples involving variations of the Scunthorpe problem, but they're not suitable for GitHub. This example uses Python's modern email API to generate the message, but the bug applies to parsing messages from any source.)
This seems to be caused by two consecutive rfc2047 encoded-words [edit: whether or not they cross a fold, see later comment]:
>>> print(message_bytes.decode())
From: =?utf-8?b?QsOpcsOpbmljZS1BbcOpbGllIFJvc2Vtb25kZSBEw7tib2lzLULDqW5h?=
=?utf-8?q?rd?= <[email protected]> RFC 2047 section 6.2 requires that space that separates adjacent encoded-words be ignored. (Specifically to allow splitting encoded-words at any character—behavior that the policy.default email.generator relies on.)
Note that the legacy compat32 parser did not have this problem. (The legacy parser requires separately calling legacy email.header.decode_header() to decode rfc2047.)
>>> from email.header import decode_header
>>> compat32_parsed = message_from_bytes(message_bytes, policy=policy.compat32)
>>> compat32_parsed_from = "".join(
... segment if charset is None and isinstance(segment, str)
... else segment.decode(charset or "ascii")
... for segment, charset in decode_header(compat32_parsed["From"])
... )
>>> assert compat32_parsed_from == address # (success)
>>> print("original:", address); print("compat32:", compat32_parsed_from); print(" default:", default_parsed_from)
original: Bérénice-Amélie Rosemonde Dûbois-Bénard <[email protected]>
compat32: Bérénice-Amélie Rosemonde Dûbois-Bénard <[email protected]>
default: Bérénice-Amélie Rosemonde Dûbois-Béna rd <[email protected]>CPython versions tested on:
3.12, 3.13
Operating systems tested on:
Linux, macOS
Linked PRs
Activitysrinivasreddy commented on Dec 20, 2024 @medmunds you are working on fixing this ? If not, I am working on it medmunds commented on Dec 20, 2024 ContributorAuthor More actions @srinivasreddy I already have a bunch of email-related PRs that have been stuck awaiting review for months, so I'm not working on new PRs until those move forward. If you have a fix for this issue, that's great! medmunds commented on Dec 22, 2024 ContributorAuthor More actions On further investigation, it seems parsing with policy.default fails to remove "any 'linear-white-space' that separates a pair of adjacent 'encoded-word's" whether or not folding is involved:
>>> message_from_bytes(b"From: =??q?ab?= =??q?cd?= <foo@example>\n", policy=policy.default)["From"]
'ab cd <foo@example>'
# expected: 'abcd <foo@example>' changed the title [-]email.parser inserts extraneous spaces when parsing folded rfc2047 headers with policy.default [/-] [+]email.parser can insert extraneous spaces when parsing rfc2047 headers with policy.default [/+] on Dec 22, 2024srinivasreddy commented on Dec 23, 2024 I started the work on it. medmunds commented on Dec 23, 2024 ContributorAuthor More actions Also, it looks like the problem only affects address headers:
>>> # Subject header is correct:
>>> message_from_bytes(b"Subject: =??q?ab?= =??q?cd?=\n", policy=policy.default)["Subject"]
'abcd'
>>> # Address header has the bug:
>>> message_from_bytes(b"To: =??q?ab?= =??q?cd?= <foo@example>\n", policy=policy.default)["To"]
'ab cd <foo@example>' medmunds commented on Dec 24, 2024 ContributorAuthor More actions Until this gets fixed, here's a workaround that can repair an EmailMessage with misparsed headers. And a short check to see if you need the workaround. medmunds commented on Feb 27, 2025 ContributorAuthor More actions srinivasreddy commented on Feb 28, 2025 @medmunds apologies . I didn’t get enough time for this . This ticket is up for grabs . jakajancar commented on May 27, 2025
Also, it looks like the problem only affects address headers:
Not sure if related or not, but Message-ID also has a whitespace problem:
#134812 multiflexi commented on Feb 26, 2026 I think the issue affects Subject as well:
[TLP:CLEAR] [Max CVSS 10.0] Cisco Catalyst SD-WAN opravuje aktivně zneužívanou zranitelnost
gets encoded with add_header() as:
b'Subject: [TLP:CLEAR] [Max CVSS 10.0] Cisco Catalyst SD-WAN opravuje \n =?utf-8?q?aktivn=C4=9B?= =?utf-8?q?zneu=C5=BE=C3=ADvanou?= zranitelnost\n\n'
which gets decoded as aktivnězneužívanou instead of aktivně zneužívanou.
When I change the string to aaaaaaaaaaa[TLP:CLEAR] [Max CVSS 10.0] Cisco Catalyst SD-WAN opravuje aktivně zneužívanou zranitelnost, the issue changes and it produces 2 spaces between opravuje and aktivně. medmunds commented on May 11, 2026 ContributorAuthor More actions
I think the issue affects Subject as well:
[TLP:CLEAR] [Max CVSS 10.0] Cisco Catalyst SD-WAN opravuje aktivně zneužívanou zranitelnost
gets encoded with add_header() as:
b'Subject: [TLP:CLEAR] [Max CVSS 10.0] Cisco Catalyst SD-WAN opravuje \n =?utf-8?q?aktivn=C4=9B?= =?utf-8?q?zneu=C5=BE=C3=ADvanou?= zranitelnost\n\n'
which gets decoded as aktivnězneužívanou instead of aktivně zneužívanou.
@multiflexi I can't reproduce that in Python 3.14.4 (or 3.15.0a8). The space between the two words seems to be correctly encoded as _ inside a single encoded-word:
>>> from email import policy, message_from_bytes
>>> from email.message import EmailMessage
>>> msg = EmailMessage()
>>> subject = "[TLP:CLEAR] [Max CVSS 10.0] Cisco Catalyst SD-WAN opravuje aktivně zneužívanou zranitelnost"
>>> msg.add_header("Subject", subject)
>>> print(msg.as_bytes().decode())
Subject: [TLP:CLEAR] [Max CVSS 10.0] Cisco Catalyst SD-WAN opravuje
=?utf-8?q?aktivn=C4=9B_zneu=C5=BE=C3=ADvanou?= zranitelnost
>>> ### The space gets ^ encoded as '_'
>>> parsed = message_from_bytes(msg.as_bytes(), policy=policy.default)
>>> parsed["Subject"]
'[TLP:CLEAR] [Max CVSS 10.0] Cisco Catalyst SD-WAN opravuje aktivně zneužívanou zranitelnost'
>>> parsed["Subject"] == subject
TrueIf you're still seeing it, please open a new issue with the exact Python version you're using and sample code to reproduce the problem. If the subject you show gets encoded as ... =?utf-8?q?aktivn=C4=9B?= =?utf-8?q?zneu=C5=BE=C3=ADvanou?= ..., that's a bug in the email generator. (That encoded subject would correctly decode as ... aktivnězneužívanou ... because the space between the encoded-words should be ignored per rfc2047.) added a commit that references this issue on May 11, 2026gh-128110: Fix rfc2047 whitespace handling in email parser address he…
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|