Bug description:

The email header parser silently drops Unicode whitespace other than SP and HTAB (for example U+00A0 NO-BREAK SPACE, U+2003 EM SPACE, U+3000 IDEOGRAPHIC SPACE) when it follows a space in an address display name. A message generated by the email package itself doesn't round-trip:

import email
from email import policy
from email.message import EmailMessage
from email.headerregistry import Address

m = EmailMessage()
m['To'] = Address('. \xa0b', 'u', 'example.com')
print(m['To'])          # =?utf-8?q?=2E_=C2=A0?=b <[email protected]>
m2 = email.message_from_string(m.as_string(), policy=policy.default)
print(ascii(m2['To'].addresses[0].display_name))   # '. b', expected '. \xa0b'

Received mail is affected too: To: =?utf-8?q?x_=C2=A0b?= <[email protected]> gives the display name 'x b'.

The cause is get_fws() in Lib/email/_header_value_parser.py:

def get_fws(value):
    """FWS = 1*WSP
    ...
    newvalue = value.lstrip()

str.lstrip() strips all Unicode whitespace, but FWS is only SP and HTAB (RFC 5322 §3.2.2: FWS = ([*WSP CRLF] 1*WSP) / obs-FWS; RFC 5234: WSP = SP / HTAB). The extra characters end up in a WhiteSpaceTerminal, whose value is always a single space, so they are lost.

I couldn't find a test or comment saying stripping all Unicode whitespace was intended.

CPython versions tested on:

3.13, CPython main branch

Operating systems tested on:

Windows

Linked PRs

• gh-161678: Don't treat non-ASCII whitespace as folding white space in email headers #161679