Description

serhiy-storchaka

opened on Jul 17, 2025

Issue body actions

Bug report

#83518 changed handling of non-ASCII characters in encodings.normalize_encoding(), but it is still inconsistent with codecs.lookup(), and not even self-consistent. For example:

>>> import encodings
>>> encodings.normalize_encoding('a¤b')
'a_b'
>>> encodings.normalize_encoding('aæb')
'ab'
>>> encodings.normalize_encoding('a-¤')
'a'
>>> encodings.normalize_encoding('a-æ')
'a_'
>>> encodings.normalize_encoding('a-¤-b')
'a_b'
>>> encodings.normalize_encoding('a-æ-b')
'a__b'

You can even get an underscore at the end or repeated underscores in the middle.

cc @malemburg, @vstinner, @shihai1991

Linked PRs

• gh-136736: Fix handling alphanumerical non-ASCII characters in encodings.normalize_encoding() #136737

Activity

1. serhiy-storchaka

   added this to Codecs and encodings issueson Jul 17, 2025
2. serhiy-storchaka

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   topic-unicode

   3.13only security fixesonly security fixes

   3.14bugs and security fixesbugs and security fixes

   3.15bugs and security fixesbugs and security fixes

   on Jul 17, 2025
3. StanFromIreland commented on Jul 17, 2025

   

   StanFromIreland

   on Jul 17, 2025

   Member

   More actions

   Please see the issue I opened yesterday #136702 and also #90666 which is a bit like a duplicate.
4. StanFromIreland commented on Jul 17, 2025

   

   StanFromIreland

   on Jul 17, 2025

   Member

   More actions

   This would not be an issue if we implemented the hard ascii limitation (we currently state it in the docs) like there is for bytes (ValueError).
5. StanFromIreland

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   on Jul 17, 2025
6. serhiy-storchaka

   added a commit that references this issue on Jul 17, 2025

   pythongh-136736: Fix handling alphanumerical non-ASCII characters in …

   8baff4c
7. serhiy-storchaka commented on Jul 17, 2025

   

   serhiy-storchaka

   on Jul 17, 2025

   MemberAuthor

   More actions

   I agree that rejecting any non-ASCII character would also solve this issue. But this issue has more narrow scope -- it is about a defect added in #83518. This is a bug fix and can be backported.
8. StanFromIreland commented on Jul 17, 2025

   

   StanFromIreland

   on Jul 17, 2025

   Member

   More actions

   This implementation will be removed when the C version is implemented.
9. malemburg commented on Jul 27, 2025

   

   malemburg

   on Jul 27, 2025

   Member

   More actions

   Once we have encoding normalization standardized in a single function, these issues should go away.

   Reacted by Stan Ulbrych
10. serhiy-storchaka commented on Jul 28, 2025

    

    serhiy-storchaka

    on Jul 28, 2025

    MemberAuthor

    More actions

    If we consider this a bug, it should be fixed in 3.13 and 3.14. And it is easier to do this while main does not differ too much from other branches.

    On other hand, there are more questions to normalization. See #88886.
11. StanFromIreland commented on Jul 28, 2025

    

    StanFromIreland

    on Jul 28, 2025

    Member

    More actions

    I guess we could backport using the C function, the initial behavior should be the same, and so it can be considered a refactor? It has not changed in 3.13/3.14/3.15.
12. malemburg commented on Jul 28, 2025

    

    malemburg

    on Jul 28, 2025

    Member

    More actions

    BTW: This is not really a bug. The scope and purpose of the encoding normalization function was different and it has always relied on getting proper data from the codecs lookup function.
13. StanFromIreland

    changed the title [-]Inconsisten handling of non-ASCII characters in encodings.normalize_encoding()[/-] [+]Inconsistent handling of non-ASCII characters in encodings.normalize_encoding()[/+] on Feb 19, 2026

Issue actions