Description

assignee = None
closed_at = None
created_at = <Date 2021-07-23.10:18:59.285>
labels = ['type-bug', '3.9', 'expert-unicode']
title = 'Codec name normalization breaks custom codecs'
updated_at = <Date 2022-01-25.02:02:30.022>
user = 'https://github.com/bodograumann'

activity = <Date 2022-01-25.02:02:30.022>
actor = 'methane'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Unicode']
creation = <Date 2021-07-23.10:18:59.285>
creator = 'bodograumann'
dependencies = []
files = []
hgrepos = []
issue_num = 44723
keywords = []
message_count = 1.0
messages = ['398042']
nosy_count = 5.0
nosy_names = ['gregory.p.smith', 'vstinner', 'ezio.melotti', 'methane', 'bodograumann']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44723'
versions = ['Python 3.9']

Activity

1. bodograumann commented on Jul 23, 2021

   

   bodograumannmannequin

   on Jul 23, 2021

   MannequinAuthor

   More actions

   This is a follow up on https://bugs.python.org/issue37751 concerning normalization of codec names.

   First of all, the changes made therein are not documented correctly.
   In the implementation
   | Normalization works as follows: all non-alphanumeric
   | characters except the dot used for Python package names are
   | collapsed and replaced with a single underscore, e.g. ' -;#'
   | becomes '_'. Leading and trailing underscores are removed.”
   Cf. encodings/init.py

   The documentation however only states that:
   | Search functions are expected to take one argument, being the encoding name in all lower case letters with hyphens and spaces converted to underscores
   Cf. https://docs.python.org/3/library/codecs.html#codecs.register

   Secondly, this change breaks lots of iconv codecs with the python-iconv binding. E.g. ASCII//TRANSLIT is now normalized to ascii_translit, which iconv does not understand. Codec names which use hyphens also break and iinm not all of them have aliases in iconv without hyphens.
   Cf. python-iconv #4

   How about first looking up the given name and only then, if the given name could not be found, looking for the codec by its normalized name?
2. bodograumann

   added

   3.9 (EOL)end of lifeend of life

   topic-unicode

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Jul 23, 2021
3. ezio-melotti

   transferred this issue fromon Apr 10, 2022
4. serhiy-storchaka

   added this to Codecs and encodings issueson May 22, 2022
5. StanFromIreland

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   and removed

   3.9 (EOL)end of lifeend of life

   on Jul 11, 2025
6. serhiy-storchaka commented on Jul 21, 2025

   

   serhiy-storchaka

   on Jul 21, 2025

   Member

   More actions

   I am not sure that any normalization is needed in codecs.lookup() (except maybe normalizing the letter case). It must be done in the search function. For example, standard encodings.search_function() needs to convert encoding name to module name -- therefore filter out /, \, :, and other characters that can be special in filesystem.
7. serhiy-storchaka commented on Jul 21, 2025

   

   serhiy-storchaka

   on Jul 21, 2025

   Member

   More actions

   @vstinner, I think that #81932 should be rolled back. Even more, there is no reason to convert space to minus.
8. serhiy-storchaka

   added 3 commits that reference this issue on Jul 28, 2025

   pythongh-88886: Remove excessive encoding name normalization

   02ae0e1

   pythongh-88886: Remove excessive encoding name normalization

   2df6169

   pythongh-88886: Remove excessive encoding name normalization

   ae1cae2
9. serhiy-storchaka

   added a commit that references this issue on Sep 9, 2025

   gh-88886: Remove excessive encoding name normalization (GH-137167)

   af58a6f
10. lkollar

    added a commit that references this issue on Sep 9, 2025

    pythongh-88886: Remove excessive encoding name normalization (pythonG…

    0a3e79b

Issue actions