GeezerVison is showing github.com — codecs.lookup() ignores non-ASCII characters, whereas encodings.normalize_encoding() copies them · Issue #83518 · python/cpython
codecs.lookup() ignores non-ASCII characters, whereas encodings.normalize_encoding() copies them #83518Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2020-10-14.15:46:55.467>
created_at = <Date 2020-01-14.21:54:42.124>
labels = ['library', '3.10']
title = 'codecs.lookup() ignores non-ASCII characters, whereas encodings.normalize_encoding() copies them'
updated_at = <Date 2020-10-14.15:55:50.301>
user = 'https://github.com/vstinner' bugs.python.org fields:
activity = <Date 2020-10-14.15:55:50.301>
actor = 'shihai1991'
assignee = 'none'
closed = True
closed_date = <Date 2020-10-14.15:46:55.467>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2020-01-14.21:54:42.124>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 39337
keywords = ['patch']
message_count = 11.0
messages = ['360004', '363645', '364180', '364449', '364452', '364456', '377772', '378281', '378626', '378628', '378630']
nosy_count = 4.0
nosy_names = ['lemburg', 'vstinner', 'serhiy.storchaka', 'shihai1991']
pr_nums = ['18845', '18987', '19069', '22219', '22360', '17997']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue39337'
versions = ['Python 3.10'] Activityvstinner commented on Jan 14, 2020 bpo-37751 changed codecs.lookup() in a subtle way: non-ASCII characters are now ignored, whereas they were copied unmodified previously.
I would prefer that codecs.lookup() and encodings.normalize_encoding() behave the same. Either always ignore or always copy.
Moreover, it seems like there is no test on how the encoding names are normalized in codecs.register(). I recall that using codecs.register() in an unit test causes troubles since there is no API to unregister a search function. Maybe we should just add a private function for test in _testcapi.
Serhiy Storchaka wrote an example on my PR:
https://github.com/python/cpython/pull/17997/files
There are other differences. For example, normalize_encoding("КОИ-8") returns "кои_8", but codecs.lookup normalizes it to "8".
The comment in the sources is also not correct.
shihai1991 commented on Mar 8, 2020
I would prefer that codecs.lookup() and encodings.normalize_encoding() behave the same. Either always ignore or always copy.
How about calling encodings.normalize_encoding() in codecs.normalizestring() to keep same behavior?(I create PR18845)
Maybe we should just add a private function for test in _testcapi
I can try to add some test cases in next weekend ;) shihai1991 commented on Mar 14, 2020
How about calling encodings.normalize_encoding() in codecs.normalizestring() to keep same behavior?(I create PR18845)
I have try this idea, but it make the testcase of test_io.py failed because some object will call codecs.Lookup() in __del__().-->extension module will be cleaned before calling __del__().
I would prefer that codecs.lookup() and encodings.normalize_encoding() behave the same. Either always ignore or always copy.
I try to add a _Py_normalize_unicode_encoding() in unicodeobject.c to support non-ASCII encoding names' normalization(PR18987), but this PR caused many testcases failed.
For example:
In master:
python3.9 -c "print('a\xac\u1234\u20ac\u8000\U0010ffff'.encode('iso-8859-15', 'namereplace'))"
result:
b'a\xac\\N{ETHIOPIC SYLLABLE SEE}\xa4\\N{CJK UNIFIED IDEOGRAPH-8000}\\U0010ffff'
after PR18987:
./python -c "print('a\xac\u1234\u20ac\u8000\U0010ffff'.encode('iso-8859-15', 'namereplace'))"
result:
b'a\xac\\N{ETHIOPIC SYLLABLE SEE}\\N{EURO SIGN}\\N{CJK UNIFIED IDEOGRAPH-8000}\\U0010ffff' shihai1991 commented on Mar 17, 2020
Maybe we should just add a private function for test in _testcapi.
Oh, there have a problem with this idea:
struct _is is defined in internal/pycore_pystate.h.
_testcapi must test the public Python C API, not CPython internal C API vstinner commented on Mar 17, 2020
_testcapi must test the public Python C API, not CPython internal C API
Use _testinternalcapi in this case. shihai1991 commented on Mar 17, 2020
Use _testinternalcapi in this case.
oh, forgive me. I don't atttention this extension module before :(
vstinner commented on Oct 1, 2020 vstinner commented on Oct 8, 2020 vstinner commented on Oct 14, 2020 vstinner commented on Oct 14, 2020 Thanks Hai Shi for the change and for new codecs and encodings tests. shihai1991 commented on Oct 14, 2020 Thanks for everyone's continus review :)
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|