GeezerVison is showing github.com — In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores. · Issue #81932 · python/cpython
In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores. #81932Description
| BPO |
37751 |
| Nosy |
@malemburg, @gpshead, @vstinner, @mark-summerfield, @ezio-melotti, @methane, @miss-islington, @shihai1991, @qigangxu, @akdor1154, @bodograumann |
| PRs |
bpo-37751: Fix normalizestring() with hyphens and spaces converted to underscores #15092bpo-37751: Document the change in What's New in Python 3.9 #17997bpo-37751: Document the change in What's New in Python 3.9 #23096bpo-37751: Update codecs.register() doc. #25643[3.9] bpo-37751: Document codecs.lookup() change in What's New in Python 3.9 (GH-23096) #25659[3.9] bpo-37751: Update codecs.register() doc. (GH-25643) #25677 |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2021-04-24.16:55:03.898>
created_at = <Date 2019-08-03.11:34:13.329>
labels = ['type-bug', '3.9', 'expert-unicode']
title = "In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores."
updated_at = <Date 2022-01-25.00:29:03.446>
user = 'https://github.com/qigangxu' bugs.python.org fields:
activity = <Date 2022-01-25.00:29:03.446>
actor = 'gregory.p.smith'
assignee = 'none'
closed = True
closed_date = <Date 2021-04-24.16:55:03.898>
closer = 'shihai1991'
components = ['Unicode']
creation = <Date 2019-08-03.11:34:13.329>
creator = 'qigangxu'
dependencies = []
files = []
hgrepos = []
issue_num = 37751
keywords = ['patch']
message_count = 31.0
messages = ['348953', '348954', '348956', '348959', '349448', '350086', '350087', '350155', '359970', '359971', '359972', '359973', '359974', '360005', '391648', '391652', '391653', '391654', '391664', '391666', '391671', '391785', '392077', '392159', '392161', '392178', '395730', '395764', '411536', '411537', '411538']
nosy_count = 11.0
nosy_names = ['lemburg', 'gregory.p.smith', 'vstinner', 'mark', 'ezio.melotti', 'methane', 'miss-islington', 'shihai1991', 'qigangxu', 'akdor1154', 'bodograumann']
pr_nums = ['15092', '17997', '23096', '25643', '25659', '25677']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue37751'
versions = ['Python 3.9'] Activityqigangxu commented on Aug 3, 2019 MannequinAuthor More actions qigangxu commented on Aug 3, 2019 MannequinAuthor More actions and I will try to fix it. shihai1991 commented on Aug 3, 2019 qigangxu commented on Aug 3, 2019 MannequinAuthor More actions malemburg commented on Aug 12, 2019 Jordon is right. Conversion has to be to underscores, not hyphens. I guess this bug was introduced when the normalization function was converted to C. vstinner commented on Aug 21, 2019 vstinner commented on Aug 21, 2019 Thanks for the fix Jordon Xu.
IMHO this change is not strictly a bugfix, but more like an enhancement. I close the issue.
If you consider that a backport to Python 3.7 and 3.8 is needed, please say so. qigangxu commented on Aug 22, 2019 MannequinAuthor More actions Thanks vstinner. I also don't think it's necessary to backport to the old version. Close this issue is fine. hroncok commented on Jan 14, 2020 vstinner commented on Jan 14, 2020
The change is backwards incompatible and a backport would break things. See for example how it breaks latexcodec:
I reopen the issue. I proposed PR 17997 to *document* the incompatible change in What's New in Python 3.8. IMO it's a deliberate change and it's correct.
I rely on Marc-Andre Lemburg who implemented codecs and encodings modules. He wrote: "Jordon is right. Conversion has to be to underscores, not hyphens.". changed the title [-]In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.[/-] [+]In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.[/+] on Apr 23, 2021malemburg commented on Apr 23, 2021 On 23.04.2021 07:47, Inada Naoki wrote:
Inada Naoki <[email protected]> added the comment:
I think it is too late. Python 3.9 has been released already. Reverting the change is also breaking change.
PEP-100 says:
"Search functions are expected to take one argument, the encoding name in all lower case letters and with hyphens and spaces converted to underscores"
https://www.python.org/dev/peps/pep-0100/#codecs-coder-decoders-lookup
But codecs.register() says:
"Search functions are expected to take one argument, being the encoding name in all lower case letters".
I don't know historical reason why two document are inconsistent.
https://docs.python.org/3/library/codecs.html#codecs.register
I guess just an oversight on my part.
PEP-100 is certainly what I meant and implemented. I should have also
made it clear in PEP-100 that I meant lower case ASCII letters. vstinner commented on Apr 23, 2021 shihai1991 commented on Apr 24, 2021 Thanks Marc-Andre for your supplement of PEP-100.
Thanks Inada, victor for your review and merge.
After PR-23096 merged, I suggest to close this bpo.
If there have any other questions, we can reopen it again. vstinner commented on Apr 27, 2021 methane commented on Apr 28, 2021 methane commented on Apr 28, 2021 vstinner commented on Apr 28, 2021 Thanks Inada-san for documenting the change in codecs.register() doc! bodograumann commented on Jun 13, 2021 Unfortunately this is not quite finished yet.
First of all, the change is bigger than what is documented: “Changed in version 3.9: Hyphens and spaces are converted to underscore.“
In reality, now
| Normalization works as follows: all non-alphanumeric
| characters except the dot used for Python package names are
| collapsed and replaced with a single underscore, e.g. ' -;#'
| becomes '_'. Leading and trailing underscores are removed.”
Cf. encodings/init.py
Secondly, this change breaks lots of iconv codecs with the python-iconv binding. E.g. ASCII//TRANSLIT is now normalized to ascii_translit, which iconv does not understand. Codec names which use hyphens also break and iinm not all of them have aliases in iconv without hyphens.
Cf. python-iconv #4
The codecs api feels extremely well-fitting for integrating iconv in python and any alternative I can think of seems unsatisfactory.
Please advise. vstinner commented on Jun 13, 2021
The codecs api feels extremely well-fitting for integrating iconv in python and any alternative I can think of seems unsatisfactory.
This issue is now closed, would you mind to open a new issue? gpshead commented on Jan 25, 2022 gpshead commented on Jan 25, 2022 (note: this might not be the true cause of that issue; though it sounds potentially related - I haven't investigated far enough yet) gpshead commented on Jan 25, 2022
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|