Description

nineteendo

opened on Oct 6, 2026

Issue body actions

Feature or enhancement

Is it expected that decoding 2-byte UTF-8 sequences to UCS1 is no faster than decoding them to UCS2? I thought UCS1 would have an advantage because the resulting string uses half as much memory and requires half as many bytes to be written.

$ python -VV
Python 3.15.0rc3 (tags/v3.15.0rc3:8a8eb0b, Oct  2 2026, 19:16:27) [MSC v.1951 64 bit (AMD64)]
$ python -m timeit -s 's = ("\x80" * 65_536).encode()' 's.decode()'  
5000 loops, best of 5: 98.8 usec per loop
$ python -m timeit -s 's = ("\u0100" * 65_536).encode()' 's.decode()'
5000 loops, best of 5: 98.7 usec per loop

For comparison, the corresponding encoding operations do show a difference:

$ python -m timeit -s 's = "\x80" * 65_536' 's.encode()'
5000 loops, best of 5: 69.1 usec per loop
$ python -m timeit -s 's = "\u0100" * 65_536' 's.encode()'           
1000 loops, best of 5: 99.9 usec per loop

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Activity

1. nineteendo

   added

   type-featureA feature request or enhancementA feature request or enhancement

   on Oct 6, 2026
2. aisk

   added

   performancePerformance or resource usagePerformance or resource usage

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   on Oct 7, 2026
3. picnixz commented on Oct 7, 2026

   

   picnixz

   on Oct 7, 2026

   Member

   More actions

   The reasons can vary and we did see some causes due to PGO itself. I suggest that you look directly at the code to analyze what happens or open a DPO thread. We don't have anything actionable here. I suspect that decoding has other bottlenecks that hide what happens. The scanner may also be generic (that I don't remember).
4. picnixz commented on Oct 7, 2026

   

   picnixz

   on Oct 7, 2026

   Member

   More actions

   Maybe @vstinner or @serhiy-storchaka do now the implementation details
5. vstinner

   added

   pendingThe issue will be closed if no feedback is providedThe issue will be closed if no feedback is provided

   on Oct 7, 2026
6. vstinner commented on Oct 7, 2026

   

   vstinner

   on Oct 7, 2026

   Member

   More actions

   Is it expected that decoding 2-byte UTF-8 sequences to UCS1 is no faster than decoding them to UCS2?

   Objects/stringlib/codecs.h generates specialized utf8_decode() functions depending on the buffer kind (UCS1, UCS2, UCS4). It can make some assumptions depending on STRINGLIB_SIZEOF_CHAR.

   Performance is a complex topic. I'm not sure how to answer. I don't have specific expectation on UCS1 vs UCS2 performance.

   If you see an opportunity to optimize further stringlib utf8_decode(), please propose a PR. This issue is just a question, I don't see what can be done. So I just suggest closing the issue.
7. nineteendo commented on Oct 8, 2026

   

   nineteendo

   on Oct 8, 2026

   ContributorAuthor

   More actions

   How about this line? It doesn’t calculate the minimal required size for the output buffer (like we do for escaping strings in the json library):

   cpython/Objects/unicodeobject.c

   Line 5390 in 20cb7fc

   PyObject *u = PyUnicode_New(maxsize, maxchr);

   So, it allocates twice as much memory as necessary.
8. kevinprimeraquecreo-stack

   added this to Project_VKon Oct 18, 2026
9. github-project-automation

   moved this to Todo in Project_VKon Oct 18, 2026
10. kevinprimeraquecreo-stack

    added this to Project_VKon Oct 18, 2026

Issue actions