Description

cmaloney

opened on Nov 1, 2026

Issue body actions

Feature or enhancement

• The tokenizer checks source is utf-8 with _PyTokenizer_ensure_utf8 one character at a time. Reuse _Py_find_first_nonascii to scan faster. That makes compile() for Lib/typing.py about 1% faster.
• Modify _Py_find_first_nonascii to test 64 bytes (32 on 32-bit platforms) of ASCII per branch. The code gets vectorized by some compilers and uses no intrinsics (https://godbolt.org/z/8bM83vx3a). (b'a' * 4096).decode() gets aout 1.5x faster.

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs

• gh-161145: Use _Py_find_first_nonascii in the tokenizer #161146
• gh-161145: Test more bytes simultaneously in find_first_nonascii #161147

Activity

1. cmaloney

   self-assigned this

   on Nov 1, 2026
2. cmaloney

   added

   type-featureA feature request or enhancementA feature request or enhancement

   performancePerformance or resource usagePerformance or resource usage

   topic-unicode

   topic-parser

   on Nov 1, 2026
3. picnixz

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   on Nov 1, 2026
4. picnixz commented on Nov 1, 2026

   

   picnixz

   on Nov 1, 2026

   Member

   More actions

   We have a similar feature for stringlib with find_max_char. Possibility to reuse it maybe?
5. cmaloney commented on Nov 1, 2026

   

   cmaloney

   on Nov 1, 2026

   ContributorAuthor

   More actions

   I looked at that but the unicodeobject.c code and stringlib/codecs.h assume copying the whole value to an output buffer in their API currently and would take a lot bigger refactor to deduplicate than I want to do currently.

   Reacted by Bénédikt Tran and Gregory P. Smith
6. cmaloney

   added 3 commits that reference this issue on Nov 1, 2026

   pythongh-161145: Use _Py_find_first_nonascii in the tokenizer

   ec3b950

   pythongh-161145: Test 32-64 bytes check in find_first_nonascii

   5fee69c

   pythongh-161145: Test more bytes simultaneously find_first_nonascii

   54f4722
7. cmaloney

   added a related issue on Nov 1, 2026

   • Speed up the parser #153568
8. cmaloney

   added a commit that references this issue on Nov 2, 2026

   pythongh-161145: Add NEWS entry

   6636b05

Issue actions