GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-161145: Use _Py_find_first_nonascii in the tokenizer by cmaloney · Pull Request #161146 · python/cpython

Conversation

@cmaloney

@cmaloney cmaloney commented Nov 1, 2026 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Skip sequence of ASCII at once rather than one char per loop reusing faster code from unicodeobject. Can't use the checking from stringlib or unicodeobject fully because those assume copying/writing to an output buffer which slows down this case. Line numbers are only needed for the error message, so count them there. The function is exported because test_peg_generator builds the tokenizer sources into an extension module.

The check drops from about 0.9% of compile() time to about 0.1% (perf record). CPU time per compile(), PGO+LTO, 20 paired runs on one pinned core:

Workload main this runs faster
typing.py 6.72 ms 1.2% faster 17/20
all top-level Lib/*.py 241 ms 0.7% faster 17/20

_PyTokenizer_ensure_utf8 instructions during compile(), callgrind:

File main this
typing.py 1,811,154 132,909
_pydatetime.py 1,246,691 91,566
test_email/test_contentmanager.py 487,525 43,273
  • Issue: Speed up UTF-8 validation in tokenizer and ASCII scanning in UTF-8 decoding #161145

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@cmaloney @claude
pythongh-161145: Use _Py_find_first_nonascii in the tokenizer
ec3b950
Skip sequence of ASCII at once rather than one char per loop reusing
faster code from unicodeobject. Can't use the checking from stringlib
or unicodeobject fully because those assume copying/writing to an output
buffer which slows down this case. Line numbers are only needed for the
error message, so count them there.  The function is exported because
test_peg_generator builds the tokenizer sources into an extension
module.

The check drops from about 0.9% of compile() time to about 0.1%
(perf record).  CPU time per compile(), PGO+LTO, 20 paired runs on one
pinned core:

| Workload | main | this | runs faster |
|---|---:|---:|---:|
| typing.py | 6.72 ms | 1.2% faster | 17/20 |
| all top-level Lib/*.py | 241 ms | 0.7% faster | 17/20 |
| synthetic CJK literals | 4.32 ms | 1.3% faster | 16/20 |

_PyTokenizer_ensure_utf8 instructions during compile(), callgrind:

| File | main | this |
|---|---:|---:|
| typing.py | 1,811,154 | 132,909 |
| _pydatetime.py | 1,246,691 | 91,566 |
| test_email/test_contentmanager.py | 487,525 | 43,273 |
| synthetic Cyrillic literals | 5,825,590 | 3,832,223 |
| synthetic CJK literals | 3,450,590 | 2,479,019 |

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@cmaloney
cmaloney requested review from lysnikolaou and pablogsal as code owners November 1, 2026 23:21
@bedevere-app bedevere-app Bot added awaiting review type-feature A feature request or enhancement labels Nov 1, 2026
@bedevere-app bedevere-app Bot mentioned this pull request Nov 1, 2026
Speed up UTF-8 validation in tokenizer and ASCII scanning in UTF-8 decoding #161145
Open
@cmaloney cmaloney added the skip news label Nov 1, 2026
@picnixz

picnixz commented Nov 1, 2026

Copy link
Copy Markdown
Member

Mmh. Benchmarks don't seem convincing. 1% faster is... quite small IMO.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@cmaloney

cmaloney commented Nov 1, 2026

Copy link
Copy Markdown
Contributor Author

The diff is small and this is currently fully duplicated code. This to me is more a refactor than perf change, and sets up for "we can make decode 1.5x faster for ASCII" (the next commit).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@cmaloney

cmaloney commented Nov 1, 2026

Copy link
Copy Markdown
Contributor Author

Related to: #153568

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pablogsal pablogsal Awaiting requested review from pablogsal pablogsal is a code owner
@lysnikolaou lysnikolaou Awaiting requested review from lysnikolaou lysnikolaou is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting review skip news type-feature A feature request or enhancement

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@cmaloney @picnixz
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.