GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | |||||||||||||||||||||||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | |||||||||||||||||||||||||||
GeezerVison is showing github.com — gh-161145: Use _Py_find_first_nonascii in the tokenizer by cmaloney · Pull Request #161146 · python/cpython Conversation
|
| Workload | main | this | runs faster |
|---|---|---|---|
| typing.py | 6.72 ms | 1.2% faster | 17/20 |
| all top-level Lib/*.py | 241 ms | 0.7% faster | 17/20 |
_PyTokenizer_ensure_utf8 instructions during compile(), callgrind:
| File | main | this |
|---|---|---|
| typing.py | 1,811,154 | 132,909 |
| _pydatetime.py | 1,246,691 | 91,566 |
| test_email/test_contentmanager.py | 487,525 | 43,273 |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
Skip sequence of ASCII at once rather than one char per loop reusing faster code from unicodeobject. Can't use the checking from stringlib or unicodeobject fully because those assume copying/writing to an output buffer which slows down this case. Line numbers are only needed for the error message, so count them there. The function is exported because test_peg_generator builds the tokenizer sources into an extension module. The check drops from about 0.9% of compile() time to about 0.1% (perf record). CPU time per compile(), PGO+LTO, 20 paired runs on one pinned core: | Workload | main | this | runs faster | |---|---:|---:|---:| | typing.py | 6.72 ms | 1.2% faster | 17/20 | | all top-level Lib/*.py | 241 ms | 0.7% faster | 17/20 | | synthetic CJK literals | 4.32 ms | 1.3% faster | 16/20 | _PyTokenizer_ensure_utf8 instructions during compile(), callgrind: | File | main | this | |---|---:|---:| | typing.py | 1,811,154 | 132,909 | | _pydatetime.py | 1,246,691 | 91,566 | | test_email/test_contentmanager.py | 487,525 | 43,273 | | synthetic Cyrillic literals | 5,825,590 | 3,832,223 | | synthetic CJK literals | 3,450,590 | 2,479,019 | Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
Mmh. Benchmarks don't seem convincing. 1% faster is... quite small IMO. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
The diff is small and this is currently fully duplicated code. This to me is more a refactor than perf change, and sets up for "we can make decode 1.5x faster for ASCII" (the next commit). |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
Related to: #153568 |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
Successfully merging this pull request may close these issues.
There was an error while loading. Please reload this page.