GeezerVison is showing github.com — gh-130167: Improve speed of `inspect.formatannotation` by replacing `re` by donbarbos · Pull Request #130242 · python/cpython
Conversation
donbarbos
commented
Feb 18, 2025
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
timeit benchmark with my script:
inspect_bench.py:
import re
import timeit
from typing import List, Dict, Any, Union
def old_format_annotation(annotation):
if getattr(annotation, "__module__", None) == "typing":
def repl(match):
text = match.group()
return text.removeprefix("typing.")
return re.sub(r"[\w\.]+", repl, repr(annotation))
return repr(annotation)
def new_format_annotation(annotation):
if getattr(annotation, "__module__", None) == "typing":
return (repr(annotation)
.replace(".typing.", "@TYPING@")
.replace("typing.", "")
.replace("@TYPING@", ".typing."))
return repr(annotation)
old_time = timeit.timeit(lambda: old_format_annotation(Union[List[str], Dict[str, Any]]), number=100_000)
new_time = timeit.timeit(lambda: new_format_annotation(Union[List[str], Dict[str, Any]]), number=100_000)
print(f"Old version (re.sub): {old_time:.6f}s")
print(f"New version (.replace()): {new_time:.6f}s")
print(f"Difference: {old_time / new_time:.2f}x")
Result: 2.0s -> 1.18s = x1.74 as fast
$ ./python -B inspect_bench.py
Old version (re.sub): 2.043900s
New version (.replace()): 1.175867s
Difference: 1.74x
✔️ This was also the only use of re in module so i got rid of the import
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
donbarbos
changed the title
Improve speed of inspect.formatannotation by replacing re
gh-130167: Improve speed of inspect.formatannotation by replacing re
Feb 18, 2025
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm less sure this is worth it, the replacement isn't clearly better maintenence-wise.
What do the benchmarks look like if you extract repl to a module-level _formatannotation_repl and use .sub() on a pre-compiled pattern?
A
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
donbarbos
commented
May 1, 2025
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
|
sorry, I forgot how python was compiled then :( but I'm pretty sure I compiled it with gcc last time
so I reran the test with the following config args '--enable-optimizations' '--with-lto' 'CC=/usr/bin/clang-18' and got a new result for my bench:
Old version (re.sub): 0.982520s
New version (.replace()): 0.701049s
Difference: 1.40x # sometimes it's 1.35x
and if we rewrite old version of benchmark as you suggested:
_pattern = re.compile(r"[\w\.]+")
def _repl(match):
text = match.group()
return text.removeprefix("typing.")
def old_format_annotation(annotation):
if getattr(annotation, "__module__", None) == "typing":
return re.sub(_pattern, _repl, repr(annotation))
return repr(annotation)
we will get a bigger difference:
Old version (re.sub): 1.027774s
New version (.replace()): 0.698902s
Difference: 1.47x # the result can drop to 1.40
P.S.: another reason i like the PR version is because we get rid of the re import that is used once here
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Out of curiosity, why use the little replace dance in the refactored code instead of just return repr(annotation).removeprefix("typing.")?
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
charmander
commented
Nov 27, 2025
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
Out of curiosity, why use the little replace dance in the refactored code instead of just return repr(annotation).removeprefix("typing.")?
- Union[typing.List[str], typing.Dict[str, typing.Any]]
+ Union[List[str], Dict[str, Any]]
The replace dance is pretty messy, though, and doesn’t preserve behaviour for footyping.Bar – not that applying a regex to repr output is perfectly reliable to begin with (def foo() -> Literal["typing."]: ...).
For what it’s worth:
pat = re.compile(r"\b(?<!\.)typing\.")
pat.sub("", repr(annotation))
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
This PR is stale because it has been open for 30 days with no activity.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
The PR is stale and has some issues, so I am closing it.
@charmander (or others) Your suggestion looks good (I tested it locally, performance gain is good). Would you like to open a PR?
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Reviewers
hugovk
hugovk left review comments
AA-Turner
Awaiting requested review from AA-Turner
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|