GeezerVison is showing github.com — gh-109653: Just import `recursive_repr` in `dataclasses` by sobolevn · Pull Request #109822 · python/cpython
Conversation
sobolevn
commented
Sep 25, 2023
•
edited by AlexWaygood
Loading
Uh oh!
There was an error while loading. Please reload this page.
Import time is unchanged. Before:
» pyperf timeit 'import dataclasses'
.....................
Mean +- std dev: 106 ns +- 2 ns
After:
» pyperf timeit 'import dataclasses'
.....................
Mean +- std dev: 105 ns +- 2 ns
But, we now drop two dependencies: functools and _thread and add just one: reprlib
https://github.com/python/cpython/blame/f2eaa92b0cc5a37a9e6010c7c6f5ad1a230ea49b/Lib/dataclasses.py#L248-L266
It was strange to have
# This function's logic is copied from "recursive_repr" function in
# reprlib module to avoid dependency.
But import two other heavier modules. The most interesting part is that functools always imports reprlib inside. See #109653 (comment)
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
The windows failure is known and unrelated: #109739
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
I don't think this change is worth making. It seems mostly like code churn: there's no actual problem being solved.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Is there a difference if the functools import is deferred to within the vendored _recursive_repr function (& we remove the top-level)?
A
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
sobolevn
commented
Sep 25, 2023
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
|
@AA-Turner it makes no difference, because inspect imports 80% of all stdlib, including functools :)
inspect is only used in two places:
|
cls_annotations = inspect.get_annotations(cls) |
|
text_sig = str(inspect.signature(cls)).replace(' -> None', '') |
But, it still does not improve the import time:
» pyperf timeit 'import dataclasses'
.....................
Mean +- std dev: 105 ns +- 1 ns
I also tried to defer inspect, but it is still the same :(
|
AA-Turner reacted with thumbs up emoji
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
...it makes no difference, because inspect imports 80% of all stdlib, including functools :)
inspect is only used in two places:
|
cls_annotations = inspect.get_annotations(cls) |
|
text_sig = str(inspect.signature(cls)).replace(' -> None', '') |
But, it still does not improve the import time:
» pyperf timeit 'import dataclasses'
.....................
Mean +- std dev: 105 ns +- 1 ns
I also tried to defer inspect, but it is still the same :(
@sobolevn
Are these pyperf timings actually outputting accurate data when it comes to imports? The numbers seem surprisingly low to me.
I get similar numbers (scale-wise) from pyperf but they don't match with other timings I've done.
pyperf timeit "import dataclasses" 1
.....................
Mean +- std dev: 84.6 ns +- 4.1 ns
python -Ximporttime -c "import dataclasses" (truncated to only show dataclasses):
import time: self [us] | cumulative | imported package
...
import time: 961 | 18112 | dataclasses
Which lines up roughly with the timings I get from hyperfine comparing launching python with no imports to importing dataclasses.
hyperfine -w10 -r100 -N 'python -c "pass"' 'python -c "import dataclasses"'
Benchmark 1: python -c "pass"
Time (mean ± σ): 11.0 ms ± 0.6 ms [User: 8.4 ms, System: 2.4 ms]
Range (min … max): 10.4 ms … 13.0 ms 100 runs
Benchmark 2: python -c "import dataclasses"
Time (mean ± σ): 28.5 ms ± 1.0 ms [User: 23.8 ms, System: 4.5 ms]
Range (min … max): 27.0 ms … 32.0 ms 100 runs
Summary
'python -c "pass"' ran
2.59 ± 0.17 times faster than 'python -c "import dataclasses"'
From previous experiments mentioned in #109653 (comment) I would expect deferring inspect to remove close to half of the import time. However, it's not actually very useful to do so as any use of @dataclass will end up importing inspect in order to get the annotations, so practically you wouldn't actually see any benefit from delaying the import unless none of the modules you use make any dataclasses.
Slightly more directly related, and also mentioned in #109653 (comment) the definition of _recursive_repr you have in dataclasses makes use of functools.wraps which uses functools.partial which in turn uses reprlib.recursive_repr on its own __repr__ so perhaps this change still makes sense from that angle?
Footnotes
-
I also get surprisingly high values for pyperf timeit 'import sys' in comparison. ↩
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
A change in the other direction (per reprlib's own implementation):
diff --git a/Lib/dataclasses.py b/Lib/dataclasses.py
index 84f8d68ce0..8579f6d605 100644
--- a/Lib/dataclasses.py
+++ b/Lib/dataclasses.py
@@ -4,7 +4,6 @@
import types
import inspect
import keyword
-import functools
import itertools
import abc
import _thread
@@ -252,7 +251,6 @@ def _recursive_repr(user_function):
# call.
repr_running = set()
- @functools.wraps(user_function)
def wrapper(self):
key = id(self), _thread.get_ident()
if key in repr_running:
@@ -263,6 +261,15 @@ def wrapper(self):
finally:
repr_running.discard(key)
return result
+
+ wrapper.__module__ = getattr(user_function, '__module__')
+ wrapper.__doc__ = getattr(user_function, '__doc__')
+ wrapper.__name__ = getattr(user_function, '__name__')
+ wrapper.__qualname__ = getattr(user_function, '__qualname__')
+ wrapper.__annotations__ = getattr(user_function, '__annotations__', {})
+ wrapper.__type_params__ = getattr(user_function, '__type_params__', ())
+ wrapper.__wrapped__ = user_function
+
return wrapper
class InitVar:
A
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
@AA-Turner we still have _thread import (1 _thread vs 1 reprlib) and quite a lot of boilerplate code that can go out of sync with the reprlib.recursive_repr (it happened before #109819) for no real reason.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
boilerplate code that can go out of sync with the reprlib.recursive_repr (it happened before #109819)
IMO, this is a good reason to merge this PR.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As long as there's not a performance regression due to importing reprlib, I'm okay with this. I have not checked the performance.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
As long as there's not a performance regression due to importing reprlib, I'm okay with this. I have not checked the performance.
You're already importing reprlib indirectly via functools so this shouldn't change import time if that's a concern (functools will still be imported via inspect so you won't see an improvement by removing that direct import).
|
sobolevn reacted with thumbs up emoji
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Agreed that reprlib is already imported (checked with 3.11).
|
sobolevn reacted with thumbs up emoji
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Reviewers
encukou
encukou approved these changes
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|