GeezerVison is showing github.com — gh-109653: Speedup import of threading module by danielhollas · Pull Request #114509 · python/cpython
Conversation
Delayed import of functools speeds up the import threading by ~50% (2ms -> 1ms) in my testing.
Since the functools module is only used in the internal _register_atexit function that is called by concurrent.futures, this seems like a worthwhile win for users of threading module who do not use asyncio.
Part of #109653
CC @AlexWaygood
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Delayed import of functools leads to 50% speedup
of import time.
|
Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.
If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
danielhollas
commented
Jan 24, 2024
•
edited by hugovk
Loading
Uh oh!
There was an error while loading. Please reload this page.
|
To be precise, compiling python with ./configure --enable-optimizations and measuring with python -Ximporttime -c "import threading", I am getting 2.44ms on main and 1.17ms on this PR.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Just a thought, is it even necessary to use functools.partial here? Could this not be replaced with lambda: f(*args, **kwargs), that would avoid importing functools at all. Is there something I'm missing here?
|
AlexWaygood reacted with thumbs up emoji
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Just a thought, is it even necessary to use functools.partial here? Could this not be replaced with lambda: f(*args, **kwargs), that would avoid importing functools at all. Is there something I'm missing here?
This was my thought as well on first seeing the patch. functools.partial can be faster than a lambda function, but here I doubt it makes a significant difference. (If we wanted to check whether using a lambda here slowed things down, we'd need to do a benchmark using concurrent.futures, since the concurrent.futures module is the only public API that makes use of this private API. It might be possible to write such a benchmark, but it also might be difficult -- not sure.)
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
functools.partial can be faster than a lambda function, but here I doubt it makes a significant difference. (If we wanted to check whether using a lambda here slowed things down, we'd need to do a benchmark using concurrent.futures, since the concurrent.futures module is the only public API that makes use of this private API. It might be possible to write such a benchmark, but it also might be difficult -- not sure.)
Looking at the code, the threading.register_atexit() is only ever called during concurrent.futures import, so I would assume any performance difference here would be marginal?
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Looking at the code, the threading.register_atexit() is only ever called during concurrent.futures import, so I would assume any performance difference here would be marginal?
Oh, great point 😄
In that case, let's just go with a lambda here -- it seems simpler :)
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks! I'd love to check with a core dev more familiar with subinterpreters before merging, though (since this feature was specifically added to help with subinterpreter support).
@ericsnowcurrently, there's no reason why switching to a lambda rather than functools.partial could be problematic for subinterpreter support, is there?
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
danielhollas reacted with heart emoji
|
@AlexWaygood thanks!
@ericsnowcurrently, there's no reason why switching to a lambda rather than functools.partial could be problematic for subinterpreter support, is there?
Just a note, if this was a problem, we could still get away with it by simply not doing either: the function is (at least currently) being called without any extra *args or **args arguments so we could make _register_atexit less general and simply pass the callback function directly to _threading_atexits list.
|
AlexWaygood reacted with thumbs up emoji
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
danielhollas
changed the title
gh-109653: Speedup import of threading module
gh-109653: Speedup import of threading module
Jan 24, 2024
|
I can't see a way in which this would cause problems — I'll go ahead and merge, since it's been a few days :)
Thanks @danielhollas!
|
danielhollas reacted with heart emoji
danielhollas reacted with rocket emoji
All reactions
-
❤️
1 reaction
-
🚀
1 reaction
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
@ericsnowcurrently, there's no reason why switching to a lambda rather than functools.partial could be problematic for subinterpreter support, is there?
I'm not aware of any such reason.
|
AlexWaygood reacted with heart emoji
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
aisk
pushed a commit
to aisk/cpython
that referenced
this pull request
Feb 11, 2024
Avoiding an import of functools leads to 50% speedup of import time.
Co-authored-by: Alex Waygood <[email protected]>
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|