GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-109653: Speedup import of threading module by danielhollas · Pull Request #114509 · python/cpython

Conversation

@danielhollas

@danielhollas danielhollas commented Jan 23, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Delayed import of functools speeds up the import threading by ~50% (2ms -> 1ms) in my testing.

Since the functools module is only used in the internal _register_atexit function that is called by concurrent.futures, this seems like a worthwhile win for users of threading module who do not use asyncio.

Part of #109653

CC @AlexWaygood

  • Issue: Improve import time of various stdlib modules #109653

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@danielhollas
Speed up import of threading module
2a73912
Delayed import of functools leads to 50% speedup
of import time.
@bedevere-app bedevere-app Bot added the awaiting review label Jan 23, 2024
@bedevere-app bedevere-app Bot mentioned this pull request Jan 23, 2024
Improve import time of various stdlib modules #109653
Closed
@bedevere-app

bedevere-app Bot commented Jan 23, 2024

Copy link
Copy Markdown

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@blurb-it
📜🤖 Added by blurb_it.
b70b7e5
@danielhollas

danielhollas commented Jan 24, 2024 •
edited by hugovk
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

To be precise, compiling python with ./configure --enable-optimizations and measuring with python -Ximporttime -c "import threading", I am getting 2.44ms on main and 1.17ms on this PR.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Eclips4 Eclips4 added the performance Performance or resource usage label Jan 24, 2024
@ajoino

ajoino commented Jan 24, 2024

Copy link
Copy Markdown

Just a thought, is it even necessary to use functools.partial here? Could this not be replaced with lambda: f(*args, **kwargs), that would avoid importing functools at all. Is there something I'm missing here?

AlexWaygood reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@AlexWaygood

AlexWaygood commented Jan 24, 2024

Copy link
Copy Markdown
Member

Just a thought, is it even necessary to use functools.partial here? Could this not be replaced with lambda: f(*args, **kwargs), that would avoid importing functools at all. Is there something I'm missing here?

This was my thought as well on first seeing the patch. functools.partial can be faster than a lambda function, but here I doubt it makes a significant difference. (If we wanted to check whether using a lambda here slowed things down, we'd need to do a benchmark using concurrent.futures, since the concurrent.futures module is the only public API that makes use of this private API. It might be possible to write such a benchmark, but it also might be difficult -- not sure.)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@danielhollas

danielhollas commented Jan 24, 2024

Copy link
Copy Markdown
Contributor Author

functools.partial can be faster than a lambda function, but here I doubt it makes a significant difference. (If we wanted to check whether using a lambda here slowed things down, we'd need to do a benchmark using concurrent.futures, since the concurrent.futures module is the only public API that makes use of this private API. It might be possible to write such a benchmark, but it also might be difficult -- not sure.)

Looking at the code, the threading.register_atexit() is only ever called during concurrent.futures import, so I would assume any performance difference here would be marginal?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@AlexWaygood

AlexWaygood commented Jan 24, 2024

Copy link
Copy Markdown
Member

Looking at the code, the threading.register_atexit() is only ever called during concurrent.futures import, so I would assume any performance difference here would be marginal?

Oh, great point 😄

In that case, let's just go with a lambda here -- it seems simpler :)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

AlexWaygood
AlexWaygood reviewed Jan 24, 2024
View reviewed changes
Comment thread Lib/threading.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@danielhollas @AlexWaygood
Let's got with lambda instead of functools.partial
e76cec2
Co-authored-by: Alex Waygood <[email protected]>
AlexWaygood
AlexWaygood approved these changes Jan 24, 2024
View reviewed changes

@AlexWaygood AlexWaygood left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks! I'd love to check with a core dev more familiar with subinterpreters before merging, though (since this feature was specifically added to help with subinterpreter support).

@ericsnowcurrently, there's no reason why switching to a lambda rather than functools.partial could be problematic for subinterpreter support, is there?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

danielhollas reacted with heart emoji
All reactions
  • ❤️ 1 reaction
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting review labels Jan 24, 2024
@AlexWaygood
AlexWaygood requested a review from ericsnowcurrently January 24, 2024 14:18
@danielhollas

danielhollas commented Jan 24, 2024

Copy link
Copy Markdown
Contributor Author

@AlexWaygood thanks!

@ericsnowcurrently, there's no reason why switching to a lambda rather than functools.partial could be problematic for subinterpreter support, is there?

Just a note, if this was a problem, we could still get away with it by simply not doing either: the function is (at least currently) being called without any extra *args or **args arguments so we could make _register_atexit less general and simply pass the callback function directly to _threading_atexits list.

AlexWaygood reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@danielhollas danielhollas mentioned this pull request Jan 24, 2024
gh-109653: Improve import time of logging by lazy loading traceback #112995
Closed
@danielhollas danielhollas changed the title gh-109653: Speedup import of threading module gh-109653: Speedup import of threading module Jan 24, 2024
@AlexWaygood

AlexWaygood commented Jan 31, 2024

Copy link
Copy Markdown
Member

I can't see a way in which this would cause problems — I'll go ahead and merge, since it's been a few days :)

Thanks @danielhollas!

danielhollas reacted with heart emoji danielhollas reacted with rocket emoji
All reactions
  • ❤️ 1 reaction
  • 🚀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@AlexWaygood
AlexWaygood merged commit 5e390a0 into python:main Jan 31, 2024
@bedevere-app bedevere-app Bot removed the awaiting merge label Jan 31, 2024
@danielhollas
danielhollas deleted the import-threading-speedup branch January 31, 2024 10:59
@ericsnowcurrently

ericsnowcurrently commented Feb 1, 2024

Copy link
Copy Markdown
Member

@ericsnowcurrently, there's no reason why switching to a lambda rather than functools.partial could be problematic for subinterpreter support, is there?

I'm not aware of any such reason.

AlexWaygood reacted with heart emoji
All reactions
  • ❤️ 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ericsnowcurrently
ericsnowcurrently reviewed Feb 1, 2024
View reviewed changes
Comment thread Lib/threading.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

aisk pushed a commit to aisk/cpython that referenced this pull request Feb 11, 2024
@danielhollas @AlexWaygood @aisk
pythongh-109653: Speedup import of threading module (python#114509)
e7225f8
Avoiding an import of functools leads to 50% speedup of import time.

Co-authored-by: Alex Waygood <[email protected]>
@danielhollas danielhollas mentioned this pull request Apr 27, 2025
gh-118761: Add test_lazy_import for more modules #133057
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ericsnowcurrently ericsnowcurrently ericsnowcurrently left review comments
@AlexWaygood AlexWaygood AlexWaygood approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

performance Performance or resource usage

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@danielhollas @ajoino @AlexWaygood @ericsnowcurrently @Eclips4
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.