GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-109653: Fix py312 regression in the import time of `random` by AlexWaygood · Pull Request #110221 · python/cpython

Conversation

@AlexWaygood

@AlexWaygood AlexWaygood commented Oct 2, 2023 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

As an optimisation to reduce the import time of the module, random first tries to import sha512 from the internal _sha512 module before falling back to hashlib. The problem, however, is that Python no longer has a _sha512 module! It was removed in 0b13575, by @gpshead. That means we're currently always falling back to the slow path in random.py, leading to the import time of random being far slower than it should be.

Importing sha512 from the correct module in the fast path cuts 60% off the import time of random.

  • Issue: Improve import time of various stdlib modules #109653

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
gpshead reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@AlexWaygood
Reduce the import time of random by 60%
a9e3194
@AlexWaygood AlexWaygood added the performance Performance or resource usage label Oct 2, 2023
@AlexWaygood
AlexWaygood requested a review from rhettinger as a code owner October 2, 2023 14:45
@bedevere-app bedevere-app Bot added the awaiting review label Oct 2, 2023
@AlexWaygood AlexWaygood changed the title Reduce the import time of random by 60% gh-109653: Reduce the import time of random by 60% Oct 2, 2023
@bedevere-app bedevere-app Bot mentioned this pull request Oct 2, 2023
Improve import time of various stdlib modules #109653
Closed
AlexWaygood
AlexWaygood commented Oct 2, 2023
View reviewed changes
Comment thread Misc/NEWS.d/next/Library/2023-10-02-15-40-10.gh-issue-109653.iB0peK.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.
@AlexWaygood
AlexWaygood requested a review from gpshead October 2, 2023 14:53
@JelleZijlstra

JelleZijlstra commented Oct 2, 2023

Copy link
Copy Markdown
Member

Given that this is a regression, should we backport it into 3.12?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@AlexWaygood

AlexWaygood commented Oct 2, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

Given that this is a regression, should we backport it into 3.12?

I was wondering that. I'd vote in favour of doing so, since it doesn't seem particularly high-risk to me. But I'd like to hear Raymond's and/or Greg's thoughts.
JelleZijlstra and gpshead reacted with thumbs up emoji
All reactions
  • 👍 2 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@AlexWaygood AlexWaygood added the stdlib Standard Library Python modules in the Lib/ directory label Oct 2, 2023
rhettinger
rhettinger approved these changes Oct 2, 2023
View reviewed changes
@bedevere-app bedevere-app Bot added awaiting core review and removed awaiting review labels Oct 2, 2023
@rhettinger

rhettinger commented Oct 2, 2023

Copy link
Copy Markdown
Contributor

A backport to 3.12 would be reasonable.
AlexWaygood reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@AlexWaygood
Update Misc/NEWS.d/next/Library/2023-10-02-15-40-10.gh-issue-109653.i…
e39e4dd 
…B0peK.rst
@AlexWaygood AlexWaygood added the needs backport to 3.12 only security fixes label Oct 2, 2023
@AlexWaygood
AlexWaygood enabled auto-merge (squash) October 2, 2023 22:30
@AlexWaygood

AlexWaygood commented Oct 2, 2023

Copy link
Copy Markdown
Member Author

A backport to 3.12 would be reasonable.

Great, I've scheduled the backport. Thanks for the review!
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@AlexWaygood AlexWaygood changed the title gh-109653: Reduce the import time of random by 60% gh-109653: Fix py312 regression in the import time of random Oct 2, 2023
AA-Turner
AA-Turner approved these changes Oct 2, 2023
View reviewed changes
@AlexWaygood
AlexWaygood merged commit 21a6263 into python:main Oct 2, 2023
@AlexWaygood
AlexWaygood deleted the random-import-time branch October 2, 2023 22:56
@miss-islington

miss-islington commented Oct 2, 2023

Copy link
Copy Markdown
Contributor

Thanks @AlexWaygood for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot removed the awaiting core review label Oct 2, 2023
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Oct 2, 2023
@AlexWaygood @miss-islington
pythongh-109653: Reduce the import time of random by 60% (pythonGH-…
6277f18 
…110221)

(cherry picked from commit 21a6263)

Co-authored-by: Alex Waygood <[email protected]>
@bedevere-app

bedevere-app Bot commented Oct 2, 2023

Copy link
Copy Markdown

GH-110247 is a backport of this pull request to the 3.12 branch.
gpshead reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot removed the needs backport to 3.12 only security fixes label Oct 2, 2023
@bedevere-app

This comment was marked as duplicate.

Sign in to view
@bedevere-bot

bedevere-bot commented Oct 2, 2023

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 Debian Clang LTO + PGO 3.x has failed when building commit 21a6263.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1084/builds/2183) and take a look at the build logs.
  4. Check if the failure is related to this commit (21a6263) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1084/builds/2183

Failed tests:

  • test.test_concurrent_futures.test_shutdown

Failed subtests:

  • test_interpreter_shutdown - test.test_concurrent_futures.test_shutdown.ProcessPoolSpawnProcessPoolShutdownTest.test_interpreter_shutdown

Summary of the results of the build (if available):

==

Click to see traceback logs 
Traceback (most recent call last):
  File "/var/lib/buildbot/workers/arm64-clang/3.x.gps-arm64-debian.clang.lto-pgo/build/Lib/test/test_concurrent_futures/test_shutdown.py", line 50, in test_interpreter_shutdown
    self.assertEqual(out.strip(), b"apple")
AssertionError: b'' != b'apple'
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
AlexWaygood added a commit that referenced this pull request Oct 2, 2023
@miss-islington @AlexWaygood
[3.12] gh-109653: Reduce the import time of random by 60% (GH-110221)…
41e8e15 
… (#110247)

gh-109653: Fix regression in the import time of `random` in Python 3.12 (GH-110221)
(cherry picked from commit 21a6263)

Co-authored-by: Alex Waygood <[email protected]>
gpshead
gpshead reviewed Oct 3, 2023
View reviewed changes
Comment thread Lib/random.py
@@ -65,7 +65,7 @@

try:
# hashlib is pretty heavy to load, try lean internal module first

@gpshead gpshead Oct 3, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code is technically awkward... It tried to speed up import time and did so by circumventing hashlib which means that it is loading and using the slowest possible sha512 implementation by default (hashlib will pick up openssl 3's accelerated sha512 support by default on most platforms and not use our builtin). so faster startup time for a slower runtime computation? thankfully this is only ever used by seed() which is a single/constant number of calls for most programs on tiny data so there is zero reason to care about sha512 performance for its purposes. The slower implementation may still be faster on small seed data anyways due to less setup overhead.

Nothing to do here. This works for random's purposes & thanks for the fixup. But this being a regression in the first place demonstrates how fragile direct use of internal details can be. (and indirectly how much in need of an overhaul hashlib.py could use)

I'd file an issue rather than leaving this comment in the merged PR void if there were anything concrete to describe and tackle, there isn't. :)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
AlexWaygood reacted with heart emoji
All reactions
  • ❤️ 1 reaction
Glyphack pushed a commit to Glyphack/cpython that referenced this pull request Sep 2, 2024
@AlexWaygood @Glyphack
pythongh-109653: Reduce the import time of random by 60% (python#11…
c35196c 
…0221)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gpshead gpshead gpshead left review comments
@rhettinger rhettinger rhettinger approved these changes
@AA-Turner AA-Turner AA-Turner approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

performance Performance or resource usage stdlib Standard Library Python modules in the Lib/ directory

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

7 participants

@AlexWaygood @JelleZijlstra @rhettinger @miss-islington @bedevere-bot @gpshead @AA-Turner
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.