GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-117953: Refactor Import Machinery Code for Extension Modules by ericsnowcurrently · Pull Request #118116 · python/cpython

Conversation

@ericsnowcurrently

@ericsnowcurrently ericsnowcurrently commented Apr 20, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

This is a large change that involves refactoring the code that loads extension modules. The objective is partly to improve the clarity of the code, but mostly it's so that I can execute a subset of the constituent operations in a subsequent PR.

The changes include:

  • ...

While this aligns with the sort of thing I've wanted to do for a long time, the main motivation at the moment is to drastically reduce the likelihood of problems from loading a single-phase init extension module in an isolated subinterpreter. Nearly all the changes here come from what I found I needed for a (relatively) simple solution to the isolation problem. Consequently, much of the refactoring follows the natural divisions in the flow of operations when loading an extension module.

For the sake of reducing the burden of code review for such a large change, I have purposefully avoided changing any behavior in this PR, regardless of how benign it might seem. That said, if it's still too much I'm okay with splitting this up.

For context, this is the first of 3 branches in a chain. Here are the other two, in order:

  • track-all-ext-modules - actual (small) changes in behavior that I didn't put in this PR
  • load-ext-in-main-interp-2 - always load modules in the main interpreter first (supercedes gh-117487)
  • Issue: Single-Phase Init Extension Module Init Functions Still Run in Isolated Interpreters #117953

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
ericsnowcurrently added 30 commits April 16, 2024 11:10
@ericsnowcurrently
Factor out _PyImport_RunDynamicModule().
cc67849
@ericsnowcurrently
Drop _PyImport_LoadDynamicModuleWithSpec().
2e5ecde
@ericsnowcurrently
Add an assert.
b810cd8
@ericsnowcurrently
Factor out fix_up_extension_for_interpreter().
8d0ef38
@ericsnowcurrently
Do _extensions_cache_set() first.
7f930cc
@ericsnowcurrently
Calculate path_bytes ahead of time.
1ffeaef
@ericsnowcurrently
Move the audit entry to _imp_create_dynamic_impl().
7500285
@ericsnowcurrently
_Py_ext_module_loader_info_from_spec() -> _Py_ext_module_loader_info_…
3c0b170
…init_from_spec()
@ericsnowcurrently
Add info.newcontext.
02e24cc
@ericsnowcurrently
Split up _PyImport_RunDynamicModule() along cross-interpreter-safe bo…
ba5ffa7
…undaries.
@ericsnowcurrently
Drop _PyImport_RunDynamicModule().
b895966
@ericsnowcurrently
Always close the file handle.
76fa273
@ericsnowcurrently
Revert the fclose() change.
1e936c5
@ericsnowcurrently
Factor out create_dynamic().
6bb07b9
@ericsnowcurrently
Drop _PyImport_FixupExtensionObject().
2705e65
@ericsnowcurrently
Move setting __file__ to fix_up_extension().
98d8b4b
@ericsnowcurrently
Pass tstate through to fix_up_extension().
ddf1135
@ericsnowcurrently
Add a note about _PyImport_FixupBuiltin().
694a763
@ericsnowcurrently
Always expect a valid modules dict passed to fix_up_extension().
72e0d73
@ericsnowcurrently
Factor out reload_singlephase_extension().
3f14a67
@ericsnowcurrently
res.singlephase -> res.kind
b5f5868
@ericsnowcurrently
Factor out _extensions_cache_init().
a5f7535
@ericsnowcurrently
Factor out _extensions_cache_find_unlocked().
e8d674a
@ericsnowcurrently
Use _PyImport_RunModInitFunc() in create_builtin().
27b365d
@ericsnowcurrently
Clean up _PyImport_FixupBuiltin().
5128702
@ericsnowcurrently
Add some asserts to _PyImport_RunModInitFunc().
aaec360
@ericsnowcurrently
Drop the replace arg to _extensions_cache_set().
f97671e
@ericsnowcurrently
Check the module returned by import_find_extension() to ensure single…
2366983
…-phase init.
@ericsnowcurrently
Allow mod->md_def to be NULL.
d5039ea
@ericsnowcurrently
Call _PyImport_RunModInitFunc() in reload_singlephase_extension().
67d217b
ericsnowcurrently added 10 commits April 18, 2024 18:15
@ericsnowcurrently
res->err is an array, not just a pointer.
9e40287
@ericsnowcurrently
Factor out import_run_extension() (and drop create_dynamic()).
b8c2a1d
@ericsnowcurrently
Factor out update_extension_cache() and drop fix_up_extension_for_int…
967c649
…erpreter().
@ericsnowcurrently
We already know the def is okay.
aecf56c
@ericsnowcurrently
Revive fix_up_extension_for_interpreter().
0d03d23
@ericsnowcurrently
Add a distinct _Py_ext_module_loader_info.path (and old path -> filen…
cace102
…ame).
@ericsnowcurrently
Do not call update_extensions_cache() in fix_up_extension().
3cf0db0
@ericsnowcurrently
Let the modules arg to fix_up_extension_for_interpreter() be NULL.
9b3088a
@ericsnowcurrently
Let the caller of update_extensions_cache() decide if m_copy should b…
4ac1644
…e populated.
@ericsnowcurrently
Fix the update_extensions_cache() signature.
393a177
@ericsnowcurrently ericsnowcurrently added the skip news label Apr 20, 2024
@ericsnowcurrently
ericsnowcurrently requested a review from encukou April 20, 2024 01:24
@bedevere-app bedevere-app Bot mentioned this pull request Apr 20, 2024
Single-Phase Init Extension Module Init Functions Still Run in Isolated Interpreters #117953
Closed
@ericsnowcurrently

ericsnowcurrently commented Apr 20, 2024

Copy link
Copy Markdown
Member Author

@encukou, content aside, do you think this change is too big, considering that it's mostly moving things around and adding a few data structures to help with that? I'd be glad to split this up if that would help, but would rather not. 😄

Don't spend much time on an answer. I'm not sure what my own answer would be yet, other than I think it's fine in priniciple. I just wanted to see if you had a quick opinion, given your experience with the code I'm changing.

FWIW, I'm going to let this sit over the weekend and give myself a review on Monday.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@encukou

encukou commented Apr 22, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Don't spend much time on an answer.

It's giant! :)

Do you see some parts that would be an improvement on their own, which could be split out and reviewed separately? Perhaps some mechanical stuff like replacing MODULES with get_modules_dict would make the big patch more focused (though if it's only that, it won't help much).

If you don't, this patch is certainly reviewable -- over a few days. I sent a similar giant patch recently, where I couldn't figure out how to break it down further.

ericsnowcurrently reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

This was referenced Apr 22, 2024
gh-117953: Track All Extension Modules #118156
Closed
gh-116322: Add Py_mod_gil module slot #116882
Merged
gh-117953: Small Cleanup of Extensions-Related Machinery Code #118167
Merged
@ericsnowcurrently

ericsnowcurrently commented Apr 23, 2024

Copy link
Copy Markdown
Member Author

Sounds good. I'm splitting this up into 6 different PRs. They should all be up by tomorrow.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ericsnowcurrently added a commit that referenced this pull request Apr 23, 2024
@ericsnowcurrently
gh-117953: Small Cleanup of Extensions-Related Machinery Code (gh-118167
23950be
)

This is a collection of very basic cleanups I've pulled out of gh-118116.  It is mostly renaming variables and moving a couple bits of code in functionally equivalent ways.
This was referenced Apr 23, 2024
gh-117953: Cleanups For fix_up_extension() in import.c #118192
Merged
gh-117953: Add Internal struct _Py_ext_module_loader_info #118194
Merged
gh-117953: Split Up _PyImport_LoadDynamicModuleWithSpec() #118203
Merged
gh-117953: Share More Machinery Code Between Builtin and Dynamic Extensions #118204
Merged
@ericsnowcurrently ericsnowcurrently closed this Apr 24, 2024
This was referenced Apr 24, 2024
gh-117953: Work Relative to Specific Extension Kinds in the Import Machinery #118205
Merged
gh-117953: Other Cleanups in the Extensions Machinery #118206
Merged
ericsnowcurrently added a commit that referenced this pull request Apr 24, 2024
@ericsnowcurrently
gh-117953: Cleanups For fix_up_extension() in import.c (gh-118192)
af3c1d8
These are cleanups I've pulled out of gh-118116.  Mostly, this change moves code around to align with some future changes and to improve clarity a little.  There is one very small change in behavior: we now add the module to the per-interpreter caches after updating the global state, rather than before.
@ericsnowcurrently ericsnowcurrently mentioned this pull request Apr 26, 2024
gh-117953: Let update_global_state_for_extension() Caller Decide If Singlephase or Not #118193
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@encukou encukou Awaiting requested review from encukou
@ZeroIntensity ZeroIntensity Awaiting requested review from ZeroIntensity ZeroIntensity will be requested when the pull request is marked ready for review ZeroIntensity is a code owner
@FFY00 FFY00 Awaiting requested review from FFY00 FFY00 will be requested when the pull request is marked ready for review FFY00 is a code owner
@brettcannon brettcannon Awaiting requested review from brettcannon brettcannon will be requested when the pull request is marked ready for review brettcannon is a code owner
@ncoghlan ncoghlan Awaiting requested review from ncoghlan ncoghlan will be requested when the pull request is marked ready for review ncoghlan is a code owner
@warsaw warsaw Awaiting requested review from warsaw warsaw will be requested when the pull request is marked ready for review warsaw is a code owner
@kumaraditya303 kumaraditya303 Awaiting requested review from kumaraditya303 kumaraditya303 will be requested when the pull request is marked ready for review kumaraditya303 is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@ericsnowcurrently @encukou
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.