GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-100227: Make the Global PyModuleDef Cache Safe for Isolated Interpreters by ericsnowcurrently · Pull Request #103084 · python/cpython

Conversation

@ericsnowcurrently

@ericsnowcurrently ericsnowcurrently commented Mar 28, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Sharing mutable (or non-immortal) objects between interpreters is generally not safe. We can work around that but not easily.
There are two restrictions that are critical for objects that break interpreter isolation.

The first is that the object's state be guarded by a global lock. For now the GIL meets this requirement, but a granular global lock is needed once we have a per-interpreter GIL.

The second restriction is that the object (and, for a container, its items) be deallocated/resized only when the interpreter in which it was allocated is the current one. This is because every interpreter has (or will have, see gh-101660) its own object allocator. Deallocating an object with a different allocator can cause crashes.

The dict for the cache of module defs is completely internal, which simplifies what we have to do to meet those requirements. To do so, we do the following:

  • add a mechanism for re-using a temporary thread state tied to the main interpreter in an arbitrary thread
    • add _PyRuntime.imports.extensions.main_tstate
    • add _PyThreadState_InitDetached() and _PyThreadState_ClearDetached() (pystate.c)
    • add _PyThreadState_BindDetached() and _PyThreadState_UnbindDetached() (pystate.c)
  • make sure the cache dict (_PyRuntime.imports.extensions.dict) and its items are all owned by the main interpreter)
  • add a placeholder using for a granular global lock

Note that the cache is only used for legacy extension modules and not for multi-phase init modules.

(This PR was derived from gh-102925, which I recently reverted. Also, there was a similar, more complex PR that I've closed: gh-102938.)

  • Issue: Improve Interpreter Isolation #100227

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
ericsnowcurrently added 9 commits March 27, 2023 17:14
@ericsnowcurrently
Make _PyRuntime.imports.extensions a struct.
ff4eb36
@ericsnowcurrently
Factor out add_threadstate().
e558d76
@ericsnowcurrently
Add _PyThreadState_InitDetached().
acf5269
@ericsnowcurrently
Add _PyThreadState_ClearDetached().
181106e
@ericsnowcurrently
Add extensions_lock_acquire() and extensions_lock_release().
b8ffbba
@ericsnowcurrently
Add _PyRuntime.imports.extensions.main_tstate.
a86aea5
@ericsnowcurrently
Add _PyThreadState_BindDetached() and _PyThreadState_UnbindDetached().
d79afe5
@ericsnowcurrently
Share the extensions dict safely.
5182186
@ericsnowcurrently
Intern the name and filename in the main interpreter.
f6db0a2
@ericsnowcurrently ericsnowcurrently added the skip news label Mar 28, 2023
@ericsnowcurrently
ericsnowcurrently requested review from brettcannon, encukou, ncoghlan and warsaw as code owners March 28, 2023 14:56
@bedevere-bot bedevere-bot mentioned this pull request Mar 28, 2023
Improve Interpreter Isolation #100227
Closed
@bedevere-bot bedevere-bot added the awaiting core review label Mar 28, 2023
@ericsnowcurrently ericsnowcurrently mentioned this pull request Mar 28, 2023
gh-100227: Make the Global PyModuleDef Cache Safe for Isolated Interpreters #102938
Closed
@brettcannon
brettcannon removed their request for review March 28, 2023 19:41
ericsnowcurrently added 2 commits March 28, 2023 16:09
@ericsnowcurrently
Fix the asserts in _PyThreadState_BindDetached() and _PyThreadState_U…
dac9c23
…nbindDetached().
@ericsnowcurrently
Swap back when done using main_tstate.
8d442e2
@ericsnowcurrently
ericsnowcurrently removed request for encukou, ncoghlan and warsaw March 28, 2023 22:10
ericsnowcurrently added 2 commits March 28, 2023 16:35
@ericsnowcurrently
Disable a prematurely-restrictive assert.
ee70a96
@ericsnowcurrently
Fix a copy-and-paste bug.
e9c37fe
@ericsnowcurrently ericsnowcurrently added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Mar 29, 2023
@bedevere-bot

bedevere-bot commented Mar 29, 2023

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @ericsnowcurrently for commit e9c37fe 🤖

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Mar 29, 2023
@ericsnowcurrently
Decref name & filename if copied.
8e005bf
@ericsnowcurrently
ericsnowcurrently merged commit dcd6f22 into python:main Mar 29, 2023
@bedevere-bot bedevere-bot removed the awaiting core review label Mar 29, 2023
@ericsnowcurrently
ericsnowcurrently deleted the extensions-dict-owned-by-main-interpreter-2 branch March 29, 2023 23:15
@ericsnowcurrently ericsnowcurrently mentioned this pull request Mar 31, 2023
gh-100227: Fix Cleanup of the Extensions Cache #103150
Merged
ericsnowcurrently added a commit that referenced this pull request Mar 31, 2023
@ericsnowcurrently
gh-100227: Fix Cleanup of the Extensions Cache (gh-103150)
dde0284
Decref the key in the right interpreter in _extensions_cache_set().

This is a follow-up to gh-103084. I found the bug while working on gh-101660.
warsaw pushed a commit to warsaw/cpython that referenced this pull request Apr 11, 2023
@ericsnowcurrently @warsaw
pythongh-100227: Make the Global PyModuleDef Cache Safe for Isolated …
9f95f32
…Interpreters (pythongh-103084)

Sharing mutable (or non-immortal) objects between interpreters is generally not safe.  We can work around that but not easily. 
 There are two restrictions that are critical for objects that break interpreter isolation.

The first is that the object's state be guarded by a global lock.  For now the GIL meets this requirement, but a granular global lock is needed once we have a per-interpreter GIL.

The second restriction is that the object (and, for a container, its items) be deallocated/resized only when the interpreter in which it was allocated is the current one.  This is because every interpreter has (or will have, see pythongh-101660) its own object allocator.  Deallocating an object with a different allocator can cause crashes.

The dict for the cache of module defs is completely internal, which simplifies what we have to do to meet those requirements.  To do so, we do the following:

* add a mechanism for re-using a temporary thread state tied to the main interpreter in an arbitrary thread
   * add _PyRuntime.imports.extensions.main_tstate` 
   * add _PyThreadState_InitDetached() and _PyThreadState_ClearDetached() (pystate.c)
   * add _PyThreadState_BindDetached() and _PyThreadState_UnbindDetached() (pystate.c)
* make sure the cache dict (_PyRuntime.imports.extensions.dict) and its items are all owned by the main interpreter)
* add a placeholder using for a granular global lock

Note that the cache is only used for legacy extension modules and not for multi-phase init modules.

python#100227
warsaw pushed a commit to warsaw/cpython that referenced this pull request Apr 11, 2023
@ericsnowcurrently @warsaw
pythongh-100227: Fix Cleanup of the Extensions Cache (pythongh-103150)
7c7f00c
Decref the key in the right interpreter in _extensions_cache_set().

This is a follow-up to pythongh-103084. I found the bug while working on pythongh-101660.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@ericsnowcurrently @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.