GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-134160: Improve multi-phase init note on isolation & subinterpreters by encukou · Pull Request #134775 · python/cpython

Conversation

@encukou

@encukou encukou commented May 27, 2025 •
edited by AA-Turner
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

(This isn't strictly related to #134160, but I think it's more useful to group this PR with the other ones there.)

  • Issue: Docs Examples Focus on Outdated Single-Phase Init Modules and Static Types #134160

📚 Documentation preview 📚: https://cpython-previews--134775.org.readthedocs.build/en/134775/c-api/module.html

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@encukou
pythongh-134160: Improve note on multi-phase init & subinterpreters
7f96223
@encukou
encukou requested a review from ericsnowcurrently as a code owner May 27, 2025 06:42
@bedevere-app bedevere-app Bot added the awaiting core review label May 27, 2025
@bedevere-app bedevere-app Bot mentioned this pull request May 27, 2025
Docs Examples Focus on Outdated Single-Phase Init Modules and Static Types #134160
Open
@bedevere-app bedevere-app Bot added docs Documentation in the Doc dir skip news labels May 27, 2025
@github-project-automation github-project-automation Bot added this to Docs PRs May 27, 2025
@github-project-automation github-project-automation Bot moved this to Todo in Docs PRs May 27, 2025
pitrou
pitrou reviewed May 27, 2025
View reviewed changes
Comment thread Doc/c-api/module.rst Outdated
Comment on lines +309 to +311
- :ref:`raising an error on repeated initialization <isolating-extensions-optout>`, or
- limiting a module to the main interpreter using
:c:data:`Py_mod_multiple_interpreters`.

@pitrou pitrou May 27, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a bit weird to present this as a way of supporting sub-interpreters, no?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@encukou encukou May 27, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. The original wording from PEP 489 would work a bit better:

All modules created using multi-phase initialization are expected to support
:ref:`sub-interpreters <sub-interpreter-support>` and multiple
:c:func:`Py_Initialize`/:c:func:`Py_Finalize` cycles correctly.
Typically, extensions ensure this in one of these ways:

or should we expand the “correctly“ to something like “work as expected or raise exceptions, rather than crash”?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@pitrou pitrou May 27, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would definitely not call this "support". So perhaps something like:

Modules created using multi-phase initialization are expected to handle
sub-interpreters in either of the following two ways:

* either they fully support multiple interpreters by
  :ref:`isolating module instances <isolating-extensions-howto>`;
* or they signal that multiple interpreters are not supported by
  :ref:`raising an error on repeated initialization <isolating-extensions-optout>`
  and/or limiting a module to the main interpreter using
  :c:data:`Py_mod_multiple_interpreters`.

This way, users will not encounter crashes or undefined behavior when
trying to use these modules from multiple interpreters.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@encukou encukou May 27, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess right thing would be to take step back and not make this so much about subinterpreters:

Multi-phase initialization allows creating multiple independent instances of the
module object.
Extensions are expected to handle this in one of the following ways:

* either by :ref:`isolating module instances <isolating-extensions-howto>`;
* and/or by :ref:`raising an error on repeated initialization <isolating-extensions-optout>`;
* and/or limiting a module to the main interpreter using
  :c:data:`Py_mod_multiple_interpreters`.

This way, users will not encounter crashes or undefined behavior when
trying to use these modules from :ref:`sub-interpreters <sub-interpreter-support>`.
The first two ways also prevent crashes when reusing the module after
Python runtime reinitialization (:c:func:`Py_Finalize` and :c:func:`Py_Initialize`)
and reimporting a module after it is removed from :py:attr:`sys.modules`.

Thanks for suggesting the effects on the users! It helps to make this less of a thou shalt commandment and more like something where you can consider the trade-offs.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@pitrou pitrou May 27, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes me wonder: why is there a third way if the second is better (and not more difficult to implement)?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@ericsnowcurrently ericsnowcurrently May 27, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not that it is critical for this PR, but I've been thinking about something related that might at least a little insight: gh-132861.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@encukou encukou May 27, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pitrou: Let's say there's subtle nuance that doesn't really belong on this page of the docs :)
It would be good to explain it in the HOWTO, of course.

AFAIK, the main interpreter is special; sometimes, being loadable once but in any interpreter isn't enough.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@AA-Turner AA-Turner May 28, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The extant prior paragraph covers much of this, so here's an alternative:

When using per-module state, module instances should be :ref:`isolated
<isolating-extensions-howto>` to ensure that they are free of unwanted
side-effects. If the module still uses global state, or it otherwise has not
yet been isolated, either or both of of the following two options should be used:

* :ref:`raising an error on repeated initialization <isolating-extensions-optout>`
* limiting a module to the main interpreter using
  :c:data:`Py_mod_multiple_interpreters`.

This way, users will not encounter crashes or undefined behavior when
trying to use these modules from :ref:`sub-interpreters <sub-interpreter-support>`.
Ensuring a module is properly isolated or raising an explicit error also prevent
crashes when reusing the module after Python runtime reinitialization
(:c:func:`Py_Finalize` and :c:func:`Py_Initialize`) or reimporting a module after
it has been removed from :py:attr:`sys.modules`.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
ericsnowcurrently
ericsnowcurrently reviewed May 27, 2025
View reviewed changes
Comment thread Doc/c-api/module.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/c-api/module.rst Outdated
Typically, extensions ensure this in one of these ways:

- :ref:`isolating module instances <isolating-extensions-howto>`,
- :ref:`raising an error on repeated initialization <isolating-extensions-optout>`, or

@ericsnowcurrently ericsnowcurrently May 27, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, I just noticed that the referenced section does not mention opting out by setting the Py_mod_multiple_interpreters slot to Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED. (I didn't think to update the docs when I added that value.) Fixing that doesn't need to be part of this PR, but it is relevant to this bullet (and the third bullet).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@encukou encukou May 27, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Please read this in a friendly voice; I can't find a way to reword this so it's clear and doesn't have a terrible sarcastic interpretation.)

Yeah. I'm not a good person to write that, as I never fully understood why a slot is better than explicitly raising an exception when you're about to clobber shared state.
It would be good to record that in the HOWTO, could you do it?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ericsnowcurrently reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
AA-Turner
AA-Turner reviewed May 28, 2025
View reviewed changes
Comment thread Doc/c-api/module.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/c-api/module.rst Outdated
All modules created using multi-phase initialization are expected to support
:ref:`sub-interpreters <sub-interpreter-support>`. Making sure multiple modules
are independent is typically enough to achieve this.
:ref:`sub-interpreters <sub-interpreter-support>`.

@AA-Turner AA-Turner May 27, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
:ref:`sub-interpreters <sub-interpreter-support>`.
:ref:`sub-interpreters <sub-interpreter-support>`, or otherwise explicitly
signal a lack of support.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Doc/c-api/module.rst Outdated
Comment on lines +309 to +311
- :ref:`raising an error on repeated initialization <isolating-extensions-optout>`, or
- limiting a module to the main interpreter using
:c:data:`Py_mod_multiple_interpreters`.

@AA-Turner AA-Turner May 28, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The extant prior paragraph covers much of this, so here's an alternative:

When using per-module state, module instances should be :ref:`isolated
<isolating-extensions-howto>` to ensure that they are free of unwanted
side-effects. If the module still uses global state, or it otherwise has not
yet been isolated, either or both of of the following two options should be used:

* :ref:`raising an error on repeated initialization <isolating-extensions-optout>`
* limiting a module to the main interpreter using
  :c:data:`Py_mod_multiple_interpreters`.

This way, users will not encounter crashes or undefined behavior when
trying to use these modules from :ref:`sub-interpreters <sub-interpreter-support>`.
Ensuring a module is properly isolated or raising an explicit error also prevent
crashes when reusing the module after Python runtime reinitialization
(:c:func:`Py_Finalize` and :c:func:`Py_Initialize`) or reimporting a module after
it has been removed from :py:attr:`sys.modules`.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@encukou
Rewrite the whole section
3fc9447
@encukou

encukou commented May 28, 2025

Copy link
Copy Markdown
Member Author

I integrated this into the existing section. It's a bigger change now, but hopefully for the better :)

@ericsnowcurrently, I'm specifically interested in suggestions to make Py_mod_multiple_interpreters look more important.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

AA-Turner
AA-Turner approved these changes May 28, 2025
View reviewed changes

@AA-Turner AA-Turner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wordsmithing:

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Doc/c-api/module.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/c-api/module.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/c-api/module.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/c-api/module.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/c-api/module.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/c-api/module.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels May 28, 2025
@AA-Turner AA-Turner mentioned this pull request May 28, 2025
MAINT: Convert umath_linalg to multi-phase init (PEP 489) numpy/numpy#29030
Merged
encukou
encukou commented May 28, 2025
View reviewed changes
Comment thread Doc/c-api/module.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

encukou and others added 2 commits May 28, 2025 09:46
@encukou @AA-Turner
Apply suggestions from code review
3b0514d
Co-authored-by: Adam Turner <[email protected]>
@encukou @AA-Turner
Apply suggestions from code review
11f8675
Co-authored-by: Adam Turner <[email protected]>
@encukou encukou changed the title gh-134160: Improve note on multi-phase init & subinterpreters gh-134160: Improve multi-phase init note on isolation & subinterpreters May 28, 2025
@AA-Turner

AA-Turner commented May 28, 2025

Copy link
Copy Markdown
Member

@ericsnowcurrently, I'm specifically interested in suggestions to make Py_mod_multiple_interpreters look more important.

Is this a blocker for merging this, or can we follow up later?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@encukou

encukou commented May 28, 2025

Copy link
Copy Markdown
Member Author

@ericsnowcurrently @pitrou, does this look good?
I plan to merge this on Friday if I don't hear back. Shout if you need more review time.

AA-Turner reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@encukou
encukou merged commit eb145fa into python:main May 30, 2025
@bedevere-app bedevere-app Bot removed the awaiting merge label May 30, 2025
@github-project-automation github-project-automation Bot moved this from Todo to Done in Docs PRs May 30, 2025
@encukou
encukou deleted the multiphase-init-doc branch May 30, 2025 14:27
@encukou encukou added the needs backport to 3.14 bugs and security fixes label May 30, 2025
@miss-islington-app

miss-islington-app Bot commented May 30, 2025

Copy link
Copy Markdown

Thanks @encukou for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request May 30, 2025
@encukou @AA-Turner @miss-islington
pythongh-134160: Improve multi-phase init note on isolation & subinte…
f2a4626
…rpreters (pythonGH-134775)

(cherry picked from commit eb145fa)

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: Adam Turner <[email protected]>
@bedevere-app

bedevere-app Bot commented May 30, 2025

Copy link
Copy Markdown

GH-134932 is a backport of this pull request to the 3.14 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label May 30, 2025
AA-Turner added a commit that referenced this pull request May 31, 2025
@miss-islington @encukou @AA-Turner
[3.14] gh-134160: Improve multi-phase init note on isolation & subint…
f734531
…erpreters (GH-134775) (#134932)

gh-134160: Improve multi-phase init note on isolation & subinterpreters (GH-134775)

(cherry picked from commit eb145fa)

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: Adam Turner <[email protected]>
@AA-Turner AA-Turner added the needs backport to 3.13 only security fixes label May 31, 2025
@miss-islington-app

miss-islington-app Bot commented May 31, 2025

Copy link
Copy Markdown

Thanks @encukou for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request May 31, 2025
@encukou @AA-Turner @miss-islington
pythongh-134160: Improve multi-phase init note on isolation & subinte…
478d143
…rpreters (pythonGH-134775)

(cherry picked from commit eb145fa)

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: Adam Turner <[email protected]>
@bedevere-app

bedevere-app Bot commented May 31, 2025

Copy link
Copy Markdown

GH-134983 is a backport of this pull request to the 3.13 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the needs backport to 3.13 only security fixes label May 31, 2025
encukou added a commit that referenced this pull request Jun 4, 2025
@miss-islington @encukou @AA-Turner
[3.13] gh-134160: Improve multi-phase init note on isolation & subint…
d446dbc
…erpreters (GH-134775) (GH-134983)

(cherry picked from commit eb145fa)

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: Adam Turner <[email protected]>
Pranjal095 pushed a commit to Pranjal095/cpython that referenced this pull request Jul 12, 2025
@encukou @AA-Turner @Pranjal095
pythongh-134160: Improve multi-phase init note on isolation & subinte…
7958d42
…rpreters (pythonGH-134775)

Co-authored-by: Adam Turner <[email protected]>
taegyunkim pushed a commit to taegyunkim/cpython that referenced this pull request Aug 4, 2025
@encukou @AA-Turner @taegyunkim
pythongh-134160: Improve multi-phase init note on isolation & subinte…
1736d46
…rpreters (pythonGH-134775)

Co-authored-by: Adam Turner <[email protected]>
gostak-dd pushed a commit to gostak-dd/cpython that referenced this pull request Jun 2, 2026
@encukou @AA-Turner @gostak-dd
pythongh-134160: Improve multi-phase init note on isolation & subinte…
0fd850e
…rpreters (pythonGH-134775)

Co-authored-by: Adam Turner <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ericsnowcurrently ericsnowcurrently ericsnowcurrently left review comments
@pitrou pitrou pitrou left review comments
@AA-Turner AA-Turner AA-Turner approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

docs Documentation in the Doc dir skip news

Projects

Docs PRs
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@encukou @AA-Turner @ericsnowcurrently @pitrou
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.