GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — gh-111506: Add _Py_OPAQUE_PYOBJECT to hide PyObject layout & related API by encukou · Pull Request #136505 · python/cpython

Conversation

encukou commented Jul 10, 2025 •

edited by bedevere-app Bot

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Member

Make it so that defining _Py_OPAQUE_PYOBJECT before including <Python.h> will make the PyObject struct opaque, and hide all limited-API structs & functions that need access to it.

Also, allow Py_LIMITED_API with Py_GIL_DISABLED if _Py_OPAQUE_PYOBJECT is defined.

Opaque PyObject will be needed for most alternatives we have of defining a Stable ABI for free-threaded builds. See Discourse thread for discussion. (We may want to either expose this to users as Py_OPAQUE_PYOBJECT, or define it implicitly for limited-API free-threaded builds.)
And if I'm wrong -- this is easy to remove, like like Py_GIL_DISABLED.

As I mentioned, I'd like to put this in main to avoid rebasing a fork and to make it easier to build on top.

Issue: --disable-gil builds are not compatible with the limited API #111506

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

encukou added 2 commits

July 10, 2025 13:12


          Allow Py_LIMITED_API for (Py_GIL_DISABLED && _Py_OPAQUE_PYOBJECT)


          Hide some API if _Py_OPAQUE_PYOBJECT is defined

7ff56f4

API that's removed when _Py_OPAQUE_PYOBJECT is defined:

    - PyObject_HEAD
    - _PyObject_EXTRA_INIT
    - PyObject_HEAD_INIT
    - PyObject_VAR_HEAD
    - struct _object (i.e. PyObject) (opaque)
    - struct PyVarObject (opaque)
    - Py_SIZE
    - Py_SET_TYPE
    - Py_SET_SIZE
    - PyModuleDef_Base (opaque)
    - PyModuleDef_HEAD_INIT
    - PyModuleDef (opaque)
    - _Py_IsImmortal
    - _Py_IsStaticImmortal

Note that the `_Py_IsImmortal` removal means _Py_OPAQUE_PYOBJECT
only works with limited API 3.14+ now.

encukou requested a review from ericsnowcurrently as a code owner

July 10, 2025 11:52

bedevere-app Bot added the awaiting core review label

Jul 10, 2025

bedevere-app Bot mentioned this pull request

Jul 10, 2025

--disable-gil builds are not compatible with the limited API #111506

Closed

encukou added the skip news label

Jul 10, 2025


          Add test for _Py_OPAQUE_PYOBJECT

33d9ae7

encukou commented Jul 11, 2025

Copy link

Copy Markdown

Member Author

We can also test this (with a workaround/hack in the test for the missing part).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

encukou mentioned this pull request

Jul 11, 2025

Make PyObject and other runtime structures optionally fully opaque capi-workgroup/api-revolution#6

Closed

encukou added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label

Jul 11, 2025

bedevere-bot commented Jul 11, 2025

Copy link

Copy Markdown

🤖 New build scheduled with the buildbot fleet by @encukou for commit 33d9ae7 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F136505%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label

Jul 11, 2025

encukou commented Jul 11, 2025

Copy link

Copy Markdown

Member Author

The tests aren't that straightforward, but the workaround can still be removed if we either scrap the “opaque PyObject” idea or implement PEP 793.
@vstinner, are you OK with having this workaround in tests in main, it order to test that Python.h compiles with opaque PyObject (and except for PyModuleDef, it can produce working extensions)?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner reviewed

Jul 11, 2025

View reviewed changes

vstinner left a comment

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the overall approach of the _Py_OPAQUE_PYOBJECT macro. I'm fine with testing it in test_cext.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

encukou reacted with heart emoji

All reactions

❤️ 1 reaction

Comment thread

Lib/test/test_cext/__init__.py Outdated

		# Test with _Py_OPAQUE_PYOBJECT
		self.check_build('_test_limited_opaque_cext', limited=True, opaque=True)

		def check_build(self, extension_name, std=None, limited=False, opaque=False):

vstinner Jul 11, 2025

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest to rename "opaque" to "opaque_pyobject" (also in _check_build).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

Lib/test/test_cext/setup.py Outdated

		std = os.environ.get("CPYTHON_TEST_STD", "")
		module_name = os.environ["CPYTHON_TEST_EXT_NAME"]
		limited = bool(os.environ.get("CPYTHON_TEST_LIMITED", ""))
		opaque = bool(os.environ.get("CPYTHON_TEST_OPAQUE_PYOBJECT", ""))

vstinner Jul 11, 2025

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest to rename opaque to opaque_pyobject.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

Lib/test/test_cext/setup.py Outdated


		# Define _Py_OPAQUE_PYOBJECT macro
		if opaque:
		version = sys.hexversion

vstinner Jul 11, 2025

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

version is not used, it can be removed.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

vstinner and others added 2 commits

July 11, 2025 21:53


          Apply suggestions from code review

781e2ab


          Merge in the main branch to solve conflict

08d963d

encukou merged commit c7d24b8 into python:main

Jul 12, 2025

bedevere-app Bot removed the awaiting core review label

Jul 12, 2025

encukou deleted the opaque-pyobject-priv branch

July 12, 2025 07:55

Pranjal095 pushed a commit to Pranjal095/cpython that referenced this pull request

Jul 12, 2025


          pythongh-111506: Add _Py_OPAQUE_PYOBJECT to hide PyObject layout & re…

0292e9d

…lated API (pythonGH-136505)

Allow Py_LIMITED_API for (Py_GIL_DISABLED && _Py_OPAQUE_PYOBJECT)


API that's removed when _Py_OPAQUE_PYOBJECT is defined:

    - PyObject_HEAD
    - _PyObject_EXTRA_INIT
    - PyObject_HEAD_INIT
    - PyObject_VAR_HEAD
    - struct _object (i.e. PyObject) (opaque)
    - struct PyVarObject (opaque)
    - Py_SIZE
    - Py_SET_TYPE
    - Py_SET_SIZE
    - PyModuleDef_Base (opaque)
    - PyModuleDef_HEAD_INIT
    - PyModuleDef (opaque)
    - _Py_IsImmortal
    - _Py_IsStaticImmortal

Note that the `_Py_IsImmortal` removal (and a few other issues)
 means _Py_OPAQUE_PYOBJECT only works with limited
API 3.14+ now.


Co-authored-by: Victor Stinner <[email protected]>

picnixz pushed a commit to picnixz/cpython that referenced this pull request

Jul 13, 2025


          pythongh-111506: Add _Py_OPAQUE_PYOBJECT to hide PyObject layout & re…

3ac5d3f

…lated API (pythonGH-136505)

Allow Py_LIMITED_API for (Py_GIL_DISABLED && _Py_OPAQUE_PYOBJECT)


API that's removed when _Py_OPAQUE_PYOBJECT is defined:

    - PyObject_HEAD
    - _PyObject_EXTRA_INIT
    - PyObject_HEAD_INIT
    - PyObject_VAR_HEAD
    - struct _object (i.e. PyObject) (opaque)
    - struct PyVarObject (opaque)
    - Py_SIZE
    - Py_SET_TYPE
    - Py_SET_SIZE
    - PyModuleDef_Base (opaque)
    - PyModuleDef_HEAD_INIT
    - PyModuleDef (opaque)
    - _Py_IsImmortal
    - _Py_IsStaticImmortal

Note that the `_Py_IsImmortal` removal (and a few other issues)
 means _Py_OPAQUE_PYOBJECT only works with limited
API 3.14+ now.


Co-authored-by: Victor Stinner <[email protected]>

taegyunkim pushed a commit to taegyunkim/cpython that referenced this pull request

Aug 4, 2025


          pythongh-111506: Add _Py_OPAQUE_PYOBJECT to hide PyObject layout & re…

7875fbe

…lated API (pythonGH-136505)

Allow Py_LIMITED_API for (Py_GIL_DISABLED && _Py_OPAQUE_PYOBJECT)


API that's removed when _Py_OPAQUE_PYOBJECT is defined:

    - PyObject_HEAD
    - _PyObject_EXTRA_INIT
    - PyObject_HEAD_INIT
    - PyObject_VAR_HEAD
    - struct _object (i.e. PyObject) (opaque)
    - struct PyVarObject (opaque)
    - Py_SIZE
    - Py_SET_TYPE
    - Py_SET_SIZE
    - PyModuleDef_Base (opaque)
    - PyModuleDef_HEAD_INIT
    - PyModuleDef (opaque)
    - _Py_IsImmortal
    - _Py_IsStaticImmortal

Note that the `_Py_IsImmortal` removal (and a few other issues)
 means _Py_OPAQUE_PYOBJECT only works with limited
API 3.14+ now.


Co-authored-by: Victor Stinner <[email protected]>

Agent-Hellboy pushed a commit to Agent-Hellboy/cpython that referenced this pull request

Aug 19, 2025


          pythongh-111506: Add _Py_OPAQUE_PYOBJECT to hide PyObject layout & re…

cae2d31

…lated API (pythonGH-136505)

Allow Py_LIMITED_API for (Py_GIL_DISABLED && _Py_OPAQUE_PYOBJECT)


API that's removed when _Py_OPAQUE_PYOBJECT is defined:

    - PyObject_HEAD
    - _PyObject_EXTRA_INIT
    - PyObject_HEAD_INIT
    - PyObject_VAR_HEAD
    - struct _object (i.e. PyObject) (opaque)
    - struct PyVarObject (opaque)
    - Py_SIZE
    - Py_SET_TYPE
    - Py_SET_SIZE
    - PyModuleDef_Base (opaque)
    - PyModuleDef_HEAD_INIT
    - PyModuleDef (opaque)
    - _Py_IsImmortal
    - _Py_IsStaticImmortal

Note that the `_Py_IsImmortal` removal (and a few other issues)
 means _Py_OPAQUE_PYOBJECT only works with limited
API 3.14+ now.


Co-authored-by: Victor Stinner <[email protected]>

encukou mentioned this pull request

Mar 30, 2026

Implement PEP 803 -- abi3t #146636

Closed

8 of 9 tasks

Reviewers

vstinner

vstinner left review comments

ericsnowcurrently

Awaiting requested review from ericsnowcurrently

ericsnowcurrently is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

encukou commented Jul 10, 2025 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

encukou commented Jul 11, 2025

Uh oh!

bedevere-bot commented Jul 11, 2025

Uh oh!

encukou commented Jul 11, 2025

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

vstinner Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

vstinner Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

vstinner Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

encukou commented Jul 10, 2025 •

edited by bedevere-app Bot

Loading

Uh oh!

There was an error while loading. Please reload this page.