GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-140550: Update xxlimited with 3.15 limited API & PEP 697 by encukou · Pull Request #142827 · python/cpython

Conversation

@encukou

@encukou encukou commented Dec 16, 2025 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member
  • Copy the existing xxlimited to xxlimited_3_13: this module serves as a rudimentary check that we don't break older Limited API. This is similar to xxlimited_35. (Nowadays we know that it's good to have a separator between the major/minor versions, but renaming to xxlimited_3_5 isn't worth the churn.)
  • Reorganize test_xxlimited to make similar copies easier in the future
  • Switch xxlimited to Limited API 3.15, PyModExport (PEP-793), standalone instance struct with extended basicsize (PEP-697)
  • treat PyObject as opaque, as proposed in PEP-803 and PEP-809, using a private knob (_Py_OPAQUE_PYOBJECT) for now to ensure we don't rely on PyObject layout details accidentally. (I'll replace/remove the private API in 3.15 beta at latest.)
  • Issue: Implement PEP 793 – PyModExport #140550

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
encukou added 6 commits December 15, 2025 16:40
@encukou
Copy xxlimited to xxlimited_3_13
8147b92
@encukou
Run tests for all the xxlimited* modules
65f3fd0
@encukou
Adjust module names in the 3_13 module
aa43538
@encukou
Update xxlimited to 3.15 limited API (with opaque PyObject)
323695f
This adds the experimental free-threading support.
@encukou
Comments & cleanups
597652b
@encukou
Add Py_mod_abi
22464eb
@encukou
encukou requested review from gpshead and vstinner December 16, 2025 14:21
@encukou
encukou requested review from a team, AA-Turner, corona10, emmatyping, ericsnowcurrently and erlend-aasland as code owners December 16, 2025 14:21
@bedevere-app bedevere-app Bot added the awaiting core review label Dec 16, 2025
@bedevere-app bedevere-app Bot mentioned this pull request Dec 16, 2025
Implement PEP 793 – PyModExport #140550
Open
19 of 25 tasks
@encukou encukou mentioned this pull request Dec 16, 2025
gh-124153: Add Py_tp_token support to xxlimited module #141644
Draft
encukou added 2 commits December 16, 2025 15:33
@encukou
Remove unused import
cc5e8f3
@encukou
Don't flag static PyABIInfo variables
945b450
@encukou encukou added the skip news label Dec 16, 2025
@encukou
encukou requested a review from ZeroIntensity December 16, 2025 14:44
@erlend-aasland
erlend-aasland removed their request for review December 16, 2025 14:45
ZeroIntensity
ZeroIntensity reviewed Dec 16, 2025
View reviewed changes
Comment thread Lib/test/test_xxlimited.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Modules/xxlimited.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Modules/xxlimited.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Modules/xxlimited.c Outdated
Comment on lines +597 to +599
/* TODO: This is not quite true yet: there is a race in Xxo_setattro
* for example.
*/

@ZeroIntensity ZeroIntensity Dec 16, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't it be better to mark this as Py_MOD_GIL_USED in the meantime, or are you trying to avoid the warning?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@encukou encukou Dec 16, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIK, the situation is unchanged since Py_MOD_GIL_NOT_USED was added everywhere in #116882.

I think the intention is/was to make everything thread-safe before some stage of free-threading implementation?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@ZeroIntensity ZeroIntensity Dec 16, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we added that everywhere under the assumption that everything was already thread-safe, and anything that should be reported to us as a bug. I'm not sure that applies with our own test suite.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@encukou encukou Dec 16, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not really a test suite, it supposed to be an example/template showing how to use the Limited API.
The limited API was never compatible with free-threading (yet), so I don't think there's a bug?

There's an earlier PR, #110764, which did this (Py_NOGIL was later renamed to Py_GIL_DISABLED):

+#ifndef Py_NOGIL
 #define Py_LIMITED_API 0x030d0000
+#endif

... but that seems to defeat the point of this module :/

Maybe we should have not built these modules at all in free-threading builds, and skip the relevant tests?
Doing that now seems like wasted effort though, when mutexes/critical sections are a genuine TODO item for the limited API.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@ZeroIntensity ZeroIntensity Dec 16, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's a template, I'm a little concerned that people will see this and think "oh, this module is thread-safe", when it's really not. The comment helps, but I don't know what percentage of people will ignore it.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

encukou reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

@encukou encukou Apr 8, 2026

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before PyCriticalSection is added to the Stable ABI, we can use PyThread_acquire_lock for per-instance locking. (It's also a nice example of a non-PyObject resource that needs freeing.)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@encukou

encukou commented Dec 16, 2025

Copy link
Copy Markdown
Member Author

I'll mark this draft -- this PR makes some shortcomings quite visible, and I guess it's better to address them before suggesting the new API :)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@encukou
encukou marked this pull request as draft December 16, 2025 17:58
@bedevere-app bedevere-app Bot removed the awaiting core review label Dec 16, 2025
encukou added 3 commits December 17, 2025 12:02
@encukou
Refcounting fix
3b3ae7c
@encukou
Style
ff0f300
@encukou
Expand TODO
29f1690
vstinner
vstinner reviewed Dec 17, 2025
View reviewed changes
Comment thread Modules/xxlimited.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Modules/xxlimited.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Modules/xxlimited.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

encukou added 2 commits December 17, 2025 14:56
@encukou
Remove comment that might be misleading
d9728de
@encukou
PyUnicode_EqualToUTF8 can't error (and doesn't type-check)
0bc0198
encukou
encukou commented Jan 6, 2026
View reviewed changes
Comment thread Modules/xxlimited.c
// traverse: Visit all references from an object, including its type
static int
Xxo_traverse(PyObject *op, visitproc visit, void *arg)
Xxo_traverse(PyObject *self, visitproc visit, void *arg)

@encukou encukou Jan 6, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@colesbury, regarding your post on Discourse:

I think that’s just PyType_GetBaseByToken and PyType_GetModuleByToken. It seems pretty rare to need to call them from a tp_traverse handler. If really necessary, I think you can work around that in extensions by calling other APIs.

How would you implement the traverse function here?

Xxo is a subclassable type, and it should work on ABIs with different sizeof(PyObject) (i.e. free-threaded & GIL-ful).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@colesbury colesbury Jan 6, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By workaround I mean something like:

static Xxo_Type_spec = {
   ...
   .flags = Py_TPFLAGS_IMMUTABLETYPE | ...
};
...
static PyTypeObject *Xxo_Type;
...
Xxo_Type = state->Xxo_Type = (PyTypeObject*)PyType_FromModuleAndSpec(...);
static XxoObject_Data *
Xxo_get_data(PyObject *self)
{
    return PyObject_GetTypeData(self, Xxo_Type);
}

We could also avoid all this if we had a PyObject_GetTypeDataByToken().

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@encukou encukou Jan 12, 2026

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

static PyTypeObject *Xxo_Type;
...
Xxo_Type = state->Xxo_Type = (PyTypeObject*)PyType_FromModuleAndSpec(...);

That'll overwrite the pointer in a subinterpreter, or if you otherwise load several copies of the module. It's not something we can recommend.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@encukou encukou mentioned this pull request Jan 13, 2026
Fix and test PyABIInfo_VAR #142225
Closed
encukou added 4 commits April 8, 2026 15:06
@encukou
Merge in the main branch
0ae6c58
@encukou
Use _DuringGC API in traversal functions
619eac7
@encukou
Don't clobber current exception in tp_dealloc
d680b17
@encukou
Avoid data race in Xxo_setattro
38e5fea
@encukou
encukou marked this pull request as ready for review April 10, 2026 12:37
@bedevere-app bedevere-app Bot added the awaiting core review label Apr 10, 2026
encukou added 4 commits May 5, 2026 17:37
@encukou
Merge in the main branch
e6f10df
@encukou
Update xxlimited to PyCriticalSection
e2903da
@encukou
Update comment
330bb76
@encukou
Update C globals checker
62f8b5b
vstinner
vstinner reviewed May 6, 2026
View reviewed changes
Comment thread Modules/xxlimited.c
}
if (PyModule_AddType(m, (PyTypeObject*)Str_Type) < 0) {
if (PyModule_AddType(m, Str_Type) < 0) {
return -1;

@vstinner vstinner May 6, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we have to call Py_DECREF(Str_Type) here (on error)?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

encukou reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

@encukou encukou May 8, 2026

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the catch!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Modules/xxlimited.c Outdated
PySlot_FUNC(Py_mod_exec, xx_modexec),

/* Module state and associated functions */
PySlot_SIZE(Py_mod_state_size, (void*)sizeof(xx_state)),

@vstinner vstinner May 6, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that the (void*) cast is needed here:

Suggested change
PySlot_SIZE(Py_mod_state_size, (void*)sizeof(xx_state)),
PySlot_SIZE(Py_mod_state_size, sizeof(xx_state)),

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Modules/xxlimited.c
static PySlot xx_slots[] = {
/* Basic metadata */
PySlot_STATIC_DATA(Py_mod_name, "xxlimited"),
PySlot_STATIC_DATA(Py_mod_doc, (void*)module_doc),

@vstinner vstinner May 6, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the (void*) cast useful here?

Suggested change
PySlot_STATIC_DATA(Py_mod_doc, (void*)module_doc),
PySlot_STATIC_DATA(Py_mod_doc, module_doc),

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
encukou added 3 commits May 7, 2026 13:15
@encukou
Address review comments
f08fac8
@encukou
Merge in the main branch
f3bb5a1
@encukou
Merge in the main branch
5b0cd4f
@encukou encukou added the needs backport to 3.15 bugs and security fixes label May 8, 2026
@encukou
Avoid warning
c52a25a
vstinner
vstinner approved these changes May 13, 2026
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

encukou reacted with heart emoji
All reactions
  • ❤️ 1 reaction
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels May 13, 2026
@encukou
encukou merged commit fa81cd9 into python:main May 13, 2026
66 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@encukou
encukou deleted the xxlimited-3.15 branch May 13, 2026 16:35
@bedevere-app bedevere-app Bot removed the awaiting merge label May 13, 2026
@miss-islington-app

miss-islington-app Bot commented May 13, 2026

Copy link
Copy Markdown

Thanks @encukou for the PR 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app

bedevere-app Bot commented May 13, 2026

Copy link
Copy Markdown

GH-149785 is a backport of this pull request to the 3.15 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the needs backport to 3.15 bugs and security fixes label May 13, 2026
encukou added a commit that referenced this pull request May 13, 2026
@miss-islington @encukou
[3.15] gh-140550: Update xxlimited with 3.15 limited API (GH-142827) (G…
b650305
…H-149785)

(cherry picked from commit fa81cd9)

Co-authored-by: Petr Viktorin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@colesbury colesbury colesbury left review comments
@ZeroIntensity ZeroIntensity ZeroIntensity left review comments
@vstinner vstinner vstinner approved these changes
@gpshead gpshead Awaiting requested review from gpshead
@ericsnowcurrently ericsnowcurrently Awaiting requested review from ericsnowcurrently ericsnowcurrently is a code owner
@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner is a code owner
@emmatyping emmatyping Awaiting requested review from emmatyping emmatyping is a code owner
@corona10 corona10 Awaiting requested review from corona10 corona10 is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@encukou @vstinner @colesbury @ZeroIntensity
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.