GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-127266: avoid data races when updating type slots v2 by nascheme · Pull Request #133177 · python/cpython

Conversation

@nascheme

@nascheme nascheme commented Apr 29, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

This is an updated version of GH-131174, which was reverted. I figured the cleanest thing to do is make a new PR.

This is the same as the previous PR with the following additional change. The update_all_slots() and type_setattro() functions are now more careful when the world is stopped. Instead of doing the MRO lookups while the world is stopped, we do them all first and collect the slot pointers to be updated. Then, we stop the world and do those updates. This makes it much easier to confirm the code running during the stop-the-world is safe and that should avoid the deadlocks.

The test_opcache test has become quite a bit slower. It seems to be due to mutex contention in the __getitem__ and __getattribute__ method assignment tests. I reduced the items count from 1000 to 100 to keep the test from becoming much slower.

  • Issue: Type slots are not thread-safe in free-threaded builds #127266

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
nascheme added 2 commits April 29, 2025 16:44
@nascheme
pythongh-127266: avoid data races when updating type slots
3094372 
In the free-threaded build, avoid data races caused by updating type
slots or type flags after the type was initially created.  For those
(typically rare) cases, use the stop-the-world mechanism.  Remove the
use of atomics when reading or writing type flags.  The use of atomics
is not sufficient to avoid races (since flags are sometimes read without
a lock and without atomics) and are no longer required.
@nascheme
For update_all_slots(), do updates more safely.
f447ce4 
To avoid deadlocks while the world is stopped, we need to avoid calling APIs
like _PyObject_HashFast() and _PyDict_GetItemRef_KnownHash().  Collect the
slot updates to be done and then apply them all at once.  This reduces the
amount of code running in the stop-the-world condition.
@nascheme nascheme added type-bug An unexpected behavior, bug, or error topic-free-threading labels Apr 29, 2025
@bedevere-app bedevere-app Bot mentioned this pull request Apr 29, 2025
Type slots are not thread-safe in free-threaded builds #127266
Closed
@nascheme
Avoid "empty structure" compile error.
d511ca6
@nascheme nascheme added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 30, 2025
@bedevere-bot

bedevere-bot commented Apr 30, 2025

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @nascheme for commit d511ca6 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F133177%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 30, 2025
nascheme added 5 commits April 29, 2025 22:40
@nascheme
Use apply_slot_updates() for type_setattro().
5e38497
@nascheme
Merge 'origin/main' into type-slot-ts-v2
e9516c7
@nascheme
Reduce number of items in test for slot updates.
8c74a0c 
Now that stop-the-world is used to do the slot update, these tests
are a lot slower in the free-threaded build.  Test with fewer items,
which will still hopefully be enough to find bugs in the specializer.
@nascheme
Add TSAN suppression for _Py_slot_tp_getattr_hook.
6cd7644
@nascheme
Queue update of tp_flags as well.
3cb2256 
The clearing of Py_TPFLAGS_HAVE_VECTORCALL must be done when the world
is stopped too.
@nascheme nascheme added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 30, 2025
@bedevere-bot

bedevere-bot commented Apr 30, 2025

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @nascheme for commit 3cb2256 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F133177%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 30, 2025
@nascheme
nascheme marked this pull request as ready for review April 30, 2025 13:29
@nascheme
nascheme requested a review from markshannon as a code owner April 30, 2025 13:29
@bedevere-app bedevere-app Bot added the awaiting core review label Apr 30, 2025
@nascheme
nascheme requested a review from colesbury April 30, 2025 13:33
nascheme added 5 commits April 30, 2025 06:45
@nascheme
Performance, skip stop-the-world when possible.
47e41c9 
Since we stack allocate one chunk, we need to check 'n' to see if there
are actually any updates to make.  It's pretty common that no updates
are actually needed.
@nascheme
Merge 'origin/main' into type-slot-ts-v2
cb848f1
@nascheme
Always clear version after __bases__ update.
9859ebf
@nascheme
Merge 'origin/main' into type-slot-ts-v2
6c74cac
@nascheme
Add test for assigning __bases__.
583c435
colesbury
colesbury reviewed May 1, 2025
View reviewed changes
Comment thread Lib/test/test_opcache.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Tools/tsan/suppressions_free_threading.txt Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Objects/typeobject.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
@nascheme

nascheme commented May 2, 2025

Copy link
Copy Markdown
Member Author

@colesbury Not sure if you would like this approach but it is a variation on your idea to modify the critical section to prevent release of the mutex. I changed it to work with a PyCriticalSection2 as well.

https://github.com/nascheme/cpython/tree/type-slot-ts-release-hack

Not using a critical section for the type dict looks kind of difficult to do. For example, _Py_dict_lookup asserts that the dict is locked. I think we would have to duplicate some dictobject functions to make non-lock-asserting versions.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@colesbury

colesbury commented May 5, 2025

Copy link
Copy Markdown
Contributor

The type-slot-ts-release-hack approach makes sense to me
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@nascheme
Avoid releasing TYPE_LOCK when stopping the world.
c01707e
@bedevere-bot

bedevere-bot commented May 5, 2025

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @nascheme for commit 3f6222b 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F133177%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 5, 2025
@nascheme
Merge 'origin/main' into type-slot-ts-v2
6f218fb
kumaraditya303
kumaraditya303 reviewed May 13, 2025
View reviewed changes
Comment thread Objects/typeobject.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
nascheme added 5 commits May 27, 2025 10:54
@nascheme
Add additional assert.
2bcf7ba
@nascheme
Merge 'origin/main' into type-slot-ts-v2
ddfdbd5
@nascheme
Revert test_opcache item size change.
63b7ae4 
This test gets a bit slower, due to stop-the-world but it is not so
dramatic.
@nascheme
Add comment for new unit test.
1a2fee1
@nascheme
Fix assert for default build.
c1f3ed5
@nascheme nascheme added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 27, 2025
@bedevere-bot

bedevere-bot commented May 27, 2025

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @nascheme for commit c1f3ed5 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F133177%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 27, 2025
colesbury
colesbury reviewed May 27, 2025
View reviewed changes
Comment thread Objects/typeobject.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
@nascheme
Improve function name.
41e54e1 
The apply_slot_updates_world_stopped() name implies that the world
is already stopped, based on convention.  Rename for clarity.
@nascheme
nascheme merged commit fbbbc10 into python:main May 28, 2025
@bedevere-app bedevere-app Bot removed the awaiting merge label May 28, 2025
@kumaraditya303 kumaraditya303 mentioned this pull request May 28, 2025
Deadlock in test_opcache with gh-131174 applied #133130
Closed
@kumaraditya303 kumaraditya303 mentioned this pull request Jul 1, 2025
More gil-disabled type thread safety #114214
Closed
Pranjal095 pushed a commit to Pranjal095/cpython that referenced this pull request Jul 12, 2025
@nascheme @Pranjal095
pythongh-127266: avoid data races when updating type slots (pythongh-…
b0f4eb0 
…133177)

In the free-threaded build, avoid data races caused by updating type
slots or type flags after the type was initially created.  For those
(typically rare) cases, use the stop-the-world mechanism.  Remove the
use of atomics when reading or writing type flags.
taegyunkim pushed a commit to taegyunkim/cpython that referenced this pull request Aug 4, 2025
@nascheme @taegyunkim
pythongh-127266: avoid data races when updating type slots (pythongh-…
895cbcf 
…133177)

In the free-threaded build, avoid data races caused by updating type
slots or type flags after the type was initially created.  For those
(typically rare) cases, use the stop-the-world mechanism.  Remove the
use of atomics when reading or writing type flags.
@mgorny mgorny mentioned this pull request Oct 14, 2025
3.15.0a1 doesn't compile with --disable-gil --with-assertions: Objects/typeobject.c:123:13: error: implicit declaration of function ‘types_world_is_stopped’ [-Wimplicit-function-declaration] #140126
Closed
@colesbury

colesbury commented Nov 5, 2025

Copy link
Copy Markdown
Contributor

Should this be backported to 3.14?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@colesbury colesbury mentioned this pull request Nov 5, 2025
gh-133467: Fix typeobject tp_base race in free threading #140549
Merged
@nascheme

nascheme commented Nov 6, 2025

Copy link
Copy Markdown
Member Author

Should this be backported to 3.14?

It feels a little scary to backport this to a stable release. It's not a simple fix. OTOH, it seems to be working in 3.15 without issue.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@xhochy xhochy mentioned this pull request Nov 28, 2025
3.15.0a2 doesn't compile with CFLAGS=-DNDEBUG and --with-pydebug #142038
Closed
@ikrommyd ikrommyd mentioned this pull request Jan 21, 2026
build: add freethreaded python ci and wheels cms-nanoAOD/correctionlib#284
Merged
3 tasks done
@colesbury colesbury mentioned this pull request Feb 6, 2026
gh-139103: Improve namedtuple scaling in free-threaded build #144332
Merged
gostak-dd pushed a commit to gostak-dd/cpython that referenced this pull request Jun 2, 2026
@nascheme @gostak-dd
pythongh-127266: avoid data races when updating type slots (pythongh-…
3f5494f 
…133177)

In the free-threaded build, avoid data races caused by updating type
slots or type flags after the type was initially created.  For those
(typically rare) cases, use the stop-the-world mechanism.  Remove the
use of atomics when reading or writing type flags.
@nascheme nascheme mentioned this pull request Aug 17, 2026
Memory leak in typeobject.c for slots update #155978
Closed
@scasella scasella mentioned this pull request Sep 25, 2026
Free-threading: PyType_Freeze() can deadlock with concurrent class updates #158194
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@colesbury colesbury colesbury approved these changes
@kumaraditya303 kumaraditya303 kumaraditya303 left review comments
@markshannon markshannon Awaiting requested review from markshannon markshannon is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

topic-free-threading type-bug An unexpected behavior, bug, or error

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@nascheme @bedevere-bot @colesbury @kumaraditya303
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.