GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-112069: Add _PySet_NextEntryRef to be thread-safe. by corona10 · Pull Request #117990 · python/cpython

Conversation

@corona10

@corona10 corona10 commented Apr 17, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member
  • Issue: Make set thread-safe in --disable-gil builds #112069
  • Add _PySet_NextEntryRef to be thread-safe.
  • Use FrozenSet if possible to use _PySet_NextEntry without locking.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@corona10
pythongh-112069: Make _PySet_NextEntry to be thread-safe.
f1776aa
@corona10 corona10 added the topic-free-threading label Apr 17, 2024
@corona10
corona10 requested a review from colesbury April 17, 2024 14:29
@corona10
corona10 requested review from ericsnowcurrently, iritkatriel, markshannon, methane and rhettinger as code owners April 17, 2024 14:29
@bedevere-app bedevere-app Bot added the awaiting core review label Apr 17, 2024
@corona10 corona10 added skip news and removed awaiting core review labels Apr 17, 2024
@bedevere-app bedevere-app Bot mentioned this pull request Apr 17, 2024
Make set thread-safe in --disable-gil builds #112069
Closed
3 tasks done
@corona10
nit
4170a35
@corona10

corona10 commented Apr 17, 2024

Copy link
Copy Markdown
Member Author

Tests / Thread sanitizer (free-threading) / Thread sanitizer (pull_request)

OKay, it's too slow.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@corona10
corona10 marked this pull request as draft April 17, 2024 15:17
colesbury
colesbury reviewed Apr 17, 2024
View reviewed changes

@colesbury colesbury left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what the right approach here is, but as written _PySet_NextEntry still returns a borrowed reference, so the cases that were not thread-safe are probably still not thread-safe.

I don't think it makes sense to try to make individual calls to _PySet_NextEntry thread-safe. The whole loop of while(_PySet_NextEntry(...)) is generally what needs to be thread-safe. For example, the comment in set_next says:

cpython/Objects/setobject.c

Lines 477 to 478 in fccedbd

* CAUTION: In general, it isn't safe to use set_next in a loop that
* mutates the table.

I think we should add locking in the use sites of _PySet_NextEntry as necessary. Most look like they're already okay to me: either they already do locking (listobject.c and dictobject.c) or operate on private or immutable data (pylifecycle.c, codeobject.c).

The two cases that concern me and may need locking are:

  • _pickle.c
  • marshal.c

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
corona10 reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@corona10
WIP
60e8d53
@corona10 corona10 changed the title gh-112069: Make _PySet_NextEntry to be thread-safe. [WIP] gh-112069: Make _PySet_NextEntry to be thread-safe. Apr 17, 2024
corona10 added 5 commits April 18, 2024 01:46
@corona10
nit
c8b531d
@corona10
Update
c81dd70
@corona10
nit
0e32d1a
@corona10
Remove uncessary locking
fbb9964
@corona10
fix
5d57a56
@corona10 corona10 changed the title [WIP] gh-112069: Make _PySet_NextEntry to be thread-safe. gh-112069: Make _PySet_NextEntry to be thread-safe. Apr 17, 2024
@corona10
corona10 marked this pull request as ready for review April 17, 2024 17:55
@bedevere-app bedevere-app Bot added the awaiting core review label Apr 17, 2024
@corona10
corona10 requested a review from colesbury April 17, 2024 17:55
@corona10 corona10 added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Apr 17, 2024
@bedevere-bot

bedevere-bot commented Apr 17, 2024

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @corona10 for commit 5d57a56 🤖

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Apr 17, 2024
@corona10

corona10 commented Apr 17, 2024

Copy link
Copy Markdown
Member Author

buildbot/AMD64 RHEL7 Refleaks PR: Unrelated

In file included from ./Include/internal/mimalloc/mimalloc/types.h:24:0,
                 from ./Include/internal/pycore_mimalloc.h:40,
                 from ./Include/internal/pycore_interp.h:30,
                 from ./Include/internal/pycore_runtime.h:17,
                 from ./Include/internal/pycore_pystate.h:12,
                 from Parser/myreadline.c:14:
./Include/internal/mimalloc/mimalloc/atomic.h:39:23: fatal error: stdatomic.h: No such file or directory
 #include <stdatomic.h>
                       ^
compilation terminated.
In file included from ./Include/internal/mimalloc/mimalloc/types.h:24:0,
                 from ./Include/internal/pycore_mimalloc.h:40,
                 from ./Include/internal/pycore_interp.h:30,
                 from ./Include/internal/pycore_runtime.h:17,
                 from ./Include/internal/pycore_long.h:13,
                 from Objects/boolobject.c:4:
./Include/internal/mimalloc/mimalloc/atomic.h:39:23: fatal error: stdatomic.h: No such file or directory
 #include <stdatomic.h>
                       ^
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
colesbury
colesbury reviewed Apr 17, 2024
View reviewed changes

@colesbury colesbury left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • We should not change the reference count behavior of _PySet_NextEntry as it's exposed publicly and used outside of CPython (like in Cython). If needed, add a new function _PySet_NextEntryRef.
  • I don't think _PyFrozenSet_NextEntry is needed. _PySet_NextEntry (or _PySet_NextEntryRef) looks sufficient.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
corona10 reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@corona10
Address code review
ae8f217
@corona10 corona10 changed the title gh-112069: Make _PySet_NextEntry to be thread-safe. [WIP] gh-112069: Make _PySet_NextEntry to be thread-safe. Apr 18, 2024
corona10 added 2 commits April 18, 2024 10:18
@corona10
Fix refleaks
3bd4ede
@corona10
nit
99fb7e6
@corona10 corona10 changed the title [WIP] gh-112069: Make _PySet_NextEntry to be thread-safe. gh-112069: Make _PySet_NextEntry to be thread-safe. Apr 18, 2024
@corona10 corona10 added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Apr 18, 2024
@bedevere-bot

bedevere-bot commented Apr 18, 2024

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @corona10 for commit 99fb7e6 🤖

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Apr 18, 2024
@corona10 corona10 changed the title gh-112069: Make _PySet_NextEntry to be thread-safe. [WIP] gh-112069: Make _PySet_NextEntry to be thread-safe. Apr 18, 2024
@corona10 corona10 added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Apr 18, 2024
@bedevere-bot

bedevere-bot commented Apr 18, 2024

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @corona10 for commit 53f949a 🤖

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Apr 18, 2024
@corona10
corona10 force-pushed the gh-112069-PySet_NextEntry branch from 53f949a to 99fb7e6 Compare April 18, 2024 03:50
@corona10 corona10 added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Apr 18, 2024
@bedevere-bot

bedevere-bot commented Apr 18, 2024

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @corona10 for commit 99fb7e6 🤖

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Apr 18, 2024
@corona10

corona10 commented Apr 18, 2024

Copy link
Copy Markdown
Member Author 
test_compile leaked [2, 2, 2] references, sum=6
test_compile leaked [4, 4, 4] memory blocks, sum=12

Need to check.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@corona10 corona10 changed the title [WIP] gh-112069: Make _PySet_NextEntry to be thread-safe. gh-112069: Make _PySet_NextEntry to be thread-safe. Apr 18, 2024
@corona10

corona10 commented Apr 18, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

test_compile: #118023
test_logging: #102412 (comment)

So leaks are not related to this PR.
Eclips4 reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@corona10
corona10 requested a review from colesbury April 18, 2024 05:47
@corona10 corona10 changed the title gh-112069: Make _PySet_NextEntry to be thread-safe. gh-112069: Add _PySet_NextEntryRef to be thread-safe. Apr 18, 2024
colesbury
colesbury approved these changes Apr 18, 2024
View reviewed changes
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Apr 18, 2024
@corona10
corona10 merged commit 94444ea into python:main Apr 18, 2024
@bedevere-app bedevere-app Bot removed the awaiting merge label Apr 18, 2024
@corona10
corona10 deleted the gh-112069-PySet_NextEntry branch April 18, 2024 15:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@colesbury colesbury colesbury approved these changes
@ericsnowcurrently ericsnowcurrently Awaiting requested review from ericsnowcurrently ericsnowcurrently is a code owner
@markshannon markshannon Awaiting requested review from markshannon markshannon is a code owner
@iritkatriel iritkatriel Awaiting requested review from iritkatriel
@methane methane Awaiting requested review from methane methane is a code owner
@rhettinger rhettinger Awaiting requested review from rhettinger rhettinger is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news topic-free-threading

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@corona10 @bedevere-bot @colesbury
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.