GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-117657: Fix data races reported by TSAN in some set methods by aisk · Pull Request #120914 · python/cpython

Conversation

@aisk

@aisk aisk commented Jun 23, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

set_add_key and set_discard_key have the same issue and should be fixed, although they have not been detected by TSAN.

After this change, the TSAN doesn't report data races for this method on my machine.

  • Issue: Make TSAN tests pass with the GIL disabled in free-threaded builds #117657

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@aisk
Fix data races reported by TSAN in some set methods
64f07da
@aisk
aisk requested a review from rhettinger as a code owner June 23, 2024 15:49
@bedevere-app bedevere-app Bot added the awaiting review label Jun 23, 2024
@bedevere-app bedevere-app Bot mentioned this pull request Jun 23, 2024
Make TSAN tests pass with the GIL disabled in free-threaded builds #117657
Closed
@aisk aisk added the skip news label Jun 23, 2024
@rhettinger
rhettinger removed their request for review June 23, 2024 17:16
@colesbury

colesbury commented Jun 25, 2024

Copy link
Copy Markdown
Contributor

Thanks! This pattern occurs in a bunch of places outside of setobject.c. Can you fix those as well? I think it's worth fixing the entire pattern at once.

I think it's also probably worth refactoring the fast hash check into its own inline function (maybe in pycore_object.h or pycore_unicodeobject.h) that does the whole check the type, check the unicode hash, return it if it's not -1, otherwise delegate to PyObject_Hash().

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@colesbury
colesbury self-requested a review June 25, 2024 17:57
aisk added 2 commits June 30, 2024 15:40
@aisk
Merge remote-tracking branch 'upstream/main' into tsan-set-keys
ec7f2ad
@aisk
Introduce _PyObject_HashFast and apply it to the codebase
1571d01
@aisk
aisk requested a review from markshannon as a code owner July 1, 2024 17:17
@aisk

aisk commented Jul 1, 2024

Copy link
Copy Markdown
Member Author

Thanks! This pattern occurs in a bunch of places outside of setobject.c. Can you fix those as well? I think it's worth fixing the entire pattern at once.

I think it's also probably worth refactoring the fast hash check into its own inline function (maybe in pycore_object.h or pycore_unicodeobject.h) that does the whole check the type, check the unicode hash, return it if it's not -1, otherwise delegate to PyObject_Hash().

Thanks for the review! I think this new function is not related to unicode object only, so I added it in pycore_object.h, and named it _PyObject_HashFast().

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@aisk
apply _PyObject_HashFast on dictobject.c
fefc810
@aisk
aisk requested a review from methane as a code owner July 1, 2024 17:59
colesbury
colesbury reviewed Jul 1, 2024
View reviewed changes

@colesbury colesbury left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks - this looks good. Just minor comments about _PyObject_HashFast

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Include/internal/pycore_object.h Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Include/internal/pycore_object.h Resolved

Uh oh!

There was an error while loading. Please reload this page.

aisk and others added 2 commits July 2, 2024 02:25
@aisk @colesbury
Update Include/internal/pycore_object.h
ad6f28e
Co-authored-by: Sam Gross <[email protected]>
@aisk
Refactor _PyObject_HashFast
ef3d438
colesbury
colesbury approved these changes Jul 1, 2024
View reviewed changes

@colesbury colesbury left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! LGTM

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting review labels Jul 1, 2024
@colesbury colesbury added awaiting review needs backport to 3.13 only security fixes awaiting merge and removed awaiting merge awaiting review labels Jul 1, 2024
@colesbury
colesbury merged commit 294e724 into python:main Jul 1, 2024
@miss-islington-app

miss-islington-app Bot commented Jul 1, 2024

Copy link
Copy Markdown

Thanks @aisk for the PR, and @colesbury for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the awaiting merge label Jul 1, 2024
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jul 1, 2024
@aisk @miss-islington
pythongh-117657: Fix data races reported by TSAN in some set methods (p…
36ed4cf
…ythonGH-120914)

Refactor the fast Unicode hash check into `_PyObject_HashFast` and use relaxed
atomic loads in the free-threaded build.

After this change, the TSAN doesn't report data races for this method.
(cherry picked from commit 294e724)

Co-authored-by: AN Long <[email protected]>
@bedevere-app

bedevere-app Bot commented Jul 1, 2024

Copy link
Copy Markdown

GH-121240 is a backport of this pull request to the 3.13 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the needs backport to 3.13 only security fixes label Jul 1, 2024
@vstinner

vstinner commented Jul 1, 2024

Copy link
Copy Markdown
Member

Nice change, I like _PyObject_HashFast() which factorizes the code.

aisk reacted with heart emoji
All reactions
  • ❤️ 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

colesbury pushed a commit that referenced this pull request Jul 1, 2024
@miss-islington @aisk
[3.13] gh-117657: Fix data races reported by TSAN in some set methods (…
06fd745
…GH-120914) (#121240)

Refactor the fast Unicode hash check into `_PyObject_HashFast` and use relaxed
atomic loads in the free-threaded build.

After this change, the TSAN doesn't report data races for this method.
(cherry picked from commit 294e724)

Co-authored-by: AN Long <[email protected]>
@bedevere-bot

bedevere-bot commented Jul 1, 2024

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Debian root 3.13 has failed when building commit 06fd745.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1441/builds/255) and take a look at the build logs.
  4. Check if the failure is related to this commit (06fd745) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1441/builds/255

Failed tests:

  • test_repl

Failed subtests:

  • test_asyncio_repl_is_ok - test.test_repl.TestInteractiveInterpreter.test_asyncio_repl_is_ok

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
  File "/root/buildarea/3.13.angelico-debian-amd64/build/Lib/asyncio/__main__.py", line 109, in run
    raise OSError(errno.ENOTTY, "tty required", "stdin")
xiting asyncio REPL...
Fatal Python error: _enter_buffered_busy: could not acquire lock for <_io.BufferedWriter name='<stderr>'> at interpreter shutdown, possibly due to daemon threads
Python runtime state: finalizing (tstate=0x0000557e5090ff50)


Traceback (most recent call last):
xiting asyncio REPL...
  File "/root/buildarea/3.13.angelico-debian-amd64/build/Lib/asyncio/__main__.py", line 109, in run
    raise OSError(errno.ENOTTY, "tty required", "stdin")
Fatal Python error: _enter_buffered_busy: could not acquire lock for <_io.BufferedWriter name='<stderr>'> at interpreter shutdown, possibly due to daemon threads
Python runtime state: finalizing (tstate=0x0000561498692f50)


Traceback (most recent call last):
  File "/root/buildarea/3.13.angelico-debian-amd64/build/Lib/test/test_repl.py", line 199, in test_asyncio_repl_is_ok
    assert_python_ok("-m", "asyncio")
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/root/buildarea/3.13.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line 180, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/root/buildarea/3.13.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line 165, in _assert_python
    res.fail(cmd_line)
    ~~~~~~~~^^^^^^^^^^
  File "/root/buildarea/3.13.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line 75, in fail
    raise AssertionError("Process return code is %d\n"
    ...<13 lines>...
                            err))
AssertionError: Process return code is -6
command line: ['/root/buildarea/3.13.angelico-debian-amd64/build/python', '-X', 'faulthandler', '-I', '-m', 'asyncio']
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@colesbury

colesbury commented Jul 1, 2024

Copy link
Copy Markdown
Contributor

Buildbot failure is unrelated to this PR.

See #119909

aisk reacted with eyes emoji
All reactions
  • 👀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@aisk
aisk deleted the tsan-set-keys branch July 2, 2024 08:30
Akasurde pushed a commit to Akasurde/cpython that referenced this pull request Jul 3, 2024
@aisk @Akasurde
pythongh-117657: Fix data races reported by TSAN in some set methods (p…
6c7d265
…ython#120914)

Refactor the fast Unicode hash check into `_PyObject_HashFast` and use relaxed
atomic loads in the free-threaded build.

After this change, the TSAN doesn't report data races for this method.
estyxx pushed a commit to estyxx/cpython that referenced this pull request Jul 17, 2024
@aisk @estyxx
pythongh-117657: Fix data races reported by TSAN in some set methods (p…
a015c3b
…ython#120914)

Refactor the fast Unicode hash check into `_PyObject_HashFast` and use relaxed
atomic loads in the free-threaded build.

After this change, the TSAN doesn't report data races for this method.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@colesbury colesbury colesbury approved these changes
@markshannon markshannon Awaiting requested review from markshannon markshannon is a code owner
@methane methane Awaiting requested review from methane methane is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@aisk @colesbury @vstinner @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.