GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-112536: Set up TSAN CI for free-threading by corona10 · Pull Request #116555 · python/cpython

Conversation

@corona10

@corona10 corona10 commented Mar 10, 2024 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member
  • Issue: Add support for thread sanitizer (TSAN) via --with-thread-sanitizer #112536

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@corona10
pythongh-112536: Set up TSAN CI for free-threading
bd5e85a
@bedevere-app bedevere-app Bot mentioned this pull request Mar 10, 2024
Add support for thread sanitizer (TSAN) via --with-thread-sanitizer #112536
Closed
@corona10 corona10 added DO-NOT-MERGE skip news topic-free-threading labels Mar 10, 2024
@corona10

This comment was marked as resolved.

Sign in to view
corona10 added 3 commits March 10, 2024 13:35
@corona10
Reduce test scope
a7a3b08
@corona10
fix
a7f4826
@corona10
Add _Py_NO_SANITIZE_THREAD annotation
84bccfb
@corona10
corona10 marked this pull request as ready for review March 10, 2024 05:16
@corona10
corona10 requested review from erlend-aasland, ezio-melotti and hugovk as code owners March 10, 2024 05:16
@bedevere-app bedevere-app Bot added the awaiting core review label Mar 10, 2024
@corona10 corona10 removed awaiting core review DO-NOT-MERGE labels Mar 10, 2024
@corona10
corona10 requested a review from colesbury March 10, 2024 05:16
corona10 added 4 commits March 10, 2024 14:17
@corona10
nit
aa15f16
@corona10
nit
0f72690
@corona10
fix
2ce5cdd
@corona10
swtich to clang
05a5882
@corona10
corona10 marked this pull request as draft March 10, 2024 05:36
corona10 added 7 commits March 10, 2024 14:38
@corona10
Update
0b68814
@corona10
Update env
d2a4476
@corona10
fix
f641cad
@corona10
fix
dd88d4a
@corona10
fix
6c96e41
@corona10
Update(?)
179bacd
@corona10
fix
691d0e4
@corona10

corona10 commented Mar 10, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

Hmm, I locally passed the thread-sanitizer test with clang, I am taking a look why CI has an issue.

python.exe(48341,0x1e0bcdc40) malloc: nano zone abandoned due to inability to reserve vm space.
Using random seed: 2688388199
Raised RLIMIT_NOFILE: 256 -> 1024
0:00:00 load avg: 7.94 Run 44 tests in parallel using 4 worker processes
0:00:01 load avg: 7.94 [ 1/44] test_binop passed
0:00:02 load avg: 7.94 [ 2/44] test_binascii passed
0:00:04 load avg: 7.94 [ 3/44] test_bisect passed
0:00:07 load avg: 7.94 [ 4/44] test_base64 passed
0:00:09 load avg: 7.94 [ 5/44] test_array passed
0:00:09 load avg: 7.71 [ 6/44] test_cmath passed
0:00:14 load avg: 7.71 [ 7/44] test_bz2 passed
0:00:18 load avg: 7.81 [ 8/44] test_complex passed
0:00:24 load avg: 7.82 [ 9/44] test_dataclasses passed
0:00:24 load avg: 7.60 [10/44] test_codecs passed
0:00:26 load avg: 7.60 [11/44] test_collections passed
0:00:41 load avg: 7.10 [12/44] test_difflib passed
0:00:42 load avg: 7.10 [13/44] test_bytes passed
0:00:46 load avg: 7.33 [14/44] test_float passed
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@corona10
fix(?)
d89f932
@corona10

corona10 commented Mar 10, 2024

Copy link
Copy Markdown
Member Author
checking for builtin __atomic_load_n and __atomic_store_n functions... no
configure: error: --disable-gil requires mimalloc memory allocator (--with-mimalloc).

Weired..

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

corona10 added 3 commits March 10, 2024 15:04
@corona10
Check order
d833618
@corona10
Weired fix
33317e7
@corona10
Revert to the gcc
cade95e
@corona10

corona10 commented Mar 10, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

I need to check on my Linux machine later.

./Include/cpython/pyatomic_gcc.h:531:3: warning: ‘atomic_thread_fence’ is not supported with ‘-fsanitize=thread’ [-Wtsan]
  531 | { __atomic_thread_fence(__ATOMIC_SEQ_CST); }
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

colesbury
colesbury reviewed Mar 10, 2024
View reviewed changes
Comment thread .github/workflows/build.yml
Comment on lines +485 to +488
- name: Set up GCC-10 for ASAN
uses: egor-tensin/setup-gcc@v1
with:
version: 11

@colesbury colesbury Mar 10, 2024

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The version says "11" but the step name says "GCC-10"

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread .github/workflows/build.yml
- name: Display build info
run: make pythoninfo
- name: Tests
run: ./python -m test --pgo -j4 # Reduce test scope

@colesbury colesbury Mar 10, 2024

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'll want a configuration specifically for tsan. Maybe a --tsan option? We'll only want to run tests that actually use threading.

Brett had a list of tests that fail with PYTHON_GIL=0 -- those are probably a good starting point:

swtaarrs@1fe9165#diff-590c02490d066378045dd48d62e86dc85d3fb4ad934b6a2b2b0b1112e59eaefb

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

erlend-aasland and corona10 reacted with thumbs up emoji
All reactions
  • 👍 2 reactions
Comment thread Objects/obmalloc.c
/***************************/

static int
_Py_NO_SANITIZE_THREAD

@colesbury colesbury Mar 10, 2024

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've had a better experience using suppression lists rather than marking functions as _Py_NO_SANITIZE_THREAD:

  1. Add a file supressions.txt somwhere (maybe Tools/tsan/supressions.txt?)
  2. Set the environment variable TSAN_OPTIONS="suppressions=<path_to_supressions.txt>"

See https://github.com/google/sanitizers/wiki/ThreadSanitizerSuppressions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

erlend-aasland and corona10 reacted with thumbs up emoji
All reactions
  • 👍 2 reactions
Comment thread .github/workflows/build.yml
with:
version: 11
- name: Configure CPython
run: ./configure --disable-gil --with-thread-sanitizer

@colesbury colesbury Mar 10, 2024

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want --with-pydebug?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@corona10

corona10 commented Mar 15, 2024

Copy link
Copy Markdown
Member Author

Actual PR for this issue will be: #116872

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@corona10 corona10 closed this Mar 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@colesbury colesbury colesbury left review comments
@erlend-aasland erlend-aasland Awaiting requested review from erlend-aasland erlend-aasland is a code owner
@ezio-melotti ezio-melotti Awaiting requested review from ezio-melotti ezio-melotti is a code owner
@hugovk hugovk Awaiting requested review from hugovk hugovk is a code owner
@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner will be requested when the pull request is marked ready for review AA-Turner is a code owner
@emmatyping emmatyping Awaiting requested review from emmatyping emmatyping will be requested when the pull request is marked ready for review emmatyping is a code owner
@itamaro itamaro Awaiting requested review from itamaro itamaro will be requested when the pull request is marked ready for review itamaro is a code owner
@webknjaz webknjaz Awaiting requested review from webknjaz webknjaz will be requested when the pull request is marked ready for review webknjaz is a code owner
@JacobCoffee JacobCoffee Awaiting requested review from JacobCoffee JacobCoffee will be requested when the pull request is marked ready for review JacobCoffee is a code owner
@StanFromIreland StanFromIreland Awaiting requested review from StanFromIreland StanFromIreland will be requested when the pull request is marked ready for review StanFromIreland is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news topic-free-threading

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@corona10 @colesbury
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.