GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-143120: pixi builds for free-threading and TSAN by crusaderky · Pull Request #142872 · python/cpython

Conversation

@crusaderky

@crusaderky crusaderky commented Dec 17, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Follow-up to #142469

  • Add pixi infrastructure for
    • freethreading: compiles with --disable-gil
    • tsan-freethreading: compiles with --disable-gil --with-thread-sanitizer
  • Refactor recipes:
    • Make recipe.yaml and pixi.toml all identical symlinks
    • Remove clang-19 since we are not building the experimental jit
    • Remove ld_impl because we are not hacking the makefiles
    • Add site_packages_path
  • Add support for MacOS Intel (specifically macos-15-intel github runners)
  • Signal support for Linux ARM (it already worked but was insofar untested)
  • Remove ASAN/TSAN DLLs hack on MacOS

Tested in CI on https://github.com/crusaderky/hdf5-pixi

  • Issue: Add pixi builds for TSan + free-threading to aid downstream development. #143120

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

lucascolley reacted with laugh emoji wolfv and FFY00 reacted with rocket emoji
All reactions
  • 😄 1 reaction
  • 🚀 2 reactions
@bedevere-app bedevere-app Bot added the awaiting review label Dec 17, 2025
crusaderky
crusaderky commented Dec 17, 2025
View reviewed changes
Comment thread Tools/pixi-packages/free-threading/recipe.yaml Outdated
script:
file: ../build.sh
env:
PYTHON_VARIANT: "free-threading"

@crusaderky crusaderky Dec 17, 2025

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

copy-paste of asan/recipe.yaml, except this one line

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
crusaderky
crusaderky commented Dec 17, 2025
View reviewed changes
Comment thread Tools/pixi-packages/tsan-free-threading/pixi.toml Outdated

@crusaderky crusaderky Dec 17, 2025

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

copy-paste of asan/pixi.toml

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
crusaderky
crusaderky commented Dec 17, 2025
View reviewed changes
Comment thread Tools/pixi-packages/tsan/recipe.yaml Outdated
script:
file: ../build.sh
env:
PYTHON_VARIANT: "tsan"

@crusaderky crusaderky Dec 17, 2025

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

copy-paste of asan/recipe.yaml, except this one line

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@lucascolley

lucascolley commented Dec 17, 2025

Copy link
Copy Markdown
Contributor

@kumaraditya303 @lucascolley any idea what's wrong here?

no clue sorry I haven't seen this before

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@crusaderky
crusaderky marked this pull request as draft December 17, 2025 12:10
@bedevere-app bedevere-app Bot removed the awaiting review label Dec 17, 2025
@crusaderky crusaderky changed the title WIP pixi builds for free-threading and tsan WIP pixi builds for free-threading and TSAN Dec 17, 2025
@crusaderky

crusaderky commented Dec 18, 2025

Copy link
Copy Markdown
Contributor Author

The tsan crash is due to the default mmap_rnd_bits on Ubuntu 24.04; it goes away after lowering it:

$ sudo sysctl vm.mmap_rnd_bits
vm.mmap_rnd_bits = 32  # too high
$ sudo sysctl vm.mmap_rnd_bits=28  # reduce it
vm.mmap_rnd_bits = 28

ubuntu-latest github actions workers seems to be unaffected.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

This was referenced Dec 23, 2025
Add pixi builds for TSan + free-threading to aid downstream development. #143120
Closed
Add note to reduce ASLR randomness for TSan Quansight-Labs/free-threaded-compatibility#304
Merged
@hugovk hugovk changed the title WIP pixi builds for free-threading and TSAN gh-143120: WIP pixi builds for free-threading and TSAN Dec 23, 2025
@hugovk hugovk added the skip news label Dec 23, 2025
@hugovk

hugovk commented Dec 23, 2025

Copy link
Copy Markdown
Member

There's a lot of copy/paste here, which can make maintenance harder. Does pixi have some concept of code reuse or parametrisation? Maybe not the best example, but something like GitHub Actions' reusable workflows?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@hugovk hugovk added the infra CI, GitHub Actions, buildbots, Dependabot, etc. label Dec 23, 2025
@ngoldbaum

ngoldbaum commented Dec 23, 2025

Copy link
Copy Markdown
Contributor

@lucascolley is there anything we can do right now to reduce the duplication here? Or does that require fixes in pixi?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@lucascolley

lucascolley commented Dec 23, 2025

Copy link
Copy Markdown
Contributor

As mentioned in the README already, this is blocked on prefix-dev/pixi#4599.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ngoldbaum
ngoldbaum reviewed Dec 23, 2025
View reviewed changes

@ngoldbaum ngoldbaum left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I opened an issue which is needed to make this mergeable. I'm also playing with a NumPy recipe based on this.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Tools/pixi-packages/README.md Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Tools/pixi-packages/build.sh Resolved

Uh oh!

There was an error while loading. Please reload this page.

@ngoldbaum ngoldbaum mentioned this pull request Dec 23, 2025
ENH: TSan pixi package definitions numpy/numpy#30510
Merged
@crusaderky
crusaderky force-pushed the tsan branch from eb0149f to 242081e Compare January 2, 2026 15:53
crusaderky
crusaderky commented Jan 2, 2026
View reviewed changes
Comment thread Tools/pixi-packages/build.sh Resolved

Uh oh!

There was an error while loading. Please reload this page.

crusaderky added 4 commits January 2, 2026 16:03
@crusaderky
pixi builds for free-threading and tsan
d930e12
@crusaderky
Rename to tsan-free-threading
57d45b9
@crusaderky
Add libclang_rt hack
5ff29b4
@crusaderky
Add troubleshooting for tsan
9a2a896
@crusaderky
crusaderky force-pushed the tsan branch from b528931 to 9a2a896 Compare January 2, 2026 16:03
@crusaderky
crusaderky marked this pull request as ready for review January 2, 2026 16:05
@bedevere-app bedevere-app Bot added the awaiting review label Jan 2, 2026
@crusaderky crusaderky changed the title gh-143120: WIP pixi builds for free-threading and TSAN gh-143120: pixi builds for free-threading and TSAN Jan 2, 2026
@crusaderky

crusaderky commented Jan 2, 2026

Copy link
Copy Markdown
Contributor Author

Ready for final review

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

crusaderky added 7 commits January 23, 2026 07:25
@crusaderky
Reapply "Update Tools/pixi-packages/README.md"
43eead1
This reverts commit cddc933.
@crusaderky
Reapply "Use libcompiler-rt and remove hack"
55242c3
This reverts commit 1ae77a5.
@crusaderky
There is no c_compiler variable
2f5daba
@crusaderky
Don't use system-wide libasan on Linux
fdacca7
@crusaderky
Merge remote-tracking branch 'upstream/main' into tsan
2d4472f
@crusaderky
Don't use symlinks 1/2
defe6a6
@crusaderky
Don't use symlinks 2/2
e33bc48
@crusaderky crusaderky mentioned this pull request Jan 23, 2026
pixi-build-rattler-build could improve variables management prefix-dev/pixi#5393
Open
@crusaderky

crusaderky commented Jan 23, 2026

Copy link
Copy Markdown
Contributor Author

The issue was that c_compiler is not a defined variable, and comparisons to undefined variables always return false.
So these two would always evaluate to false:

      - if: c_compiler == "gcc" and "san" in variant
        then:
          - libsanitizer
      - if: c_compiler == "clang" and "san" in variant
        then:
          - libcompiler-rt

Linux happened to work because, at least on ubuntu-latest and on my desktop, libasan.so and libtsan.so are installed by default system-wide.

I've opened an improvement ticket upstream: prefix-dev/pixi#5393

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@crusaderky

crusaderky commented Jan 23, 2026

Copy link
Copy Markdown
Contributor Author

I've removed the symlinks and replaced them with physical copies + a sync script.
This PR has no known outstanding issues and should be ready for final review and merge.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@crusaderky crusaderky mentioned this pull request Feb 2, 2026
When TSAN build fails for env reasons, it corrupts the build dir #144392
Closed
@crusaderky

crusaderky commented Feb 2, 2026

Copy link
Copy Markdown
Contributor Author

Gentle ping.
This PR has no known outstanding issues and should be ready for final review and merge.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@lucascolley

lucascolley commented Feb 3, 2026

Copy link
Copy Markdown
Contributor

trying this out downstream — I get a solve failure for an env that includes the ASAN Python on this branch, ASAN NumPy on main, compilers, and pythran: https://gist.github.com/lucascolley/2d4335f7f80b770a9aec702e7a8f75f8

removing the pythran dependency results in a successful solve. Any ideas @crusaderky @isuruf ?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@crusaderky

crusaderky commented Feb 3, 2026

Copy link
Copy Markdown
Contributor Author

trying this out downstream — I get a solve failure for an env that includes the ASAN Python on this branch, ASAN NumPy on main, compilers, and pythran: https://gist.github.com/lucascolley/2d4335f7f80b770a9aec702e7a8f75f8

removing the pythran dependency results in a successful solve. Any ideas @crusaderky @isuruf ?

ASAN NumPy on main points to a specific commit of CPython main. Could you try repointing numpy to numpy/numpy#30510 ?

[dependencies]
python.git = "https://github.com/crusaderky/cpython"
python.rev = "tsan"
python.subdirectory = "Tools/pixi-packages/asan"
numpy.git = "https://github.com/ngoldbaum/numpy"
numpy.rev = "tsan-ft-pixi"
numpy.subdirectory = "pixi-packages/asan"
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@crusaderky

crusaderky commented Feb 3, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

Anyway your problem is that you have an upper pin for python in pythran:

  │  ├─ pythran 0.16.1 | 0.17.0 would require
  │  │  └─ python >=3.13.0rc1,<3.14.0a0, which cannot be installed because there are no viable options:

but this recipe builds Python 3.15.
You can backport the whole directory to your fork of the 3.13 branch to work around it (and then fork numpy as well and repoint your fork of numpy to your fork of cpython 3.13).

Alternatively you can fork pythran and just remove the pin and hope it was just out of an abundance of caution.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@lucascolley

lucascolley commented Feb 3, 2026

Copy link
Copy Markdown
Contributor

Anyway your problem is that you have an upper pin for python in pythran

ah thanks, makes sense. Using pythran from git source may get around the issue in the short-term

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

kumaraditya303
kumaraditya303 approved these changes Feb 3, 2026
View reviewed changes

@kumaraditya303 kumaraditya303 left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested the tsan+freethreading pixi config with the following and it worked:

[workspace]
channels = ["conda-forge"]
name = "cpython-pixi-test"
platforms = ["linux-64"]
preview = ["pixi-build"]

[dependencies]
python = { git = "https://github.com/crusaderky/cpython", rev = "tsan", subdirectory = "Tools/pixi-packages/tsan-freethreading" }

I'll merge this by tomorrow if there are no objections.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting review labels Feb 3, 2026
@kumaraditya303
Merge branch 'main' into tsan
325dc99
@kumaraditya303
kumaraditya303 merged commit 37c3554 into python:main Feb 4, 2026
47 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the awaiting merge label Feb 4, 2026
hugovk
hugovk reviewed Feb 4, 2026
View reviewed changes
Comment thread Tools/pixi-packages/README.md
- Using a single `pixi.toml` and `recipe.yaml` for all package variants is blocked on https://github.com/prefix-dev/pixi/issues/4599
- A workaround can be removed from the build script once https://github.com/prefix-dev/rattler-build/issues/2012 is resolved
- Using a single `pixi.toml` and `recipe.yaml` for all package variants is blocked on
[pixi-build-backends#532](https://github.com/prefix-dev/pixi-build-backends/pull/532)

@hugovk hugovk Feb 4, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to update the README.md right now, but noting this repo has been archived and the new PR is at prefix-dev/pixi#5364.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Tools/pixi-packages/README.md
```
To fix it, try reducing `mmap_rnd_bits`:

```bash

@hugovk hugovk Feb 4, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tip for next time: bash for when there's only commands, and console for when there's commands with output. Compare:

$ sudo sysctl vm.mmap_rnd_bits
vm.mmap_rnd_bits = 32  # too high for TSan
$ sudo sysctl vm.mmap_rnd_bits=28  # reduce it
vm.mmap_rnd_bits = 28
$ sudo sysctl vm.mmap_rnd_bits
vm.mmap_rnd_bits = 32  # too high for TSan
$ sudo sysctl vm.mmap_rnd_bits=28  # reduce it
vm.mmap_rnd_bits = 28

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@crusaderky
crusaderky deleted the tsan branch February 6, 2026 12:14
thunder-coding pushed a commit to thunder-coding/cpython that referenced this pull request Feb 15, 2026
@crusaderky @isuruf
@lucascolley @thunder-coding
pythongh-143120: pixi builds for free-threading and TSAN (python#142872)
ecc48a6
Co-authored-by: Isuru Fernando <[email protected]>
Co-authored-by: Lucas Colley <[email protected]>
@h-vetinari h-vetinari mentioned this pull request Feb 17, 2026
[Python] Improve the pyarrow build system apache/arrow#36411
Closed
@raulcd raulcd mentioned this pull request Feb 18, 2026
[Python] Add a PyArrow sanitizers build apache/arrow#49321
Open
@lucascolley lucascolley mentioned this pull request Feb 26, 2026
CI: pixi CI is failing due to upstream changes numpy/numpy#30888
Closed
@ngoldbaum ngoldbaum mentioned this pull request Apr 22, 2026
gh-148858: Remove duplicated recipe.yaml files in Tools/pixi-packages #148859
Merged
ljfp pushed a commit to ljfp/cpython that referenced this pull request Apr 25, 2026
@crusaderky @isuruf
@lucascolley @ljfp
pythongh-143120: pixi builds for free-threading and TSAN (python#142872)
1d76191
Co-authored-by: Isuru Fernando <[email protected]>
Co-authored-by: Lucas Colley <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hugovk hugovk hugovk left review comments
@ngoldbaum ngoldbaum ngoldbaum left review comments
@kumaraditya303 kumaraditya303 kumaraditya303 approved these changes
+2 more reviewers
@isuruf isuruf isuruf left review comments
@lucascolley lucascolley lucascolley left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

infra CI, GitHub Actions, buildbots, Dependabot, etc. skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@crusaderky @lucascolley @hugovk @ngoldbaum @isuruf @kumaraditya303
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.