GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — GH-140479: Update JIT builds to use LLVM 21 by savannahostrowski · Pull Request #140973 · python/cpython

Conversation

@savannahostrowski

@savannahostrowski savannahostrowski commented Nov 4, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Well, here's a fun treat...looks like the logic to fetch the LLVM binary as a release artifact from https://github.com/python/cpython-bin-deps/releases/tag/llvm-21.1.4.0 never actually worked 😓. CI only passed for Windows on #140329 because LLVM 20 is included in the base image we use for Windows builds (example: https://github.com/actions/partner-runner-images/blob/main/images/arm-windows-11-image.md#language-and-runtime).

As such, we should merge #141002 before this one (Windows CI will fail otherwise!).

  • Issue: Update JIT builds to use LLVM 21 #140479

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
Fidget-Spinner, Sacul0457, and v-fox reacted with rocket emoji
All reactions
  • 🚀 3 reactions
savannahostrowski added 9 commits November 3, 2025 15:57
@savannahostrowski
Initial commit for LLVM 21
7e7ebaf
@savannahostrowski
Update CI
ae4a01d
@savannahostrowski
Another CI fix
da9b795
@savannahostrowski
Add debugging
d079df9
@savannahostrowski
Add fix for exe
322ea9b
@savannahostrowski
Update tarball extraction
27a91ac
@savannahostrowski
Fix for windows
2ca8603
@savannahostrowski
Clean up after debugging
39e1339
@savannahostrowski
Uncomment CI
38105ce
@savannahostrowski savannahostrowski added the build The build process and cross-build label Nov 4, 2025
@savannahostrowski
savannahostrowski requested a review from brandtbucher as a code owner November 4, 2025 04:56
@savannahostrowski savannahostrowski added the dependencies Pull requests that update a dependency file label Nov 4, 2025
@savannahostrowski
savannahostrowski requested review from a team, diegorusso, ezio-melotti and hugovk as code owners November 4, 2025 04:56
@savannahostrowski savannahostrowski added the topic-JIT label Nov 4, 2025
@savannahostrowski
savannahostrowski requested a review from AA-Turner as a code owner November 4, 2025 04:56
@savannahostrowski savannahostrowski added the infra CI, GitHub Actions, buildbots, Dependabot, etc. label Nov 4, 2025
@bedevere-app bedevere-app Bot added the awaiting core review label Nov 4, 2025
@bedevere-app bedevere-app Bot mentioned this pull request Nov 4, 2025
Update JIT builds to use LLVM 21 #140479
Closed
@blurb-it
📜🤖 Added by blurb_it.
e85c8f8
@savannahostrowski savannahostrowski mentioned this pull request Nov 4, 2025
Bump to Fedora 43 for JIT builds with LLVM 21 python/cpython-devcontainers#62
Merged
savannahostrowski added 2 commits November 3, 2025 21:01
@savannahostrowski
Fix README
3ca9733
@savannahostrowski
Merge branch 'llvm-21' of https://github.com/savannahostrowski/cpython …
4508fc5 
…into llvm-21
savannahostrowski
savannahostrowski commented Nov 4, 2025
View reviewed changes
Comment thread Tools/jit/README.md Resolved

Uh oh!

There was an error while loading. Please reload this page.
@diegorusso

diegorusso commented Nov 4, 2025

Copy link
Copy Markdown
Contributor

Well, here's a fun treat...looks like the logic to fetch the LLVM binary as a release artifact from https://github.com/python/cpython-bin-deps/releases/tag/llvm-21.1.4.0 never actually worked 😓. CI only passed for Windows on #140329 because LLVM 20 is included in the base image we use for Windows builds (see https://github.com/actions/partner-runner-images/blob/main/images/arm-windows-11-image.md#language-and-runtime).

This PR patches up the logic for extracting the tarball in get_externals.py and then...just bumps the version to 21.

  • Issue: Update JIT builds to use LLVM 21 #140479

What about the Windows x86_64 image? Does it have llvm 20 as well?

Also, in this PR we do 2 things: fix some logic and bump to 21. Wouldn't it be better to do 2 PRs? In case we need to revert to llvm 20 (for any reason) we will revert to a working fetching for version 20.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@savannahostrowski savannahostrowski mentioned this pull request Nov 4, 2025
GH-136895: Fixes for pulling LLVM as a release artifact #141002
Merged
@savannahostrowski
Remove fixes for Windows release artifacts
ef52c5a
@savannahostrowski

savannahostrowski commented Nov 4, 2025

Copy link
Copy Markdown
Member Author

@diegorusso Same deal with x86_64 - see https://github.com/actions/runner-images/blob/win22/20251021.76/images/windows/Windows2022-Readme.md#language-and-runtime

Sure - see #141002
diegorusso reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
diegorusso
diegorusso approved these changes Nov 4, 2025
View reviewed changes

@diegorusso diegorusso left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Provided #141002 gets merged before, this can go in as well. Thanks for splitting them up.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
savannahostrowski reacted with heart emoji
All reactions
  • ❤️ 1 reaction
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Nov 4, 2025
@savannahostrowski
Merge branch 'main' into llvm-21
9778bdd
Fidget-Spinner
Fidget-Spinner approved these changes Nov 7, 2025
View reviewed changes

@Fidget-Spinner Fidget-Spinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks simple enough, let's do this!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
savannahostrowski reacted with heart emoji
All reactions
  • ❤️ 1 reaction
@pablogsal

pablogsal commented Nov 9, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

I have tried to rebase https://github.com/pablogsal/cpython-unwind to check the unwinding. Looks like the function to test now needs to be more complicated beause the JIT refuses to call functions if there are branches now and requires tons of iterations to start jitting. I would love if someone can take a look at run.py. With the code we can validate all unwinding methods
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@Fidget-Spinner

Fidget-Spinner commented Nov 9, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

The main branch's JIT doesn't seem to want to compile the example due to trace stack underflow. So I reduced the side exit count on my tracing JIT example and tried. Seems like LLVM 21 doesn't fix the problem :(

PYTHON_JIT=0
-----------
Testing various unwinding methods on the same deep call stack...

[GNU backtrace]
Total frames: 27
Showing first 10 frames:
  # 0  /home/ken/Documents/GitHub/cpython-unwind/stackunwind.cpython-315-x86_64-linux-gnu.so(+0x280a) [0x78674f7a580a]
  # 1  ../cpython/python(PyObject_Vectorcall+0x33) [0x58c0e08d2303]
  # 2  ../cpython/python(_PyEval_EvalFrameDefault+0xf7a1) [0x58c0e086b141]
  # 3  ../cpython/python(+0x2860aa) [0x58c0e0a430aa]
  # 4  ../cpython/python(PyObject_Vectorcall+0x33) [0x58c0e08d2303]
  # 5  ../cpython/python(_PyEval_EvalFrameDefault+0x3fb3) [0x58c0e085f953]
  # 6  ../cpython/python(+0x2860aa) [0x58c0e0a430aa]
  # 7  ../cpython/python(PyObject_Vectorcall+0x33) [0x58c0e08d2303]
  # 8  ../cpython/python(_PyEval_EvalFrameDefault+0x3fb3) [0x58c0e085f953]
  # 9  ../cpython/python(+0x2860aa) [0x58c0e0a430aa]
  ... (17 more frames)

[libunwind]
Total frames: 26
Showing first 10 frames:
  # 0  PyObject_Vectorcall+0x33
  # 1  _PyEval_EvalFrameDefault+0xf7a1
  # 2  _PyEval_Vector+0x21a
  # 3  PyObject_Vectorcall+0x33
  # 4  _PyEval_EvalFrameDefault+0x3fb3
  # 5  _PyEval_Vector+0x21a
  # 6  PyObject_Vectorcall+0x33
  # 7  _PyEval_EvalFrameDefault+0x3fb3
  # 8  _PyEval_Vector+0x21a
  # 9  PyObject_Vectorcall+0x33
  ... (16 more frames)

[libdw (DWARF)]
Total frames: 26
Showing first 10 frames:
  # 0  PyObject_Vectorcall
  # 1  _PyEval_EvalFrameDefault
  # 2  _PyEval_Vector
  # 3  PyObject_Vectorcall
  # 4  _PyEval_EvalFrameDefault
  # 5  _PyEval_Vector
  # 6  PyObject_Vectorcall
  # 7  _PyEval_EvalFrameDefault
  # 8  _PyEval_Vector
  # 9  PyObject_Vectorcall
  ... (16 more frames)

[Manual frame pointers]
Total frames: 4
Showing first 4 frames:
  # 0  0x58c0e08d2303
  # 1  0x2
  # 2  0x58c0e0dbb7e0
  # 3  0x2


PYTHON_JIT=1
------------
[GNU backtrace]
Total frames: 6
Showing first 6 frames:
  # 0  /home/ken/Documents/GitHub/cpython-unwind/stackunwind.cpython-315-x86_64-linux-gnu.so(+0x280a) [0x7a0a357d080a]
  # 1  ../cpython/python(PyObject_Vectorcall+0x33) [0x57385e3d3303]
  # 2  ../cpython/python(_PyEval_EvalFrameDefault+0xf7a1) [0x57385e36c141]
  # 3  ../cpython/python(+0x2860aa) [0x57385e5440aa]
  # 4  ../cpython/python(PyObject_Vectorcall+0x33) [0x57385e3d3303]
  # 5  [0x7a0a3398d4fb]

[libunwind]
Total frames: 5
Showing first 5 frames:
  # 0  PyObject_Vectorcall+0x33
  # 1  _PyEval_EvalFrameDefault+0xf7a1
  # 2  _PyEval_Vector+0x21a
  # 3  PyObject_Vectorcall+0x33
  # 4  <unknown>

[libdw (DWARF)]
Total frames: 5
Showing first 5 frames:
  # 0  PyObject_Vectorcall
  # 1  _PyEval_EvalFrameDefault
  # 2  _PyEval_Vector
  # 3  PyObject_Vectorcall
  # 4  <unknown>

[Manual frame pointers]
Total frames: 4
Showing first 4 frames:
  # 0  0x57385e3d3303
  # 1  0x(nil)
  # 2  0x57385e8bc7e0
  # 3  0x(nil)

Notice the missing frames!

However, I have made a very interesting discovery --- compiling the interpreter with a recent enough clang version (tested with clang-20/21) to the JIT restores the frames!!!

[GNU backtrace]
Total frames: 29
Showing first 10 frames:
  # 0  /home/ken/Documents/GitHub/cpython-unwind/stackunwind.cpython-315-x86_64-linux-gnu.so(+0x280a) [0x797cf3f7c80a]
  # 1  ../cpython/python(PyObject_Vectorcall+0x43) [0x64be6334e693]
  # 2  ../cpython/python(_PyEval_EvalFrameDefault+0x5d3e) [0x64be6347f63e]
  # 3  ../cpython/python(+0x217706) [0x64be63479706]
  # 4  ../cpython/python(PyObject_Vectorcall+0x43) [0x64be6334e693]
  # 5  ../cpython/python(_PyEval_EvalFrameDefault+0x3768) [0x64be6347d068]
  # 6  ../cpython/python(+0x217706) [0x64be63479706]
  # 7  ../cpython/python(PyObject_Vectorcall+0x43) [0x64be6334e693]
  # 8  ../cpython/python(_PyEval_EvalFrameDefault+0x3768) [0x64be6347d068]
  # 9  ../cpython/python(+0x217706) [0x64be63479706]
  ... (19 more frames)

[libunwind]
Total frames: 28
Showing first 10 frames:
  # 0  PyObject_Vectorcall+0x43
  # 1  _PyEval_EvalFrameDefault+0x5d3e
  # 2  _PyEval_Vector+0x266
  # 3  PyObject_Vectorcall+0x43
  # 4  _PyEval_EvalFrameDefault+0x3768
  # 5  _PyEval_Vector+0x266
  # 6  PyObject_Vectorcall+0x43
  # 7  _PyEval_EvalFrameDefault+0x3768
  # 8  _PyEval_Vector+0x266
  # 9  PyObject_Vectorcall+0x43
  ... (18 more frames)

[libdw (DWARF)]
Total frames: 28
Showing first 10 frames:
  # 0  PyObject_Vectorcall
  # 1  _PyEval_EvalFrameDefault
  # 2  _PyEval_Vector
  # 3  PyObject_Vectorcall
  # 4  _PyEval_EvalFrameDefault
  # 5  _PyEval_Vector
  # 6  PyObject_Vectorcall
  # 7  _PyEval_EvalFrameDefault
  # 8  _PyEval_Vector
  # 9  PyObject_Vectorcall
  ... (18 more frames)

[Manual frame pointers]
Total frames: 16
Showing first 10 frames:
  # 0  0x64be6334e693
  # 1  0x2
  # 2  0x64be63479706
  # 3  0x2
  # 4  0x64be63479706
  # 5  0x2
  # 6  0x64be63479706
  # 7  0x(nil)
  # 8  0x64be63479706
  # 9  0x(nil)
  ... (6 more frames)

So the solution to the stack unwinding seems to be to mandate compiilng CPython with the same clang as the JIT!
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@pablogsal

pablogsal commented Nov 9, 2025

Copy link
Copy Markdown
Member

So the solution to the stack unwinding seems to be to mandate compiilng CPython with the same clang as the JIT!

We need to understand what the hell it's going on here because otherwise the risk it's that this breaks randomly in the future
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@Fidget-Spinner

Fidget-Spinner commented Nov 9, 2025

Copy link
Copy Markdown
Member

Another data point: old GCC (11) compiled with frame pointers with JIT also allows us to see the frames.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@brandtbucher

brandtbucher commented Nov 10, 2025

Copy link
Copy Markdown
Member

Just to be clear, merely upgrading to LLVM 21 isn’t sufficient to get unwinding. We also need to do the reserved frame pointer thing, which isn’t in this branch.
savannahostrowski and Fidget-Spinner reacted with thumbs up emoji
All reactions
  • 👍 2 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@brandtbucher

brandtbucher commented Nov 10, 2025

Copy link
Copy Markdown
Member

That can come after.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@savannahostrowski

savannahostrowski commented Nov 10, 2025

Copy link
Copy Markdown
Member Author

Yeah, was chatting with Brandt earlier this weekend about this and will probably go investigate the unwinding piece after this gets merged.
Fidget-Spinner reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@Fidget-Spinner

Fidget-Spinner commented Nov 10, 2025

Copy link
Copy Markdown
Member

I think it would be good to merge this soon. The meta benchmarking runners don't have JIT results this week because they don't have LLVM 20. It would be a little bit of churn to request they upgrade twice (once for LLVM 20, once for LLVM 21), so I was thinking if I could ask them nicely to just upgrade to LLVM 21 once :).
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@savannahostrowski

savannahostrowski commented Nov 12, 2025

Copy link
Copy Markdown
Member Author

Discussed in our weekly sync, merging 🚀 !
hugovk and Fidget-Spinner reacted with rocket emoji
All reactions
  • 🚀 2 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@savannahostrowski
Merge branch 'main' into llvm-21
39438c0
@savannahostrowski
savannahostrowski enabled auto-merge (squash) November 12, 2025 17:13
@savannahostrowski
savannahostrowski merged commit d162c42 into python:main Nov 12, 2025
109 of 111 checks passed

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot removed the awaiting merge label Nov 12, 2025
@chris-eibl chris-eibl mentioned this pull request Nov 30, 2025
Jit stencils on Windows contain debug data in 3.15 #142050
Closed
@savannahostrowski
savannahostrowski deleted the llvm-21 branch December 4, 2025 22:59
StanFromIreland pushed a commit to StanFromIreland/cpython that referenced this pull request Dec 6, 2025
@savannahostrowski @StanFromIreland
pythonGH-140479: Update JIT builds to use LLVM 21 (python#140973)
f0d1af7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@diegorusso diegorusso diegorusso approved these changes
@Fidget-Spinner Fidget-Spinner Fidget-Spinner approved these changes
@brandtbucher brandtbucher Awaiting requested review from brandtbucher brandtbucher is a code owner
@ezio-melotti ezio-melotti Awaiting requested review from ezio-melotti ezio-melotti is a code owner
@hugovk hugovk Awaiting requested review from hugovk hugovk is a code owner
@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

build The build process and cross-build dependencies Pull requests that update a dependency file infra CI, GitHub Actions, buildbots, Dependabot, etc. topic-JIT

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@savannahostrowski @diegorusso @pablogsal @Fidget-Spinner @brandtbucher
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.