GeezerVison is showing github.com — GH-113464: Get LLVM from `cpython-bin-deps` on Windows by brandtbucher · Pull Request #133278 · python/cpython
Conversation
brandtbucher
commented
May 2, 2025
•
edited by bedevere-app
Bot
Loading
Uh oh!
There was an error while loading. Please reload this page.
This makes LLVM easier to acquire for Windows JIT builds (like the one @zooba is going to do next week). We can now get it from https://github.com/python/cpython-bin-deps/tree/llvm.
I also cleaned up a couple things I tripped over while working on this:
- Fix the regen target to work better with recent changes to how
jit_stencils.h is built (and clean up after itself).
- Don't touch
jit_stencils.h unless it's actually changing, to avoid triggering a rebuild.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Looks great! Though so far, this is the part that makes me sad (watch the timestamps). I guess it's unavoidable though.
Fri, 02 May 2025 00:38:42 GMT
Fetching external libraries...
Fri, 02 May 2025 00:38:42 GMT
Fetching bzip2-1.0.8...
Fri, 02 May 2025 00:38:46 GMT
Fetching mpdecimal-4.0.0...
Fri, 02 May 2025 00:38:47 GMT
Fetching sqlite-3.49.1.0...
Fri, 02 May 2025 00:38:48 GMT
Fetching xz-5.2.5...
Fri, 02 May 2025 00:38:49 GMT
Fetching zlib-ng-2.2.4...
Fri, 02 May 2025 00:38:50 GMT
Fetching external binaries...
Fri, 02 May 2025 00:38:50 GMT
Fetching libffi-3.4.4...
Fri, 02 May 2025 00:38:51 GMT
Fetching openssl-bin-3.0.16.2...
Fri, 02 May 2025 00:38:53 GMT
Fetching tcltk-8.6.[15](https://github.com/python/cpython/actions/runs/14786163251/job/41515167379?pr=133278#step:4:16).0...
Fri, 02 May 2025 00:38:55 GMT
Fetching llvm-19.1.7.0...
Fri, 02 May 2025 00:42:13 GMT
Finished.
Just to confirm, these lines in the build output mean that it's working, yes? (I think the last one isn't from your script, but it's not an error, so that implies success)
=============================================================
JIT support for x86_64-pc-windows-msvc is still experimental!
Please report any issues you encounter.
=============================================================
Regenerate cases
Generated sources are up to date
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Another possibility is to integrate the LLVM fetch into the script itself, so that we can check whether the files are up to date first and only download LLVM if it's needed (using/ignoring a separate install, as appropriate - I could see this being a case where we'd prefer to ignore it when we don't know which version it is)
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Do we need to change 18 to 19 here to unify the build?
|
You can also use a specific version of clang-cl downloaded from |
|
https://github.com/llvm/llvm-project/releases, e.g. |
|
clang+llvm-18.1.8-x86_64-pc-windows-msvc.tar.xz. |
|
Given you have extracted that to <my-clang-dir>, you can use it like so |
|
build.bat --pgo "/p:PlatformToolset=ClangCL" "/p:LLVMInstallDir=<my-clang-dir>" "/p:LLVMToolsVersion=18" |
|
|
|
Setting LLVMToolsVersion to the major version is enough, although you |
|
can be specific and use 18.1.8 in the above example, too. |
|
zooba reacted with thumbs up emoji
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
brandtbucher
commented
May 2, 2025
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
Do we need to change 18 to 19 here to unify the build?
Not sure what you mean by "unify the build", but that LLVM version is unrelated. That's for if you want to build all of CPython with LLVM; this change is just for generating the templates the JIT compiler uses, which has stricter version requirements and can be different than the compiler or version used to build the interpreter itself.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Looks great! Though so far, this is the part that makes me sad (watch the timestamps). I guess it's unavoidable though.
Yeah, it's a bummer that it takes a few minutes (maybe we could trim stuff we don't need from the hosted install, but I imagine most of the size is in the parts that we depend on). But if I understand correctly, it should download only once per machine that's doing Windows builds and remain cached, so it's not like Windows JIT core devs' daily build times just went up or anything.
Plus, we don't actually download it unless you're building the JIT, so most people should be unaffected (for now).
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Plus, we don't actually download it unless you're building the JIT, so most people should be unaffected (for now).
Ah, yeah, I didn't catch that first time through.
And yeah, it'll be once per machine. I was mostly thinking about CI, which uses a fresh machine each time, and also CI that bypasses build.bat (msbuild pcbuild.proj is the typical way, and it's not unreasonable to do it when building releases).
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
brandtbucher
commented
May 2, 2025
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
And yeah, it'll be once per machine. I was mostly thinking about CI, which uses a fresh machine each time, and also CI that bypasses build.bat (msbuild pcbuild.proj is the typical way, and it's not unreasonable to do it when building releases).
Yeah. For the former we need to install it anyways. For the latter, they'll just need to make sure to set the right options to omit the LLVM install when it's not needed if they want to avoid waiting.
(Another option could be to make the LLVM install default to "false" in get_externals.bat, and have build.bat enable it when needed. Not sure if that makes things better, or just more confusing.)
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Reviewers
zooba
Awaiting requested review from zooba
Uh oh!
There was an error while loading. Please reload this page.
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|