GeezerVison is showing github.com — gh-138451: Support custom LLVM installation path by xhochy · Pull Request #138452 · python/cpython
gh-138451: Support custom LLVM installation path
#138452
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement . We’ll occasionally send you account related emails.
Already on GitHub?
Sign in
to your account
Uh oh!
There was an error while loading. Please reload this page.
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
3111c2d
Support custom LLVM installation path
xhochy Sep 3, 2025
8321790
📜🤖 Added by blurb_it.
blurb-it[bot] Sep 3, 2025
e6082d4
Use `LLVM_TOOLS_INSTALL_DIR` to search for LLVM tools
xhochy Sep 5, 2025
8bd8a98
Don't join on empty values.
xhochy Sep 7, 2025
297fe1c
Fix syntax error
xhochy Sep 7, 2025
e9c2cc3
Document LLVM_TOOLS_INSTALL_DIR
xhochy Oct 26, 2025
a57427c
Add LLVM_TOOLS_INSTALL_DIR to configure
xhochy Oct 27, 2025
b90f60b
Fix wording in README
xhochy Dec 2, 2025
06c9e51
Expect tools in <toolsdir>/bin
xhochy Dec 2, 2025
c801103
Add environment variables to PCbuild
xhochy Dec 2, 2025
4801b37
Be more specific in NEWS entry
xhochy Dec 2, 2025
d5e2421
Merge branch 'main' into gh-138451
savannahostrowski Apr 17, 2026
b3a330c
Merge branch 'main' into gh-138451
savannahostrowski Apr 20, 2026
File filter
Filter by extension
.ac
(1)
.md
(1)
.py
(3)
.rst
(1)
.targets
(1)
No extension
(1)
All 6 file types selected
Viewed files
Conversations
Failed to load comments.
Loading
Uh oh!
There was an error while loading. Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading
Uh oh!
There was an error while loading. Please reload this page.
Diff view
Hide whitespace
Diff view
Hide whitespace
There are no files selected for viewing
Original file line number
Diff line number
Diff line change
@@ -0,0 +1 @@
Allow for custom LLVM path using ``LLVM_TOOLS_INSTALL_DIR`` during JIT build.
Original file line number
Diff line number
Diff line change
Expand Up
@@ -129,7 +129,7 @@
<JITArgs Condition="$(Platform) == 'x64'">x86_64-pc-windows-msvc</JITArgs>
<JITArgs Condition="$(Configuration) == 'Debug'">$(JITArgs) --debug</JITArgs>
</PropertyGroup>
<Exec Command='$(PythonForBuild) "$(PySourcePath)Tools\jit\build.py" $(JITArgs) --output-dir "$(GeneratedJitStencilsDir)" --pyconfig-dir "$(PySourcePath)PC"'/>
<Exec Command='$(PythonForBuild) "$(PySourcePath)Tools\jit\build.py" $(JITArgs) --output-dir "$(GeneratedJitStencilsDir)" --pyconfig-dir "$(PySourcePath)PC" --llvm-version="$(LLVM_VERSION)" --llvm-tools-install-dir="$(LLVM_TOOLS_INSTALL_DIR)" '/>
</Target>
<Target Name="_CleanJIT" AfterTargets="Clean">
<Delete Files="@(_JITOutputs)"/>
Expand Down
Original file line number
Diff line number
Diff line change
Expand Up
@@ -9,7 +9,12 @@ Python 3.11 or newer is required to build the JIT.
The JIT compiler does not require end users to install any third-party dependencies, but part of it must be *built* using LLVM[^why-llvm]. You are *not* required to build the rest of CPython using LLVM, or even the same version of LLVM (in fact, this is uncommon).
LLVM version 21 is the officially supported version. You can modify if needed using the `LLVM_VERSION` env var during configure. Both `clang` and `llvm-readobj` need to be installed and discoverable (version suffixes, like `clang-19`, are okay). It's highly recommended that you also have `llvm-objdump` available, since this allows the build script to dump human-readable assembly for the generated code.
LLVM version 21 is the officially supported version. Both `clang` and `llvm-readobj` need to be installed and discoverable (version suffixes, like `clang-21`, are okay). It's highly recommended that you also have `llvm-objdump` available, since this allows the build script to dump human-readable assembly for the generated code.
You can customize the LLVM configuration using environment variables before running configure:
- LLVM_VERSION: Specify a different LLVM version (default: 21)
- LLVM_TOOLS_INSTALL_DIR: Point to a specific LLVM installation prefix when multiple installations exist (the tools are expected in `<dir>/bin`)
It's easy to install all of the required tools:
Expand Down
Expand Up
@@ -62,7 +67,7 @@ choco install llvm --version=21.1.0
### Dev Containers
If you are working on CPython in a [Codespaces instance](https://devguide.python.org/getting-started/setup-building/#using-codespaces), there's no
If you are working on CPython in a [Codespaces instance](https://devguide.python.org/getting-started/setup-building/#using-codespaces), there's no
need to install LLVM as the Fedora 43 base image includes LLVM 21 out of the box.
## Building
Expand Down
Original file line number
Diff line number
Diff line change
Expand Up
@@ -80,7 +80,18 @@ async def _get_brew_llvm_prefix(llvm_version: str, *, echo: bool = False) -> str
@_async_cache
async def _find_tool(tool: str, llvm_version: str, *, echo: bool = False) -> str | None:
async def _find_tool(
tool: str,
llvm_version: str,
llvm_tools_install_dir: str | None,
*,
echo: bool = False,
) -> str | None:
# Explicitly defined LLVM installation location
if llvm_tools_install_dir:
path = os.path.join(llvm_tools_install_dir, "bin", tool)
if await _check_tool_version(path, llvm_version, echo=echo):
return path
# Unversioned executables:
path = tool
if await _check_tool_version(path, llvm_version, echo=echo):
Expand Down
Expand Up
@@ -114,10 +125,11 @@ async def maybe_run(
args: typing.Iterable[str],
echo: bool = False,
llvm_version: str = _LLVM_VERSION,
llvm_tools_install_dir: str | None = None,
) -> str | None:
"""Run an LLVM tool if it can be found. Otherwise, return None."""
path = await _find_tool(tool, llvm_version, echo=echo)
path = await _find_tool(tool, llvm_version, llvm_tools_install_dir, echo=echo)
return path and await _run(path, args, echo=echo)
Expand All
@@ -126,10 +138,17 @@ async def run(
args: typing.Iterable[str],
echo: bool = False,
llvm_version: str = _LLVM_VERSION,
llvm_tools_install_dir: str | None = None,
) -> str:
"""Run an LLVM tool if it can be found. Otherwise, raise RuntimeError."""
output = await maybe_run(tool, args, echo=echo, llvm_version=llvm_version)
output = await maybe_run(
tool,
args,
echo=echo,
llvm_version=llvm_version,
llvm_tools_install_dir=llvm_tools_install_dir,
)
if output is None:
raise RuntimeError(f"Can't find {tool}-{llvm_version}!")
return output
Original file line number
Diff line number
Diff line change
Expand Up
@@ -53,6 +53,7 @@ class _Target(typing.Generic[_S, _R]):
cflags: str = ""
frame_pointers: bool = False
llvm_version: str = _llvm._LLVM_VERSION
llvm_tools_install_dir: str | None = None
known_symbols: dict[str, int] = dataclasses.field(default_factory=dict)
pyconfig_dir: pathlib.Path = pathlib.Path.cwd().resolve()
Expand Down
Expand Up
@@ -85,7 +86,11 @@ async def _parse(self, path: pathlib.Path) -> _stencils.StencilGroup:
group = _stencils.StencilGroup()
args = ["--disassemble", "--reloc", f"{path}"]
output = await _llvm.maybe_run(
"llvm-objdump", args, echo=self.verbose, llvm_version=self.llvm_version
"llvm-objdump",
args,
echo=self.verbose,
llvm_version=self.llvm_version,
llvm_tools_install_dir=self.llvm_tools_install_dir,
)
if output is not None:
# Make sure that full paths don't leak out (for reproducibility):
Expand All
@@ -105,7 +110,11 @@ async def _parse(self, path: pathlib.Path) -> _stencils.StencilGroup:
f"{path}",
]
output = await _llvm.run(
"llvm-readobj", args, echo=self.verbose, llvm_version=self.llvm_version
"llvm-readobj",
args,
echo=self.verbose,
llvm_version=self.llvm_version,
llvm_tools_install_dir=self.llvm_tools_install_dir,
)
# --elf-output-style=JSON is only *slightly* broken on Mach-O...
output = output.replace("PrivateExtern\n", "\n")
Expand Down
Expand Up
@@ -184,7 +193,11 @@ async def _compile(
# Allow user-provided CFLAGS to override any defaults
args_s += shlex.split(self.cflags)
await _llvm.run(
"clang", args_s, echo=self.verbose, llvm_version=self.llvm_version
"clang",
args_s,
echo=self.verbose,
llvm_version=self.llvm_version,
llvm_tools_install_dir=self.llvm_tools_install_dir,
)
if not is_shim:
self.optimizer(
Expand All
@@ -196,7 +209,11 @@ async def _compile(
).run()
args_o = [f"--target={self.triple}", "-c", "-o", f"{o}", f"{s}"]
await _llvm.run(
"clang", args_o, echo=self.verbose, llvm_version=self.llvm_version
"clang",
args_o,
echo=self.verbose,
llvm_version=self.llvm_version,
llvm_tools_install_dir=self.llvm_tools_install_dir,
)
return await self._parse(o)
Expand Down
Original file line number
Diff line number
Diff line change
Expand Up
@@ -43,6 +43,9 @@
"--cflags", help="additional flags to pass to the compiler", default=""
)
parser.add_argument("--llvm-version", help="LLVM version to use")
parser.add_argument(
"--llvm-tools-install-dir", help="Installation location of LLVM tools"
)
args = parser.parse_args()
for target in args.target:
target.debug = args.debug
Expand All
@@ -52,6 +55,8 @@
target.pyconfig_dir = args.pyconfig_dir
if args.llvm_version:
target.llvm_version = args.llvm_version
if args.llvm_tools_install_dir:
target.llvm_tools_install_dir = args.llvm_tools_install_dir
target.build(
comment=comment,
force=args.force,
Expand Down
2 changes: 1 addition & 1 deletion
2
configure
Oops, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Original file line number
Diff line number
Diff line change
Expand Up
@@ -2850,7 +2850,7 @@ AS_VAR_IF([jit_flags],
[],
[AS_VAR_APPEND([CFLAGS_NODIST], [" $jit_flags"])
AS_VAR_SET([REGEN_JIT_COMMAND],
["\$(PYTHON_FOR_REGEN) \$(srcdir)/Tools/jit/build.py ${ARCH_TRIPLES:-$host} --output-dir . --pyconfig-dir . --cflags=\"$CFLAGS_JIT\" --llvm-version=\"$LLVM_VERSION\""])
["\$(PYTHON_FOR_REGEN) \$(srcdir)/Tools/jit/build.py ${ARCH_TRIPLES:-$host} --output-dir . --pyconfig-dir . --cflags=\"$CFLAGS_JIT\" --llvm-version=\"$LLVM_VERSION\" --llvm-tools-install-dir=\"$LLVM_TOOLS_INSTALL_DIR\" "])
Comment thread
xhochy marked this conversation as resolved.
Resolved
Uh oh!
There was an error while loading. Please reload this page.
AS_VAR_IF([Py_DEBUG],
[true],
[AS_VAR_APPEND([REGEN_JIT_COMMAND], [" --debug"])],
Expand Down
Loading
Oops, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.