GeezerVison is showing github.com — Let a configured JIT "on" compile through cmake on Windows by eriknw · Pull Request #659 · python-graphblas/python-graphblas
Conversation
Stacked on #656. One commit: "Let a configured "on" compile through cmake where
SuiteSparse uses cmake". #652 adds a sentence to the user guide saying that on Windows
a configured "on" becomes "run" ("Say that the JIT stays at "run" on Windows, where
SuiteSparse compiles with cmake"); this PR rewrites that sentence.
The problem
SuiteSparse built with MSVC, as conda-forge builds it for Windows, compiles JIT kernels
only through cmake: gb.ss.config["jit_use_cmake"] is True there and cannot be set
False. SuiteSparse passes the configured compiler to cmake as CMAKE_C_COMPILER, which
cmake's Visual Studio generator ignores; it finds Visual Studio's compiler itself.
conda-forge's builds configure the build machine's cl.exe, a path under
C:/Program Files/Microsoft Visual Studio/ that users rarely have. #652 sets "on" at
init only where the configured compiler exists, so on Windows a configured "on"
(GRAPHBLAS_JIT_C_CONTROL=on, gb.config.set(jit_c_control="on") or a YAML file)
quietly became "run", and nothing compiled. main behaves the same.
The cmake path itself works on Windows CI. #652's test_set_jit_c_control_at_init
pointed the compiler setting at a directory and expected the probe to fail; on every
Windows job with a JIT build in one CI run of this stack (81 jobs, SuiteSparse 8.2.0
through 10.5.2 and upstream), the probe compiled through cmake and the control stayed
"on" (fixed in #652 by "Skip the unusable-compiler probe check where SuiteSparse
compiles with cmake").
What changes
- A configured "on" looks for cmake where SuiteSparse uses it. At init, where
jit_use_cmake is True, python-graphblas checks for cmake on the PATH
(shutil.which) instead of the configured compiler path, then probes as before by
compiling a small type. A failing probe still means "run", with no error. Nothing
changes where jit_use_cmake is False, which is everywhere but MSVC builds unless a
user sets it.
- UDT work never raises the default "run" where
jit_use_cmake is True. Elsewhere,
UDT operators and registration from C source raise the default "run" to "on" when
they need a kernel. Through cmake each kernel costs about 10 seconds (below), so
there only an "on" the user set compiles, even where the configured compiler happens
to exist (the build machine's cl.exe path, or conda's clang.exe, which the
compiler repair finds). UDT operators then call their Numba functions. Registering C
source under that "run" raises an error that says SuiteSparse compiles through cmake
here and that gb.ss.config["jit_c_control"] = "on" opts in, instead of suggesting
gb.ss.fix_jit_config(), which does not help there.
- The probe's name includes the cmake settings. The probe type is named after a
hash of the toolchain settings, so a toolchain the cache has not seen compiles a
probe of its own rather than loading one that another toolchain left there.
jit_use_cmake and jit_c_cmake_libs are now part of that hash; without them, a
probe through cmake on Linux or macOS loaded the probe compiled directly and tested
nothing. Each existing cache compiles its probe once more (about 0.3 s directly). A
test fails if either setting is taken out of the hash, and checks that init, which
reads the settings through the C API, names the probe as later calls do.
- Diagnostics.
scripts/jit_diagnostics.py prints cmake's path and
jit_use_cmake, times the diagnostic type's registration, and ends with a new
section that sets "on" as init does, with burble on, then prints the resulting
control and how long the probe took, through cmake or the compiler. When
fix_jit_config() returns None it now says "no compiler found to repair from"; it
said "no conda", also with CONDA_PREFIX set.
- Docs. The user guide's JIT section says that on Windows "on" needs cmake
(conda install cmake) and Visual Studio or its Build Tools rather than the
configured compiler, that UDT work does not raise the default "run" there, and that a
new kernel takes about 10 seconds through cmake on Windows. The UDT page's list of
what raises "run" gains the exception, and so do the jit_c_control and
jit_use_cmake entries of gb.ss.config.
Cost of a kernel through cmake
In that CI run the test made exactly one cold compile through cmake per Windows job, so
the time it took measures one kernel. Over the 81 jobs: median 10.4 s, 90th percentile
16.2 s, fastest 2.6 s, slowest 56 s. Each kernel gets a fresh build directory and a
fresh cmake configure, so a real kernel costs at least this much. For comparison, on an
M-series Mac the probe compiles in 0.2 to 0.3 s directly and in about 0.6 s through
cmake with its Makefile generator.
What Windows CI should show
- JIT diagnostics: a
cmake: which=... line and jit_use_cmake: True. The "JIT
compilation test" still fails under the default "run", now with the cmake message.
The new last section shows the cmake and MSBuild output, then Result: on and
Probe through cmake: N s, one more sample of the cost above in every job. A wheel
draw (SuiteSparse without the JIT) shows Result: run and No probe: ....
- Tests: on JIT draws,
test_ss_import_leaves_jit_c_control_alone[on] (a fresh
process with GRAPHBLAS_JIT_C_CONTROL=on) and
test_set_jit_c_control_at_init_compiles_through_cmake assert "on" after a real
compile through cmake; if it fails, the assertion asks whether Visual Studio's C
compiler is installed. The rest of test_ssjit.py still skips its JIT tests on
Windows (see the second question below), and the tests that need UDT work to raise
"run" skip wherever SuiteSparse compiles only through cmake.
Tested locally on macOS (arm64, SuiteSparse 10.5.2, cmake 4.3): test_ssjit.py and
test_core.py with --runslow (90 passed, 1 skipped), and test_op.py and
test_dtype.py with --runslow. The cmake branches ran there with jit_use_cmake set
to True, including a real probe through cmake on a fresh cache; MSVC and the Visual
Studio generator run only on CI.
Open questions
- Should UDT work ever use cmake by default? In this PR it never does. At about
10 s per kernel, a session that touches a handful of UDT operations would wait tens
of seconds to minutes the first time (the cache keeps the kernels for later runs).
Without it, Windows users get the Numba fallback, typically 2 to 3 times slower for
elementwise operations, unless they set "on".
fix_jit_config() in the tests' _setup_jit. It repairs only the compiler path,
so on Windows it returns None and test_ssjit.py compiles no JIT kernels there.
Making it accept cmake would run them all through cmake: test_ssjit.py and
test_core.py compile about 261 kernels into a fresh cache, roughly 45 minutes per
Windows job at 10 s each.
🤖 Generated with Claude Code
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
codecov
Bot
commented
Nov 3, 2026
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
|
❌ Patch coverage is 91.00529% with 17 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (51-coverage-main-gaps@d78f352). Learn more about missing BASE report.
Additional details and impacted files
@@ Coverage Diff @@
## 51-coverage-main-gaps #659 +/- ##
========================================================
Coverage ? 99.42%
========================================================
Files ? 109
Lines ? 35247
Branches ? 4014
========================================================
Hits ? 35044
Misses ? 117
Partials ? 86
🚀 New features to boost your workflow:
- ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
SuiteSparse built with MSVC always compiles JIT kernels through cmake, and
passes the configured compiler to it as CMAKE_C_COMPILER, which cmake's
Visual Studio generator ignores. conda-forge's Windows builds name the build
machine's cl.exe, which users rarely have, so the check at init that the
configured compiler exists turned a configured "on" into "run" there.
Where jit_use_cmake is True, init now looks for cmake on the PATH instead,
and probes as before: a failing probe still means "run".
UDT work no longer raises the default "run" where jit_use_cmake is True,
even where the configured compiler exists (the build machine's cl.exe on a
host with that Visual Studio, or conda's clang.exe, which the repair finds):
a cold probe through cmake took a median of 10.4 s (2.6 to 56 s) over 81
wave-5 Windows CI jobs, so there only an "on" the user set compiles.
Registering C source under that "run" says so, and how to opt in. Tests that
need UDT work to raise "run" skip where SuiteSparse compiles only through
cmake.
The probe's type is named after jit_use_cmake and jit_c_cmake_libs too, so
switching to cmake compiles a probe of its own rather than loading the one
compiled directly; a test checks that, and that init names the probe as
later calls do. scripts/jit_diagnostics.py reports where cmake is and
jit_use_cmake, times the diagnostic type's registration, and sets "on" as
init does, timing the probe, with burble on: on Windows CI that times a
compile through cmake, about the least each new kernel costs there.
The user guide and the jit_use_cmake and jit_c_control entries of
gb.ss.config say what "on" needs on Windows (cmake and Visual Studio), that
UDT work does not raise "run" there, and that a new kernel takes about 10
seconds there.
test_ssjit's _jit_through_cmake_here decided whether an "on" set at init
compiles through cmake by setting "on" and reading it back, which runs
no compile. On Windows with psg's 8.0.2.1 wheel, built with MinGW,
SuiteSparse compiles through cmake but passes gcc-style flags, which
cmake's Visual Studio generator hands to cl.exe, which refuses them
(D8021: invalid numeric argument '/Wundef'). Init rightly keeps "run"
there, but the helper said yes, so
test_ss_import_leaves_jit_c_control_alone[on] and
test_set_jit_c_control_at_init_compiles_through_cmake asserted "on" and
failed, with a message asking whether Visual Studio was installed, which
it was.
The helper now asks a real probe, as init does. A failed probe with
gcc-style flags (-W..., -std=, -f...) means cmake cannot work here, so
the tests check the "run" fallback instead; a failed probe with flags
for cl.exe still counts, so the tests report a broken Visual Studio
setup. The message no longer blames a missing Visual Studio. A test
drives the helper with a stand-in probe and flags on any host.
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|