Description

matthiasgoergens

opened on Jul 30, 2026

Issue body actions

Feature or enhancement

Proposal

The UBSan job in .github/workflows/build.yml runs against exactly one configuration: GIL-enabled, no JIT. Two configurations that CI already builds — and that distributions already ship — are never sanitized. Both are cheap to add, and I have measured what each would have caught.

1. UBSan × free-threading: true

The sanitizer matrix pairs TSan with free-threading: {false, true} but UBSan only with false:

        free-threading:
        - false
        - true
        sanitizer:
        - TSan
        include:
        - check-name: Undefined behavior
          sanitizer: UBSan
          free-threading: false

So files that only compile under Py_GIL_DISABLED are never built under UBSan. That is not a hypothetical gap: Python/uniqueid.c:185 shifted a negative per-thread refcount delta left, which is UB, and it fired from interpreter startup and from every GC cycle. Over the full suite with all suppressions disabled:

	UB reports	test failures	failing files
before	1762	529	61
after #154915	0	0	0

Once #154915 lands, this configuration is green, so adding it is a regression guard rather than a source of new work.

2. UBSan × --enable-experimental-jit

jit.yml builds the JIT in several configurations and tail-call.yml builds --with-tail-call-interp, but neither is ever combined with a sanitizer. The JIT's own C code — the stencil patcher in Python/jit.c, and the tier-2 optimiser in Python/optimizer*.c which only compiles under -D_Py_TIER2=1 — is therefore never sanitized in CI.

This one has already proven productive when done by hand: gh-139269 (an unaligned uint64_t store in Python/jit.c's patch_* functions, which segfaulted release builds) was found by building --enable-experimental-jit with -fsanitize=address,undefined, and gh-142476 was an ASan-found leak in allocate_executor.

I ran the full test suite against a machine-code JIT build under UBSan, with every entry in Tools/ubsan/suppressions.txt disabled: clean, zero reports, 51,459 tests across 493 files, 4m50s. Verified non-vacuous — _testinternalcapi.get_jit_backend() returns jit and a hot loop produces real machine-code ranges, so this was the copy-and-patch JIT rather than a silent fallback to the uop interpreter.

Since it is already green, this is also a pure regression guard.

Two smaller findings from the same sweep

• local-bounds is enabled nowhere. --with-undefined-behavior-sanitizer injects plain -fsanitize=undefined, and local-bounds is not in that group even though it detects genuine UB. I built with it on and ran the full suite: zero findings, including no false positives from the trailing variable-length array idiom, which was the obvious risk. It looks free to add.
• float-divide-by-zero (also not in the group) yields exactly one hit, Modules/expat/xmlparse.c:869, in vendored libexpat, which deliberately relies on IEEE-754 +inf and documents that in a comment. Not worth enabling without excluding expat.

On cost

The usual objection to widening the matrix is build time. For reference, a full PGO + ThinLTO + UBSan build — the heaviest combination I tried, including the instrumented build, the profile run, the rebuild and the ThinLTO link — took 9 min 43 s at --jobs=8. A plain UBSan build is far cheaper, and the JIT suite run above took under five minutes.

(That PGO/LTO configuration found nothing new, incidentally. It is worth noting only because --enable-optimizations and --with-lto appear in no workflow at all, while Arch, Debian and python-build-standalone all ship them. UBSan's checks are source-level, so PGO/LTO mostly changes which UB is reached; miscompilation from UB would show up as test failures rather than sanitizer reports, which is a different exercise.)

Linked PRs

• gh-148286: Fix undefined behaviour in per-thread refcount merging #154915 — fixes the free-threaded UB, prerequisite for adding that matrix entry

Has this already been discussed elsewhere?

This is a follow-up to gh-148286.

Links to previous discussion of this feature:

• Various UB warnings found by UBSan #148286
• JIT: UB: unaligned store in patch_* functions #139269
• ASan memory leak in test_gc with --enable-experimental-jit (allocate_executor) #142476

Activity

1. picnixz

   added

   type-featureA feature request or enhancementA feature request or enhancement

   infraCI, GitHub Actions, buildbots, Dependabot, etc.CI, GitHub Actions, buildbots, Dependabot, etc.

   on Jul 30, 2026
2. StanFromIreland commented on Jul 30, 2026

   

   StanFromIreland

   on Jul 30, 2026

   Member

   More actions

   How well is this covered by Buildbots? I'm also not sure we can expand it to the JIT currently with the development pause, it does fall under a new feature (albeit an infra one).
3. matthiasgoergens commented on Aug 1, 2026

   

   matthiasgoergens

   on Aug 1, 2026

   ContributorAuthor

   More actions

   Good questions — I checked the fleet rather than guessing.

   Corrections. This comment has been edited twice, and I would rather the
   record show that than quietly read as if it were right first time.

   2026-08-08: the UBSan row of the table below said 0. There are 20. I
   matched on the builder name "UBSan"; they are called "Usan".

   2026-08-09: the two measurements quoted at the bottom did not survive
   re-running. Both are corrected there, and the issue body has been updated
   with the details.

   Buildbot coverage

   Querying buildbot.python.org/api/v2/builders (1588 builders, all
   branches and PR builders included), matching on the ubsan/asan/nogil
   tags rather than on builder names:

   sanitizer / config	builders
   UBSan	20
   TSan	0
   ASan	34
   free-threading	136
   JIT	0

   The UBSan builders are AMD64 Arch Linux Usan {3.9 … 3.15, 3.x, PR} and
   AMD64 Arch Linux Usan Function {3.9 … 3.15, 3.x, PR}, all tagged
   clang, ubsan, sanitizer. They are active, not vestigial: on the
   development branch, AMD64 Arch Linux Usan 3.x and
   AMD64 Arch Linux Usan Function 3.x both last built green on 2026-08-08.
   (The one exception is AMD64 Arch Linux Usan PR, which last ran 2026-01-05
   and failed in its git/clean steps rather than in any test — its worker is
   online and its sibling Usan Function PR ran fine on 2026-07-30, so this
   looks like a stale workspace on a builder that only runs when a PR requests
   it.)

   So buildbots do cover UBSan, on Arch with clang, across every active
   branch — and that partly duplicates the GitHub Actions job. I was wrong to
   say otherwise.

   What is still true, and is what this issue asks for: none of those twenty
   builders carries a nogil or jit tag. UBSan × free-threading and
   UBSan × JIT are covered nowhere — not in GitHub Actions, not on the
   buildbots. That matters concretely here, because the uniqueid.c bug lives
   in code that only compiles under Py_GIL_DISABLED, so no existing UBSan
   builder could have caught it, however many of them there are.

   Worth adding, since it bears on how much the sanitizer job is worth:
   UBSan only makes a selected set of undefined behaviour observable, and
   only when you build for it. None of the three bugs behind this proposal
   misbehaves in an ordinary build — I checked. So sanitizer coverage is
   not redundant with the ordinary Buildbot fleet; it is the only thing
   that sees these at all.

   The JIT half, and the development pause

   Fair, and I do not want to argue release policy. Happy to split this:

   • UBSan × free-threading is a one-line matrix addition to an
     existing job, and once gh-148286: Fix undefined behaviour in per-thread refcount merging #154915 lands the configuration is green. On
     unpatched main it is worse than red: with this workflow's own
     halt_on_error=1, the free-threaded build aborts during interpreter
     startup and regrtest never runs. With gh-148286: Fix undefined behaviour in per-thread refcount merging #154915 applied, and using this
     workflow's suppression file, options and test command on its own compiler:
     Result: SUCCESS, 51,517 tests, no runtime error: reports.
     (The figure I originally quoted here, "1762 UB reports → 0", did not
     survive re-running and has been withdrawn — see the issue body.)
   • UBSan × JIT can wait for the pause to lift. I will keep running it
     by hand in the meantime. (I originally cited "51,459 tests, zero reports,
     with every suppression disabled" as though it were a result on main; it was
     measured on a tree that already carried the three fixes under review, so it
     shows the JIT adds no UB beyond those. Corrected in the issue body.)

   If splitting sounds right I will retitle this issue to the
   free-threading half and open a separate one for the JIT.

Issue actions