GeezerVison is showing github.com — Concurrent compilation changes marshal output in free-threaded builds · Issue #156504 · python/cpython
Concurrent compilation changes marshal output in free-threaded builds #156504Labels interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)topic-free-threadingtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DescriptionBug description:
Concurrent compilation on free-threaded CPython builds can produce marshal output that differs byte-for-byte from serial compilation of the same source.
This occurs both with the GIL disabled and with the GIL explicitly re-enabled. It reproduces on Windows and Linux/WSL. Standard CPython builds used as controls did not reproduce the difference.
This was discovered while implementing threaded bytecode compilation in pypa/pip#14286.
Reproducer
A standalone standard-library-only reproducer is available here:
It does not import or execute pip. It:
- creates 64 copies of a minimized source fixture;
- compiles corresponding files serially and with a
ThreadPoolExecutor, in fresh child interpreters;
- fixes logical filenames,
PYTHONHASHSEED, and SOURCE_DATE_EPOCH;
- compares checked-hash
.pyc files or raw marshal.dumps(compile(...)) payloads;
- unmarshals differing outputs and recursively compares documented code-object fields.
Example using the raw marshal backend:
python3.14t repro_threaded_pyc_determinism.py --gil off --backend marshal --workers 4 --output repro-ft-off-marshal --require-difference
python3.14t repro_threaded_pyc_determinism.py --gil on --backend marshal --workers 4 --output repro-ft-on-marshal --require-difference The output directory must be new or empty.
Expected behavior
Serial and concurrent compilation of the same source, using the same logical filename and deterministic environment settings, should produce byte-identical marshal output.
Actual behavior
On free-threaded builds, some concurrently compiled files have different serialized bytes from their serial counterparts.
The differing payloads have:
- equal lengths;
- equal
.pyc headers when using py_compile;
- equal values for all recursively inspected code-object fields after unmarshalling, including bytecode, constants, names, filenames, line tables, exception tables, and nested code objects.
The raw marshal backend reproduces the behavior without py_compile or .pyc headers. Its differing offsets are exactly 16 bytes earlier than the corresponding .pyc offsets, confirming that the .pyc header is not responsible.
An empty semanticDifferences result does not establish that the object identity or alias graph is identical. The precise cause has not yet been determined.
Observed results
Windows
Windows 11 prerelease VM, eight virtual CPUs, CPython 3.15.0rc1 free-threaded build and corresponding standard CPython 3.15 build:
| Build |
GIL |
Backend |
Result |
| Free-threaded |
Off |
py_compile |
Reproduced; 49 differing bytes in a 3,447-byte .pyc |
| Free-threaded |
On |
py_compile |
Reproduced; 49 differing bytes in a 3,447-byte .pyc |
| Free-threaded |
Off |
marshal |
Reproduced; 49 differing bytes in a 3,431-byte payload |
| Free-threaded |
On |
marshal |
Reproduced; 49 differing bytes in a 3,431-byte payload |
| Standard |
On/default |
Both |
No difference in 10 attempts |
Linux under WSL2
Ubuntu 26.04 LTS, kernel 6.18.33.2-microsoft-standard-WSL2, native ext4 storage, four available CPUs:
| Build |
GIL |
Backend |
Result |
| CPython 3.14.7 free-threaded |
Off |
py_compile |
Reproduced on attempt 1 |
| CPython 3.14.7 free-threaded |
On |
py_compile |
Reproduced on attempt 1 |
| CPython 3.14.7 free-threaded |
Off |
marshal |
Reproduced on attempt 1 |
| CPython 3.14.7 free-threaded |
On |
marshal |
Reproduced on attempt 1 |
| CPython 3.14.4 standard |
On/default |
Both |
No difference in 10 attempts |
The WSL differences contained 50 differing bytes per output. All inspected decoded code-object field values were equal.
Impact
Although the decoded code objects appear semantically equivalent, byte-different outputs affect:
- reproducible builds;
- artifact hashing;
- build caches;
- deterministic CI output;
- tools that record
.pyc hashes.
Re-enabling the GIL on a free-threaded build does not prevent the behavior. As a temporary precaution, pip’s proposed threaded compilation path detects Py_GIL_DISABLED at build time and remains serial for free-threaded builds.
CPython versions tested on:
3.15
Operating systems tested on:
Linux
Linked PRs
ActivitySurbhiAgarwal1 commented on Aug 29, 2026 Hi @khmyznikov , I took a look at marshal.c and pycore_object.h on main, think I found the mechanism.
Marshal picks how to encode a string (inline vs TYPE_REF, and TYPE_SHORT_ASCII_INTERNED vs TYPE_SHORT_ASCII) based on _PyObject_IsUniquelyReferenced(v). On GIL builds that's just Py_REFCNT(ob) == 1 — stable. On free-threaded builds it's:
return (_Py_IsOwnedByCurrentThread(ob) &&
_Py_atomic_load_uint32_relaxed(&ob->ob_ref_local) == 1 &&
_Py_atomic_load_ssize_relaxed(&ob->ob_ref_shared) == 0);That depends on thread ownership + cross-thread refcounts, which can shift while multiple threads are compiling concurrently.
Your repro compiles 64 near-identical fixtures in parallel, so they share a ton of interned identifiers (self, dunders, etc). Two threads racing to touch the same interned string can flip this check — which flips the encoding of that string, not its value. Matches everything you saw: bytes differ but decoded fields are equal, GIL re-enabled at runtime doesn't help (compile-time macro, not runtime), standard builds never hit it, and the 16-byte offset gap is just the .pyc header.
Not confirmed yet — next step is mapping the differing byte offsets to marshal's type bytes and checking if they land on shared identifiers. If that holds, fix is probably decoupling the type-byte choice (only needs PyUnicode_CHECK_INTERNED, race-free) from the TYPE_REF caching (doesn't need global refcount at all, the per-dump memo already tracks what's written).
Happy to dig into this further could someone assign this to me so I can run the offset-mapping check and follow up with a patch if it confirms out?
Refs: [marshal.c](https://github.com/python/cpython/blob/main/Python/marshal.c) · [pycore_object.h](https://github.com/python/cpython/blob/main/Include/internal/pycore_object.h) Reacted by Gleb Khmyznikov dylanpulver commented on Aug 31, 2026 Confirmed on main@03503dc (free-threaded, macOS/arm64 — a new platform on top of the Windows/WSL data). In-process, 64 identical fixtures, serial vs ThreadPoolExecutor(8): the threaded output differs from serial in 49 bytes out of 3426, and marshal.loads() of both is structurally identical (co_code byte-identical, same consts/names). The payloads are equivalent; only the encoding moves.
Mechanism. It's w_ref() in Python/marshal.c, the early-return guard:
if (_PyObject_IsUniquelyReferenced(v) &&
!(PyUnicode_CheckExact(v) && PyUnicode_CHECK_INTERNED(v)))
return 0; // write inline, do NOT add to the ref table_PyObject_IsUniquelyReferenced is Py_REFCNT(ob)==1 on the default build, but on free-threading it's owned-by-current-thread && ob_ref_local==1 && ob_ref_shared==0. That predicate is timing-dependent: the moment any other thread touches an object, biased reference counting moves it to the shared state and ob_ref_shared is no longer 0. So an object inlined in the serial run gets memoized in the threaded run — its type byte gains FLAG_REF (| 0x80) and it enters the reference table.
Causal proof (deterministic, single-threaded): marshal a runtime-built small tuple that is uniquely referenced vs one to which I hold one extra reference — identical value, and the inner type byte flips 0x29→0xa9 (TYPE_SMALL_TUPLE | FLAG_REF), nothing else. Flip one reference, one bit moves.
Why re-enabling the GIL doesn't help (the part the report flagged as surprising): a single cross-thread read of an otherwise-unique object flips the same byte, and it flips identically under -X gil=1. Re-enabling the GIL doesn't change the free-threaded build's refcount representation, so the shared-refcount transition still fires.
Why 49 bytes from only ~5 real flips: in this repro exactly 5 objects gain FLAG_REF (3 TYPE_SMALL_TUPLE, 2 TYPE_STRING). Each extra registration shifts every later TYPE_REF index up by one, so the other 44 differing bytes are downstream index low-bytes (+5 ×39, +2 ×5; max delta 5 = the number of new registrations). The value survives because a larger-but-consistent ref index decodes to the same object.
One refinement to @SurbhiAgarwal1's note: the culprit predicate is exactly right, but the byte that moves is FLAG_REF on non-interned objects (bytes/tuples), not TYPE_SHORT_ASCII_INTERNED↔TYPE_SHORT_ASCII. That byte depends only on PyUnicode_CHECK_INTERNED (race-free), and interned strings take the memo path unconditionally anyway.
Fix direction (not proposing a PR here): the _PyObject_IsUniquelyReferenced fast-path in w_ref/w_complete is a size optimization that is inherently non-deterministic under free-threading, because the memoization decision depends on a live refcount. Reproducible marshal output means that decision can't read refcounts — gate it on something stable, or always memoize eligible types. Happy to help chase whichever direction the marshal maintainers prefer.
Disclosure: this diagnosis was produced by Claude (Opus) working as a coding agent; the reproductions, causal tests, and byte accounting above were all run and verified against a free-threaded build.
MetadataMetadatainterpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)topic-free-threadingtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DevelopmentNo branches or pull requestsIssue actions
|