GeezerVison is showing github.com — 3.14: objects de-immortalized by old stable-ABI extensions break truthiness checks in specialized bytecode · Issue #158401 · python/cpython
3.14: objects de-immortalized by old stable-ABI extensions break truthiness checks in specialized bytecode #158401Labels 3.14bugs and security fixesbugs and security fixes3.15bugs and security fixesbugs and security fixes3.16new features, bugs and security fixesnew features, bugs and security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DescriptionBug report
Bug description:
Extensions compiled against 3.11 or older headers implement Py_DECREF as an inline --ob_refcnt. That includes abi3 wheels built on 3.11, which are common. Since 3.12 the interpreter hands out references to immortal objects without incrementing their refcount, so each such reference the extension releases is a net decrement. 3.14 starts immortal objects at 3 << 30 and treats (int32_t)ob_refcnt < 0 as immortal, which leaves a margin of 2**30 (see gh-125174).
That margin runs out in real long-running processes. Home Assistant on 3.14.6 crashes after hours to days because PyAV's cp311-abi3 wheels drain True by a few counts per av.open() and per FFmpeg log line (PyAV-Org/PyAV#2420, home-assistant/core#178218). A core dump from an affected instance had _Py_TrueStruct.ob_refcnt == 0x7ffb2327, which is 2**30 + 318,681 below 0xC0000000.
On 3.12 and 3.13, losing immortality is fairly benign: the object is then refcounted like any other, and results stay correct. On 3.14 it is a silent correctness failure. The reason is in the default build's stackrefs:
PyStackRef_IsTrue/IsFalse/IsNone compare the tagged bits exactly._PyStackRef_FromPyObjectNew and PyStackRef_MakeHeapSafe decide whether to tag with _Py_IsImmortal(), which reads the refcount.PyStackRef_FromPyObjectSteal decides with ob_flags.
Once True's refcount is below 2**31, a True loaded by a specialized attribute load (LOAD_ATTR_INSTANCE_VALUE and others) or passed through RETURN_VALUE is untagged, and POP_JUMP_IF_TRUE treats it as false. True from constants, comparisons and C calls still tests true, so the failures look random. In Home Assistant, Thread.name asserted Thread.__init__() not called, Condition.notify raised cannot notify on un-acquired lock, traceback printed None as the exception type, and so on, across threads, until the process exited.
Reproducer. ctypes stands in for the extension's unbalanced decrefs:
import ctypes, sys
print(sys.version)
class C:
def __init__(self):
self.flag = True
def check_attr(o):
if not o.flag: # LOAD_ATTR_INSTANCE_VALUE, POP_JUMP_IF_TRUE
return "WRONG: o.flag is True but tested false"
return "ok"
def get_true():
return True
def check_return():
if not get_true(): # CALL_PY_EXACT_ARGS, RETURN_VALUE, POP_JUMP_IF_TRUE
return "WRONG: get_true() returned True but tested false"
return "ok"
o = C()
for _ in range(1000): # warm up so both functions get specialized
check_attr(o), check_return()
refcnt = ctypes.c_uint32.from_address(id(True)) # low 32 bits of ob_refcnt (64-bit little-endian)
print("before: ob_refcnt = %#x" % refcnt.value, check_attr(o), check_return())
refcnt.value = 2**31 - 10**6 # 3.14: 2**30 + 10**6 raw decrements from 0xc0000000
print("after: ob_refcnt = %#x" % refcnt.value, check_attr(o), check_return())
print("o.flag is True:", o.flag is True, "| bool(o.flag):", bool(o.flag))Output with the official Docker images (python:3.x, Linux aarch64, default build):
3.14.7 (main, Sep 19 2026, 03:34:22) [GCC 14.2.0]
before: ob_refcnt = 0xc0000000 ok ok
after: ob_refcnt = 0x7ff0bdc0 WRONG: o.flag is True but tested false WRONG: get_true() returned True but tested false
o.flag is True: True | bool(o.flag): True
3.15.0rc2 (main, Sep 19 2026, 03:32:36) [GCC 14.2.0]
before: ob_refcnt = 0xc0000000 ok ok
after: ob_refcnt = 0x7ff0bdc0 WRONG: o.flag is True but tested false WRONG: get_true() returned True but tested false
o.flag is True: True | bool(o.flag): True
3.13.15 (main, Sep 19 2026, 03:38:46) [GCC 14.2.0]
before: ob_refcnt = 0xffffffff ok ok
after: ob_refcnt = 0x7ff0bdc0 ok ok
o.flag is True: True | bool(o.flag): True
3.12.14 (main, Sep 19 2026, 03:37:20) [GCC 14.2.0]
before: ob_refcnt = 0xffffffff ok ok
after: ob_refcnt = 0x7ff0bdc0 ok ok
o.flag is True: True | bool(o.flag): True
The value has to be far enough below 2**31. At exactly 0x7fffffff, the first mortal-path incref brings True back to 0x80000000, and it is immortal again.
The drift itself is the extension's bug, and PyAV is fixing its build. But 3.14 turns it from a lost optimization into wrong branches, and nothing in the process points at the cause. Some options, in no particular order:
- Tag consistently.
_PyStackRef_FromPyObjectNew and PyStackRef_MakeHeapSafe could use ob_flags & _Py_IMMORTAL_FLAGS as PyStackRef_FromPyObjectSteal does. Alternatively, PyStackRef_IsTrue/IsFalse/IsNone could ignore the tag bit. Either would fix the truth tests, though not the drift.
- Treat statically allocated objects (
_Py_STATICALLY_ALLOCATED_FLAG) as immortal whatever their refcount says.
- Periodically reset the refcount of static immortal singletons, for example during GC, as suggested in "Immortal" objects aren't immortal and that breaks things. #125174.
CPython versions tested on:
3.12, 3.13, 3.14, 3.15
Operating systems tested on:
Linux Reacted by Patrick Decat Reacted by Patrick Decat Activitypicnixz commented on Sep 29, 2026 should I understand that the problem is extensions not properly managing refcounts or is there something else? I don't understand the issue exactly here. AFAIU:
The drift itself is the extension's bug, and PyAV is fixing its build
So that's the extension problem? or is it ours? clairernovotny commented on Oct 1, 2026 Both, in different ways.
The drift itself is the extension's bug: PyAV's abi3 wheels were compiled against 3.11 headers, so their inline Py_DECREF is a raw decrement, applied to references (e.g. Py_True from PyObject_RichCompare) that 3.12+ hands out without an incref. PyAV 19.0.0 builds against 3.12 headers and no longer drifts (measured: 0 vs. −12k to −34k per 4,000 av.open calls on 17.x/18.x).
The part that may be CPython's is what happens once such drift takes a singleton below the immortality threshold. On 3.12/3.13 that is harmless, and the reproducer above prints ok ok. On 3.14/3.15, specialized code starts treating True as false (WRONG: o.flag is True but tested false), and in a real process that showed up as "impossible" assertion failures across threads, then a crash. Since the stable ABI still lets extensions built with 3.11 headers load on 3.14, the question is whether 3.14 should tolerate a de-immortalized singleton rather than silently miscompute, for example by not deriving the stackref tag from _Py_IsImmortal, or by checking ob_flags consistently. picnixz commented on Oct 1, 2026 Ah I see, ok now it's clearer. What we never want is true becoming false at some point if extensions correctly manage their references. If this is the case, then this means something is wrong.
The problem is that 3.11 is security-only and this can't really be considered a security issue. The best we can do is possibly detecting on 3.14+ whether we are using old stable ABI or not to determine what to do with the refcount properly.
I will let @markshannon decide whwat to do because I don't have enough insight on whether checking immortality properly could actually slow down things (I belieev it can, but by how much I don't know). In some sense, may I read this issue as "3.14 broke Stable ABI because of immortality assumptions"? clairernovotny commented on Oct 1, 2026 Yes, I think that's a fair reading, with one refinement: the extension is managing its references correctly by the rules it was compiled for.
Under 3.11 semantics, PyObject_RichCompare returns a new reference to Py_True and the extension Py_DECREFs it, which is balanced. On 3.12+, the runtime skips the incref for immortal objects, but the 3.11-header inline Py_DECREF in the extension still does a raw --ob_refcnt. So code that is correct under the limited API it targets produces a net decrement per call, and in a long-running process that adds up. In the Home Assistant case it crossed the threshold in hours to days, depending on how much FFmpeg logged.
PEP 683 anticipated old extensions touching immortal refcounts and relied on the margin being too large to matter. That held on 3.12/3.13 because losing immortality there is benign. On 3.14 it isn't, because the stackref fast paths derive the tag from _Py_IsImmortal() while other paths use the static flags, so True can compare unequal to itself.
So I agree the fix belongs in 3.14+, not 3.11. A cheap option might be to make PyStackRef_IsTrue/IsFalse/IsNone (and the tagging of these static singletons) independent of the current refcount, for example by comparing the untagged pointer or by tagging statically allocated immortals from ob_flags. That avoids an immortality check on the hot incref/decref paths. Whether that is acceptable performance-wise is Mark's call. nthanhcongaasc commented on Nov 4, 2026 Another affected extension: PyQt6 (6.11.0 and the latest 6.12.0, cp310-abi3 wheels), CPython 3.14.7, Windows x64.
Drift source. Every PyQt6 method that returns None adds exactly +1 to None's raw refcount, and that +1 is never taken back. Methods returning other values add nothing. True/False were unaffected in my measurements.
| call (100k times) |
None raw refcount per call |
QPoint.setX(), QObject.setObjectName(), QColor.setRed(), QApplication.processEvents() |
+1.00 |
QWidget.isVisible() (returns bool) |
0 |
| pure-Python function / 3.14 builtin returning None |
0 |
Time to failure is hours, not days. The pattern while w.isVisible(): app.processEvents() is very common in PyQt apps, and every iteration is +1. In a real app this was about 160k/s, so None lost immortality after roughly 1h50m. In the live process I read _Py_NoneStruct as low32=0x0edd14b7, ob_overflow=1, while True, False, ... and () were all still 0xC0000000.
The None case is more severe than the truthiness case. Once None is no longer immortal, the following break process-wide and stay broken until restart:
- every Python-level
__init__: TypeError: __init__() should return None, not 'NoneType'
yield from over a builtin iterator: AttributeError: 'range_iterator' object has no attribute 'send' (seen in pandas' RangeIndex.__iter__)warnings.catch_warnings(): TypeError: _py_warnings.simplefilter() argument after * must be an iterable, not NoneType
As a result, logging, traceback and pandas all fail, so the application cannot even log the error.
In the broken process I checked the co_consts of every live function, about 20k of which contain None. All of them point to the same _Py_NoneStruct. It is the same object; only the stackref tagging differs.
Minimal repro (no memory writes; takes about 5 minutes):
import ctypes
from PyQt6.QtCore import QPoint
low32 = lambda: ctypes.c_uint32.from_address(id(None)).value
overflow = lambda: ctypes.c_uint16.from_address(id(None) + 4).value
class A:
def __init__(self):
pass
f = QPoint().setX
before = low32()
for _ in range(100_000):
f(5)
print("None refcount delta per call:", (low32() - before) / 100_000) # 1.0
while overflow() == 0: # about 1.07e9 calls
for _ in range(5_000_000):
f(5)
A() # TypeError: __init__() should return None, not 'NoneType'Because a well-behaved, widely used abi3 extension hits this within hours of normal use, +1 for making PyStackRef_IsNone/IsTrue/IsFalse independent of the current refcount, for example by tagging static immortals from ob_flags. picnixz commented on Nov 4, 2026 vstinner commented on Nov 7, 2026
Extensions compiled against 3.11 or older headers implement Py_DECREF as an inline --ob_refcnt. (...) PyAV's cp311-abi3 wheels
and:
PyQt6 (6.11.0 and the latest 6.12.0, cp310-abi3 wheels)
If you build PyAV and PyQt6 with Python 3.12 or newer, do you still hit the issue?
I changed Py_INCREF/DECREF() implementation to make them an opaque function call in Python 3.12 to fix this exact problem: https://discuss.python.org/t/limited-c-api-implement-py-incref-and-py-decref-as-function-calls/27592. Sadly, it means that you need a different binary for Python 3.12+ than the binary for Python 3.11 and older.
This refcount issue is just a timebomb awaiting to explode :-( Building C extensions with Python 3.12 or newer fix the root issue.
In the past, I added _Py_IsImmortalLoose() (commit b826e45) to try to reduce the risk of hitting this refcnt/stable ABI issue (issues gh-121528 and gh-118997). Later, Mark made commit c901437 to "GH-125174: Make immortal objects more robust, following design from PEP 683". nthanhcongaasc commented on Nov 8, 2026
If you build PyAV and PyQt6 with Python 3.12 or newer, do you still hit the issue?
In my case,the PyQt maintainer will build future PyQt6 releases and snapshots against the 3.14 headers (mailing list). I'll rerun the repro above on the first such build and report back. Reacted by Victor Stinner da-woods commented on Nov 9, 2026 So I think this is mainly a Python bug and should be fixed there. But at least the PyAV code comes from Cython (PyQt6 doesn't though).
One thing we discussed at the time was using Py_IncRef/Py_DecRef is the Limited API rather than using the macros? We dismissed it at the time as a lot of effort for no real benefit. Would it be worth reconsidering this? vstinner commented on Nov 9, 2026
One thing we discussed at the time was using Py_IncRef/Py_DecRef is the Limited API rather than using the macros? We dismissed it at the time as a lot of effort for no real benefit. Would it be worth reconsidering this?
I only changed Py_INCREF/DECREF() implementation to become a function call in the limited C API version 3.12, because I was worried about the impact on performance. Py_INCREF() implements as ob->ob_refcnt++ is more efficient! You can also see that in 2023, the discussion on that change was quite long: https://discuss.python.org/t/limited-c-api-implement-py-incref-and-py-decref-as-function-calls/27592. So it wasn't an easy decision to take.
But well, since 2023 a few things changed. For example, Free Threading is now around, required to design its own ABI (abi3t), and also requires to implement Py_INCREF/DECREF() as function calls.
Technically, Py_DecRef() and Py_IncRef() exist since the stable ABI version 3.2 (oldest stable ABI), whereas _Py_DecRef() and _Py_IncRef() require Python 3.10 or newer. So we can use function calls in all limited C versions.
@da-woods: Which limited C API version should start using function calls? da-woods commented on Nov 10, 2026
@da-woods: Which limited C API version should start using function calls?
I was proposing to apply this in Cython rather than in the Python headers. So if I did it, then it'd be for 3.9-3.11 (i.e. any version that Cython supports that doesn't do this automatically)
Obviously you could update the Python headers but that'd only apply when people are compiling old limited API against new Python headers, which is probably rare (I don't think cibuildwheel works this way by default for example) da-woods commented on Nov 10, 2026 One thing I spot (ignoring the possibility of a bigger change):
|
# if Py_LIMITED_API+0 >= 0x030a00A7 |
I think that should be # if Py_LIMITED_API+0 >= 0x030a0000 (and the same in DECREF) since Py_LIMITED_API is almost always set to end with 0000 and so most people defining it for 3.10 won't get the version that uses the function. (The only other exception to this in the Python source code looks to be a single function in sliceobject.h but that's explicitly documented so shouldn't change). vstinner commented on Nov 10, 2026
I think that should be # if Py_LIMITED_API+0 >= 0x030a0000 (and the same in DECREF) since Py_LIMITED_API is almost always set to end with 0000 and so most people defining it for 3.10 won't get the version that uses the function.
Oh, you're right! I wrote a way larger change to use _Py_PACK_VERSION() #161622 which replaces 0x030a00A7 with _Py_PACK_VERSION(3, 10) :-) It solves that issue.
MetadataMetadata3.14bugs and security fixesbugs and security fixes3.15bugs and security fixesbugs and security fixes3.16new features, bugs and security fixesnew features, bugs and security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DevelopmentNo branches or pull requestsIssue actions
|