GeezerVison is showing github.com — `tracemalloc` raw-domain allocator hook violates `PYMEM_DOMAIN_RAW` contract · Issue #155725 · python/cpython
tracemalloc raw-domain allocator hook violates PYMEM_DOMAIN_RAW contract #155725
Labels 3.15bugs and security fixesbugs and security fixes3.16new features, bugs and security fixesnew features, bugs and security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DescriptionRelated:
Bug description:
tracemalloc's raw-domain allocator hook requires an attached thread
state (PyGILState_Ensure(): tracemalloc_alloc / tracemalloc_realloc) , which violates the documented PYMEM_DOMAIN_RAW contract:
MRE (Minimal Reproducible Example):
Pure Python MRE
import ctypes
import sys
import threading
import tracemalloc
if not sys.platform.startswith("linux"):
msg = "This MRE requires Linux"
raise RuntimeError(msg)
pythonapi = ctypes.pythonapi
pythonapi.PyMem_RawMalloc.restype = ctypes.c_void_p
pythonapi.PyMem_RawMalloc.argtypes = [ctypes.c_size_t]
pythonapi.PyMem_RawFree.argtypes = [ctypes.c_void_p]
# this is only needed to creat a native thread
# without a `PyThreadState` attached to it
libc_cdll = ctypes.CDLL(None)
libc_cdll.pthread_create.restype = ctypes.c_int
libc_cdll.usleep.argtypes = [ctypes.c_uint]
# unlike CDLL, PyDLL does NOT release the GIL around the call
# this is intentional, so that the main thread holds the GIL for
# the full 5 seconds creating conditions under which the raw-allocator
# hook cannot acquire the GIL, even if it were to request it
blocking_libc = ctypes.PyDLL(None)
blocking_libc.sleep.argtypes = [ctypes.c_uint]
THREAD_FN = ctypes.CFUNCTYPE(ctypes.c_void_p, ctypes.c_void_p)
done = threading.Event()
@THREAD_FN
def native_thread_entry(_arg: ctypes.c_void_p) -> ctypes.c_void_p:
# this thread was created using `pthread_create()`, not
# `threading.Thread` it does NOT have an associated `PyThreadState`
libc_cdll.usleep(500_000)
# according to the `PyMem_SetAllocator` contract for `PYMEM_DOMAIN_RAW`
# ("a thread state is not attached when the allocator is called"),
# the call below MUST execute without blocking on the GIL
ptr = pythonapi.PyMem_RawMalloc(64)
pythonapi.PyMem_RawFree(ptr)
# if the allocator internally calls `PyGILState_Ensure()`,
# then this code will never reach this point while the GIL is busy
done.set()
def main() -> None:
# tracemalloc.start() sets a hook on `PYMEM_DOMAIN_RAW`
tracemalloc.start()
thread_id = ctypes.c_ulong()
libc_cdll.pthread_create(ctypes.byref(thread_id), None, native_thread_entry, None)
# The main thread holds the GIL continuously for 5 seconds (`PyDLL.sleep` does not
# release the GIL). If the raw-allocator hook requires the GIL, a deadlock occurs
blocking_libc.sleep(5)
# deadlock
print("DEADLOCK" if not done.is_set() else "OK")
main()
mre.c:
#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include <pthread.h>
#include <stdatomic.h>
#include <stdbool.h>
#include <unistd.h>
static atomic_bool done = false;
static void *native_thread_entry(void *arg)
{
(void)arg;
usleep(500000);
void *ptr = PyMem_RawMalloc(64);
PyMem_RawFree(ptr);
atomic_store(&done, true);
return NULL;
}
static PyObject *mre_run(PyObject *self, PyObject *args)
{
(void)self;
(void)args;
atomic_store(&done, false);
pthread_t thread;
int err = pthread_create(&thread, NULL, native_thread_entry, NULL);
if (err != 0) {
return PyErr_SetFromErrno(PyExc_OSError);
}
sleep(5);
printf("%s\n", atomic_load(&done) ? "OK" : "DEADLOCK");
pthread_detach(thread);
Py_RETURN_NONE;
}
static PyMethodDef methods[] = {
{
"run",
mre_run,
METH_NOARGS,
"Run the raw allocator MRE.",
},
{NULL, NULL, 0, NULL},
};
static struct PyModuleDef module = {
PyModuleDef_HEAD_INIT,
"mre",
NULL,
-1,
methods,
};
PyMODINIT_FUNC PyInit_mre(void)
{
return PyModule_Create(&module);
}setup.py:
from setuptools import Extension, setup
setup(
name="mre",
ext_modules=[
Extension(
"mre",
sources=["mre.c"],
extra_compile_args=["-std=c11"],
),
],
)test.py:
import tracemalloc
import mre
tracemalloc.start()
mre.run() CPython versions tested on:
3.10, 3.11, 3.12, 3.13, 3.14, 3.15
Operating systems tested on:
Linux
Linked PRs
ActivityZeroIntensity commented on Aug 13, 2026 The definition of tracemalloc's raw allocator is here:
|
static void* |
|
tracemalloc_raw_malloc(void *ctx, size_t size) |
|
{ |
|
return tracemalloc_alloc(1, 0, ctx, 1, size); |
|
} |
In tracemalloc_alloc, this sets need_gil to 1, which will in turn call PyGILState_Ensure to get an attached thread state:
|
static void* |
|
tracemalloc_alloc(int need_gil, int use_calloc, |
|
void *ctx, size_t nelem, size_t elsize) |
So, the problem isn't that tracemalloc's raw allocator needs the caller to hold the GIL (if it did, I suspect you'd see a crash at runtime when calling PyMem_RawMalloc). What's the expected behavior of the MRE? I see it prints "DEADLOCK", but the program finishes. chirizxc commented on Aug 13, 2026 ContributorAuthor More actions
What's the expected behavior of the MRE?
DEADLOCK chirizxc commented on Aug 13, 2026 ContributorAuthor More actions The PYMEM_DOMAIN_RAW contract is still violated: the documentation explicitly requires that "a thread state is not attached when the allocator is called", and calling PyGILState_Ensure() to obtain that state is a hidden, undocumented block on the GIL inside a call that is supposed to be safe without it. In the MRE, this manifests not as a "permanent deadlock" but as:
-
An unpredictable delay in PyMem_RawMalloc() / PyMem_RawFree(), lasting up to however long another thread holds the GIL (up to 5 seconds in the example), which by itself violates the contract that the allocator "can operate without an attached thread state," i.e., without blocking on the GIL at all.
-
A potential genuine deadlock in more complex scenarios, for example, if the thread holding the GIL is itself waiting on the result of an operation that depends on PyMem_RawMalloc() completing in the native thread (a classic lock-ordering inversion deadlock), or if the GIL is held indefinitely (not just 5 seconds, but say, until a long blocking operation in a C extension finishes).
ZeroIntensity commented on Aug 13, 2026 The contract doesn't specify that a thread state can't be attached inside the callback. It only specifies that the callback isn't called with one.
A potential genuine deadlock in more complex scenarios, for example, if the thread holding the GIL is itself waiting on the result of an operation that depends on PyMem_RawMalloc() completing in the native thread (a classic lock-ordering inversion deadlock), or if the GIL is held indefinitely (not just 5 seconds, but say, until a long blocking operation in a C extension finishes).
A thread shouldn't be holding the GIL at all if it's waiting on some operation (except in some very special circumstances). chirizxc commented on Aug 13, 2026 ContributorAuthor More actions It seems more accurate to say not that the contract is "violated," but that the implementation makes optional a blocking call for something the documentation explicitly names as the reason the raw domain exists in the first place - an allocator that "can operate without an attached thread state". In other words, the hook does technically work (it doesn't crash, and there's no UB), but it stops being non-blocking / GIL-independent in cases where the GIL is busy - this is a regression in expected behavior/performance, not undefined behavior or a literal violation of the letter of the spec. ZeroIntensity commented on Aug 13, 2026 Yes, but looking at the code, changing tracemalloc to not need the GIL would likely be a significant change. I guess we could try swapping out PyGILState_Ensure calls for PyMutex calls to a static lock, but I don't know what implications that would have, or if that's even possible (if tracemalloc is calling into something that needs the GIL, such as incrementing an object's reference count, then there's definitely no easy way to fix this). Reacted by Nathan Goldbaum emmatyping commented on Aug 13, 2026
changing tracemalloc to not need the GIL would likely be a significant change
FWIW I expect tracemalloc will need to be re-architected for free-threaded and subinterpreters. There are thread-safety issues (e.g. #126315) and issues with sub-interpreters (e.g. #134604). I expect to fix both tracemalloc will need a fair bit of work to be functional in those scenarios. ZeroIntensity commented on Aug 14, 2026 Honestly, I think it would be better to deprecate and remove tracemalloc. It's slow, buggy, and not very useful. Memray is better in basically every way. emmatyping commented on Aug 14, 2026 Yeah. Even with a re-architecture, unless we are okay with a lot more <unknown> frames, an attached thread state is needed to get the current thread's Python stack frames. I suppose we could queue acquiring the stack trace until the next time the interpreter re-attaches? But I'd need to think more if that can be done safely. emmatyping commented on Aug 14, 2026 I do think an allocation in another thread causing a deadlock with tracemalloc is pretty bad. We should at very minimum document that this may happen and ideally conservatively collect Python frame information for PYMEM_DOMAIN_RAW (either not require a threadstate or defer collecting the trace). ZeroIntensity commented on Aug 14, 2026 I'm still not sure I understand the scenario where this will deadlock. Why is a thread waiting on an allocation without releasing the GIL? ngoldbaum commented on Aug 14, 2026 PyMem_RawMalloc isn't documented as a possibly-blocking API. It's also not blocking in-practice except when tracemalloc is enabled. I would only think about discipline around releasing the GIL if I'm calling a possibly-blocking API.
Usually when an API is documented as being usable without an attached thread state you can also use it with a thread state. In this case that's not true if tracemalloc is enabled. ZeroIntensity commented on Aug 15, 2026
Usually when an API is documented as being usable without an attached thread state you can also use it with a thread state. In this case that's not true if tracemalloc is enabled.
I think I'm missing some context. Even if tracemalloc is enabled, PyMem_RawMalloc doesn't deadlock when called with an attached thread state. kumaraditya303 commented on Aug 26, 2026
Would that mean that PyMem_Raw* allocations aren't traced by tracemalloc?
No, they would be traced and be reflected in total traced memory, you just won't get traceback for them if raw allocator is called without an active thread state. Reacted by Gregory P. Smith ZeroIntensity commented on Aug 26, 2026 So we're just making tracemalloc even less useful than it already is? I guess it's not a huge deal, but I think this is a bad long-term solution. We shouldn't strip away features from tracemalloc when they become inconvenient for someone; let's follow process and deprecate and remove tracemalloc entirely.
But anyway, it would be ideal if PyO3 didn't rely on the allocator being non-blocking. Tools like Memray are also completely capable of acquiring the GIL in their raw allocator hook. We can't just pretend that this has always been a part of the contract for PYMEM_DOMAIN_RAW. ngoldbaum commented on Aug 26, 2026 In case it isn't clear, this latest round of comments comes from gh-155725, where Kumar tried to use RawMalloc in the ssl module. kumaraditya303 commented on Aug 26, 2026
So we're just making tracemalloc even less useful than it already is? I guess it's not a huge deal, but I think this is a bad long-term solution. We shouldn't strip away features from tracemalloc when they become inconvenient for someone; let's follow process and deprecate and remove tracemalloc entirely.
I've added tracebacks for raw allocations back, an active thread state was required earlier because tracemalloc stores file names as python objects, now it is stored as a raw string so thread state is not needed. With that, there is no behavior change on my latest version of PR.
FWIW I don't care too much about memory tracking using tracemalloc, but it is blocker for using raw allocator in ssl module so I want to fix it rather than going through the route of deprecation and wait 2 releases. gpshead commented on Aug 30, 2026 Note that the "pure Python minimal reproducer" is invalid. ctypes CFUNCTYPE callbacks call PyGILState_Ensure() themselves. kumaraditya303 commented on Aug 31, 2026 I've fixed this in main and it is backported to 3.15. Reacted by Emma Smith, chiri and Petr Vaganov
MetadataMetadata3.15bugs and security fixesbugs and security fixes3.16new features, bugs and security fixesnew features, bugs and security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DevelopmentNo branches or pull requestsIssue actions
|