Description

rostan-t

opened on Jan 22, 2025

Issue body actions

Crash report

What happened?

_PyTraceMalloc_Fini is called before _PyImport_Fini in _Py_Finalize. This can lead to crashes when using tracemalloc if calls to PyTraceMalloc_Untrack are performed during finalization of module (e.g in the destructor of some global variable).

Commit 6b47499 moved the call to TABLES_LOCK before the check tracemalloc_config.tracing so now the issue whether or not tracemalloc is started.

The following extension module reproduces the issue:

#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include <numpy/arrayobject.h>

static PyObject* arr = NULL;

static struct PyModuleDef repromodule = {
    PyModuleDef_HEAD_INIT,
    "repro",
    NULL,
    -1,
    NULL
};

PyMODINIT_FUNC
PyInit_repro(void)
{
    PyObject* m;

    m = PyModule_Create(&repromodule);
    if (!m)
        return NULL;

    import_array();

    arr = PyArray_SimpleNew(1, (npy_intp[]){0}, NPY_DOUBLE);
    if (!arr || PyModule_AddObject(m, "arr", arr) < 0) {
        Py_XDECREF(arr);
        Py_DECREF(m);
        return NULL;
    }

    return m;
}

$ python -c 'import repro'

It declares a global NumPy array and array_dealloc calls PyTraceMalloc_Untrack, creating a segfault.

For context, I initially ran into the issue in a pybind11 extension declaring functions with numpy array as defaults arguments.

CPython versions tested on:

CPython main branch, 3.13

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.13.1+ (heads/3.13:a3797492179, Jan 22 2025, 11:46:15) [GCC 11.4.0]

Linked PRs

• gh-129185: Fix crash during interpreter finalization in PyTraceMalloc_Untrack #129188
• gh-129185: Fix PyTraceMalloc_Untrack() at Python exit #129191
• [3.13] gh-129185: Fix PyTraceMalloc_Untrack() at Python exit (#129191) #129217
• gh-129185: Remove internal TRACE_RAW_MALLOC macro #129218
• [3.12] gh-129185: Fix PyTraceMalloc_Untrack() at Python exit (#129191) (#129217) #129221
• gh-129185: Use PyMutex in tracemalloc #129246
• gh-129185: Simplify PyTraceMalloc_Track() #129256

Activity

1. rostan-t

   added

   type-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump

   on Jan 22, 2025
2. picnixz

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   on Jan 22, 2025
3. picnixz commented on Jan 22, 2025

   

   picnixz

   on Jan 22, 2025

   Member

   More actions

   Btw, PyModule_AddObject steals a reference, so you might want to incref arr before since you're storing it in a global variable but I don't know whether this would solve the issue. Now, can you actually produce a reproducer that does not depend on numpy so that we can possibly add a regression test?

   cc @vstinner
4. vstinner

   added 2 commits that reference this issue on Jan 22, 2025

   pythongh-129185: Fix PyTraceMalloc_Untrack() at Python exit

   7522e4b

   pythongh-129185: Fix PyTraceMalloc_Untrack() at Python exit

   a1ea2fe
5. vstinner commented on Jan 22, 2025

   

   vstinner

   on Jan 22, 2025

   Member

   More actions

   I wrote a different fix with a test: PR gh-129191.
6. vstinner

   added 5 commits that reference this issue on Jan 23, 2025

   gh-129185: Fix PyTraceMalloc_Untrack() at Python exit (#129191)

   46c7e13

   pythongh-129185: Fix PyTraceMalloc_Untrack() at Python exit (python#1…

   62eebd3

   pythongh-129185: Remove internal TRACE_RAW_MALLOC macro

   f2957f1

   gh-129185: Remove internal TRACE_RAW_MALLOC macro (#129218)

   e579cdb

   [3.13] gh-129185: Fix PyTraceMalloc_Untrack() at Python exit (#129191) (

   e3b3e01
7. vstinner commented on Jan 23, 2025

   

   vstinner

   on Jan 23, 2025

   Member

   More actions

   Thanks @rostan-t for the bug report and for the initial fix! It's now fixed in 3.13 and main branches, and the 3.12 fix is incoming.
8. vstinner

   closed this as completedon Jan 23, 2025
9. vstinner

   added 7 commits that reference this issue on Jan 23, 2025

   [3.12] gh-129185: Fix PyTraceMalloc_Untrack() at Python exit (#129191) (

   bb7c54d

   pythongh-129185: Use PyMutex in tracemalloc

   7ae64db

   pythongh-129185: Use PyMutex in tracemalloc

   32053f8

   gh-129185: Use PyMutex in tracemalloc (#129246)

   c005ea4

   pythongh-129185: Simplify PyTraceMalloc_Track()

   0f4024a

   pythongh-129185: Simplify PyTraceMalloc_Track()

   abfbaa0

   gh-129185: Simplify PyTraceMalloc_Track() (#129256)

   fc6bc1e

Issue actions