Description

tom-pytel

opened on Mar 3, 2025

Issue body actions

Crash report

What happened?

When cycling a lot of threads and modifying objects which use PyMem_FreeDelayed() in those threads the queues of memory blocks to free build up without ever being freed until the process runs out of memory, unless something else causes them to be freed. For example in the reproducer below commenting in the gc.collect() or the for loop will cause the delayed blocks to be freed correctly. Note that this is an extreme case.

Reproducer:

import gc
import threading


def fupmem(b, l):
    b.wait()
    l *= 2


def check(funcs, *args):
    barrier = threading.Barrier(len(funcs))
    thrds = []

    for func in funcs:
        thrd = threading.Thread(target=func, args=(barrier, *args))

        thrds.append(thrd)
        thrd.start()

    for thrd in thrds:
        thrd.join()


if __name__ == "__main__":
    count = 0

    while True:
        print(count := count + 1)

        check([fupmem] * 10, [None] * 256)

        # gc.collect()

        # for i in range(1000):
        #     l = []
        #     del l

CPython versions tested on:

3.14

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.14.0a5+ experimental free-threading build (heads/main:b3c18bfd828, Mar 3 2025, 09:13:46) [GCC 11.4.0]

Linked PRs

• gh-130794: Process interpreter QSBR queue in _PyMem_AbandonDelayed. #130808
• [3.13] gh-130794: Process interpreter QSBR queue in _PyMem_AbandonDelayed. (gh-130808) #130857

Activity

1. tom-pytel

   added

   type-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump

   on Mar 3, 2025
2. tom-pytel commented on Mar 3, 2025

   

   tom-pytel

   on Mar 3, 2025

   ContributorAuthor

   More actions

   ping @colesbury
3. picnixz

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   topic-free-threading

   on Mar 3, 2025
4. colesbury commented on Mar 3, 2025

   

   colesbury

   on Mar 3, 2025

   Contributor

   More actions

   Thanks for the bug report. Typically, each thread periodically processes their queue of memory blocks to be freed. This happens every 254 calls to _PyMem_FreeDelayed(). The problems is that:

   1. Each thread only makes one call to _PyMem_FreeDelayed(), so they don't call _PyMem_ProcessDelayed.
   2. The thread then exits and "abandons" it's queue to the shared per-interpreter queue
   3. No thread processes the shared per-interpreter queue, because that happens in _PyMem_ProcessDelayed

   The fix is relatively straightforward: when the thread exits and merges its queue into the shared per-interpreter queue, it should also take the opportunity to process the shared queue.
5. colesbury

   added 5 commits that reference this issue on Mar 3, 2025

   pythongh-130794: Process interpreter QSBR queue in _PyMem_AbandonDela…

   15bd961

   Merge branch 'main' into pythongh-130794-process-queue

   d344c73

   gh-130794: Process interpreter QSBR queue in _PyMem_AbandonDelayed. (g…

   2f6e0e9

   [3.13] pythongh-130794: Process interpreter QSBR queue in _PyMem_Aban…

   baba972

   [3.13] gh-130794: Process interpreter QSBR queue in _PyMem_AbandonDel…

   0752275
6. kumaraditya303 commented on Mar 11, 2025

   

   kumaraditya303

   on Mar 11, 2025

   Contributor

   More actions

   Closing as it was fixed by #130808 and backported.
7. kumaraditya303

   closed this as completedon Mar 11, 2025

Issue actions