Description

ZeroIntensity

opened on Jun 26, 2025

Issue body actions

Bug report

Bug description:

Currently, "pre-finalization" (callbacks executed while the interpreter is still fully intact) looks like this:

cpython/Python/pylifecycle.c

Lines 2023 to 2038 in 642e5df

	wait_for_thread_shutdown(tstate);
	// Make any remaining pending calls.
	_Py_FinishPendingCalls(tstate);
	/* The interpreter is still entirely intact at this point, and the
	* exit funcs may be relying on that. In particular, if some thread
	* or exit func is still waiting to do an import, the import machinery
	* expects Py_IsInitialized() to return true. So don't say the
	* runtime is uninitialized until after the exit funcs have run.
	* Note that Threading.py uses an exit func to do a join on all the
	* threads created thru it, so this also protects pending imports in
	* the threads created via Threading.
	*/
	_PyAtExit_Call(tstate->interp);

Threads are joined first, then pending calls are executed, and finally atexit callbacks are executed. The problem is that any of these three can create one another, such as a pending call creating a thread, or an atexit handler scheduling a pending call. The best way to demonstrate this is by creating a new thread inside of an atexit callback:

import atexit
import threading
import time


def run():
    print(24)
    time.sleep(1)
    print(42)

@atexit.register
def start_thread():
    threading.Thread(target=run).start()

On 3.13+ (and probably earlier), you'll get one of these three outputs:

1. 24 is printed.
2. Nothing is printed.
3. A fatal error occurs due to the stdout lock being unavailable.

This is because the thread created is incorrectly non-daemon, because the call to threading._shutdown (which should join it) has already happened.

@ericsnowcurrently, I discussed this with you at the PyCon sprints. I think we came to the agreement that the best way to fix this is by looping until all three of them execute no callbacks.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-136003: Execute pre-finalization callbacks in a loop #136004
• gh-136003: Skip non-daemon threads when exceptions occur during finalization #139129
• gh-136003: Close file descriptors in test #139225

Reacted by Eric Snow

Activity

1. ZeroIntensity

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   topic-C-API

   on Jun 26, 2025
2. ericsnowcurrently commented on Jul 3, 2025

   

   ericsnowcurrently

   on Jul 3, 2025

   Member

   More actions

   Yeah, this is what we discussed. Thanks for remembering and opening this issue. I have some thoughts, but first I'll take a look at your PR.
3. ZeroIntensity

   added a commit that references this issue on Sep 18, 2025

   gh-136003: Execute pre-finalization callbacks in a loop (GH-136004)

   2191497
4. ZeroIntensity

   closed this as completedon Sep 18, 2025
5. ZeroIntensity

   added a commit that references this issue on Sep 18, 2025

   gh-136003: Skip non-daemon threads when exceptions occur during final…

   3eec897
6. encukou commented on Sep 22, 2025

   

   encukou

   on Sep 22, 2025

   Member

   More actions

   The test test_pending_call_creates_thread_subinterpreter has a reference leak.
   edit: a file descriptor leak. I'll send a fix.
7. encukou

   reopened this on Sep 22, 2025
8. encukou

   added a commit that references this issue on Sep 22, 2025

   gh-136003: Close file descriptors in test (GH-139225)

   61e54e5
9. encukou

   closed this as completedon Sep 22, 2025
10. ericsnowcurrently commented on Sep 22, 2025

    

    ericsnowcurrently

    on Sep 22, 2025

    Member

    More actions

    Sorry, @ZeroIntensity, I should have warned you about closing the file descriptors.
11. ericsnowcurrently commented on Sep 22, 2025

    

    ericsnowcurrently

    on Sep 22, 2025

    Member

    More actions

    @ZeroIntensity, I do want to circle back on why os.pipe() wasn't working in the test when we tried it that way.
12. ZeroIntensity commented on Sep 22, 2025

    

    ZeroIntensity

    on Sep 22, 2025

    MemberAuthor

    More actions

    I should have warned you about closing the file descriptors.

    No worries. For the future, it would be nice if regrtest was able to detect unclosed FDs automatically without running in refleak mode.

    I do want to circle back on why os.pipe() wasn't working in the test when we tried it that way.

    I'll experiment with it more later today. Maybe my laptop has some weird setting that prevents IPC with pipe?
13. ZeroIntensity commented on Sep 22, 2025

    

    ZeroIntensity

    on Sep 22, 2025

    MemberAuthor

    More actions

    Ah, @ericsnowcurrently, we need to use pass_fds for the child process to inherit the fd:

    import subprocess
    import sys
    import os
    
    r, w = os.pipe()
    script = f"""if True:
        import os
        os.write({w}, b'a')
    """
    subprocess.run([sys.executable, '-c', script], pass_fds=[w])
    print(os.read(r, 1))

    You may have been thinking of multiprocessing.Pipe, which seems to do this for you.

    Reacted by Eric Snow
14. ericsnowcurrently commented on Sep 23, 2025

    

    ericsnowcurrently

    on Sep 23, 2025

    Member

    More actions

    For the future, it would be nice if regrtest was able to detect unclosed FDs automatically without running in refleak mode.

    That would be worth investigating. I would not be surprised if @vstinner or others have given this some consideration, but, even then, perhaps it's time to revisit it.
15. vstinner commented on Sep 23, 2025

    

    vstinner

    on Sep 23, 2025

    Member

    More actions

    Checking for FD leaks in regular regrtest mode may slow down the code. At least, it's good that we have the refleak mode to check for FD leaks!

    Reacted by Eric Snow

Issue actions