GeezerVison is showing github.com — multi-threading + fork warning when threads are stopped before fork · Issue #137109 · python/cpython
multi-threading + fork warning when threads are stopped before fork #137109DescriptionBug report
Bug description:
If all threads are stopped before a fork, and then restarted in the parent (and optionally the child) process, the new multi-threading with fork warning is given
import os
import threading
stop = False
def target():
while not stop:
pass
thread = threading.Thread(target=target)
def stop_thread():
global stop
stop = True
thread.join()
def restart_thread():
global thread, stop
stop = False
(thread := threading.Thread(target=target)).start()
os.register_at_fork(before=stop_thread, after_in_child=restart_thread, after_in_parent=restart_thread)
thread.start()
if os.fork() == 0:
os._exit(0)If the thread is not restarted in the parent, no warning is given. I believe warn_about_fork_with_threads should be refactored so that a warning is given to the user only if there are threads detected after the call to PyOS_BeforeFork. If it is not possible to give a warning there, as stated in the comments, perhaps one can just do the thread count check there and take note of the result, then warn after PyOS_AfterFork_Parent, when safe to do so.
CPython versions tested on:
3.13
Operating systems tested on:
macOS
Linked PRs
Activitypicnixz commented on Jul 25, 2025 Let's remove the callback in the child. The order of execution would be:
- Start thread.
- Request to fork.
- Join thread.
- Do the fork.
- Restart the thread in the parent.
I think the problem is with the fact that thread from the parent process is not entirely joined. And so it's effectively the case that we're still multi-threaded even though it's not the case maybe.
cc @ZeroIntensity as someone who has better threading knowledge. P403n1x87 commented on Jul 25, 2025 ContributorAuthor More actions It looks to me that warn_about_fork_with_threads is warning on the new thread in the parent, because it is restarted by PyOS_AfterFork_Parent, before warn_about_fork_with_threads is called. So the warning is about the new thread running in the parent after the fork, rather than the original thread (that has actually been stopped and joined before the call to the fork syscall). colesbury commented on Jul 28, 2025 Yeah, it'd be better to call the after-fork-parent calls after we check and warn about threads. That would avoid the spurious warning. Reacted by Gregory P. Smith gpshead commented on Nov 12, 2025 Code paths that warn_about_fork_with_thread can take depend on PyOS_AfterFork_Parent() having been called already to avoid deadlock. gpshead commented on Nov 12, 2025 A workaround for this could be for it to be called by PyOS_AfterFork_Parent() itself right before the run_at_forkers(...) call. gpshead commented on Nov 12, 2025 But that'd still leave deadlocks possible as the Python world it may need to call into could rely on other afterfork parent calls that have been registered. gpshead commented on Nov 12, 2025 Okay I think I have a solution for the most common case. The warn function can be split up into the OS specific APIs to get the number of threads (which can be called before AfterFork_Parent) and the Python APIs that must wait until after. This won't fix it in situations where to lack platform specific thread detection code or when those APIs fail, but will in the most widely used scenarios. PR coming after testing. gpshead commented on Nov 12, 2025 re: race conditions, it may be OS specific as to if the thread could still show up in OS APIs as existing in the process after the internal C pthread_join() that the Python thread.join() did has returned. I'd like to think not, but that is up to the OS as there are no posix APIs to get a list of threads for a process so it is undefined behavior and up to the kernel implementation as to if it has finished tearing down that state and cleaning it up synchronously or not. gpshead commented on Nov 24, 2025 The PRs merged should've fixed this for the most common case.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|