Description

gpshead

opened on Dec 14, 2022

Issue body actions

os.fork() in a multi-threaded application is a likely source of deadlocks on many platforms. We should raise a warning when people call os.fork() from a process that we know has other threads running.

This came from discussion in https://discuss.python.org/t/switching-default-multiprocessing-context-to-spawn-on-posix-as-well/21868/, though I believe many of us have pondered doing it in the past.

Linked PRs

• gh-100228: Warn from os.fork() if other threads exist. #100229
• gh-100228: Document the os.fork threads DeprecationWarning. #109767
• [3.12] gh-100228: Document the os.fork threads DeprecationWarning. (GH-109767) #109773

Activity

1. gpshead

   added

   type-featureA feature request or enhancementA feature request or enhancement

   on Dec 14, 2022
2. gpshead

   added a commit that references this issue on Dec 29, 2022

   gh-100228: Warn from os.fork() if other threads exist. (#100229)

   894f2c3
3. hauntsaninja commented on Jan 29, 2023

   

   hauntsaninja

   on Jan 29, 2023

   Contributor

   More actions

   Thanks for adding this!

   Reacted by Gregory P. Smith and Mufan Qiu
4. hauntsaninja

   closed this as completedon Jan 29, 2023
5. gpshead

   added

   3.12only security fixesonly security fixes

   on Feb 9, 2023
6. pitrou commented on Feb 11, 2023

   

   pitrou

   on Feb 11, 2023

   Member

   More actions

   Why a DeprecationWarning? This suggests that something is deprecated while it is not... ResourceWarning would sound more appropriate IMHO.
7. gpshead commented on Feb 12, 2023

   

   gpshead

   on Feb 12, 2023

   MemberAuthor

   More actions

   Yeah, that question came up on the PR: https://github.com/python/cpython/pull/100229/files#r1048391023

   This is one of those cases where we worry about being too noisy if we use a warning that people see outside of their test environment (where deprecation warnings tend to be turned on, they're off by default otherwise).

   Lots of code in the world for better or worse "works fine" calling fork() from a threaded application because the software stacks that so many things build on top of don't care: glibc and it's malloc being the prime example. POSIX says it is never okay to do it, but because it worked on old glibc's and glibc malloc: that implementation gets maintained in such a super conservative manner that it does not as a general rule cause problems with threads+fork itself. Other things do, other libc implementations, many other malloc implementations (which often for much better performance specifically in the face of threading), other libraries code may use, etc.

   So we want it to be a soft nudge to start with. We can "increase" the verbosity to a different warning type in the future. Or if this proves too disruptive as is, reconsider. Unfortunately we don't have an existing Warning class for this kind of situation.

   A perhaps more appropriately named subclass (so that it is hidden by default) of DeprecationWarning could be created so that it at least doesn't sound like we're removing the ability for people to try and shoot themselves in the foot in the future?

   I'm expecting we'll hear from users about the noise level by the time we're in the beta phase.
8. gpshead commented on Sep 12, 2023

   

   gpshead

   on Sep 12, 2023

   MemberAuthor

   More actions

   Relevant discussion about this warning in 3.12 release candidate end user testing: https://discuss.python.org/t/concerns-regarding-deprecation-of-fork-with-alive-threads/33555
9. gpshead

   added a commit that references this issue on Sep 23, 2023

   gh-100228: Document the os.fork threads DeprecationWarning. (#109767)

   5e7ea95
10. miss-islington

    added a commit that references this issue on Sep 23, 2023

    pythongh-100228: Document the os.fork threads DeprecationWarning. (py…

    951212b
11. Yhg1s

    added a commit that references this issue on Sep 24, 2023

    [3.12] gh-100228: Document the os.fork threads DeprecationWarning. (G…

    62df559
12. csm10495

    added a commit that references this issue on Sep 28, 2023

    pythongh-100228: Document the os.fork threads DeprecationWarning. (py…

    f1cffd7
13. Glyphack

    added a commit that references this issue on Sep 2, 2024

    pythongh-100228: Document the os.fork threads DeprecationWarning. (py…

    d11105b
14. gpauloski

    added a commit that references this issue on Sep 5, 2024

    Use spawn in process-pool executor fixture

    2a6060c

Issue actions