GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used · Issue #90622 · python/cpython

concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used #90622

New issue

Copy link

New issue

Copy link

Closed

concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used#90622

#92497 (+3)

Copy link

Assignees

Labels

3.10 (EOL)end of lifeend of life3.11only security fixesonly security fixes3.9 (EOL)end of lifeend of lifeperformancePerformance or resource usagePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

Description

yilei

mannequin

opened

on Jan 21, 2022

Issue body actions

BPO	46464
Nosy	@gpshead, @pitrou, @methane, @yilei
PRs	gh-90622: Prevent concurrent.futures.process deadlock due to fork() after spawning a thread. #30847

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gpshead'
closed_at = None
created_at = <Date 2022-01-21.23:32:53.148>
labels = ['3.11', 'library', '3.9', '3.10', 'performance']
title = 'concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used'
updated_at = <Date 2022-01-25.11:06:46.406>
user = 'https://github.com/yilei'

bugs.python.org fields:

activity = <Date 2022-01-25.11:06:46.406>
actor = 'pitrou'
assignee = 'gregory.p.smith'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2022-01-21.23:32:53.148>
creator = 'yilei'
dependencies = []
files = []
hgrepos = []
issue_num = 46464
keywords = ['patch', '3.9regression']
message_count = 7.0
messages = ['411208', '411213', '411451', '411543', '411544', '411545', '411574']
nosy_count = 4.0
nosy_names = ['gregory.p.smith', 'pitrou', 'methane', 'yilei']
pr_nums = ['30847']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue46464'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Activity

yileimannequin

on Jan 21, 2022

MannequinAuthor

More actions

When Python is built and linked with tcmalloc, using ProcessPoolExecutor may deadlock. Here is a reproducible example:

$ cat t.py
from concurrent import futures
import sys
def work(iteration, item):
  sys.stdout.write(f'working: iteration={iteration}, item={item}\n')
  sys.stdout.flush()
for i in range(0, 10000):
    with futures.ProcessPoolExecutor(max_workers=2) as executor:
        executor.submit(work, i, 1)
        executor.submit(work, i, 2)

$ python t.py
working: iteration=0, item=1
working: iteration=0, item=2
working: iteration=1, item=1
working: iteration=1, item=2
...
working: iteration=3631, item=1
working: iteration=3631, item=2
<hang here>

The child process fails to finish. It's more likely to reproduce when the system is busy.

With some bisect search internally, this commit 1ac6e37 "triggered" the deadlock threshold with tcmalloc.

yilei
added
3.9 (EOL)end of lifeend of life

3.10 (EOL)end of lifeend of life

3.11only security fixesonly security fixes

stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory
on Jan 21, 2022
gpshead commented on Jan 21, 2022
gpshead
on Jan 21, 2022
Member
More actions
For context, the fundamental problem is that os.fork() is unsafe to use in any process with threads. concurrent/futures/process.py violates this. So long as its design involves a thread, it can never be guaranteed not to deadlock.

POSIX forbids execution of most code after fork() has been called. Returning to the CPython interpreter after os.fork() in the child process is never safe unless the parent process had no threads at fork time.

The change that triggered the issue moved from doing all of the os.fork() calls for the workers up front *before the thread was spawned* to doing them on demand after the thread was running.

The bugfix for this is simply a rollback to revert the bpo-39207 change. It was a performance enhancement, but it causes a new deadlock in code that didn't already have one when thread tuned malloc implementations are used.

The only way to safely launch worker processes on demand is to spawn a worker launcher process spawned prior to any thread creation that remains idle, with a sole job of spawn new worker processes for us. That sounds complicated. That'd be a feature. Lets go with the bugfix first.
gpshead
self-assigned this
on Jan 21, 2022
gpshead
added
performancePerformance or resource usagePerformance or resource usage
on Jan 21, 2022
gpshead
self-assigned this
on Jan 21, 2022
gpshead
added
performancePerformance or resource usagePerformance or resource usage
on Jan 21, 2022
gpshead commented on Jan 24, 2022
gpshead
on Jan 24, 2022
Member
More actions
https://bugs.python.org/issue44733 for 3.11 attempts to build upon the dynamic spawning ability and will need reverting unless a non-thread+fork implementation is provided.
14 remaining items
gpshead
added 3 commits that reference this issue on May 8, 2022
gh-90622: Do not spawn ProcessPool workers on demand via fork method. (…
ebb37fc
[3.10] pythongh-90622: Do not spawn ProcessPool workers on demand via…
5595472
[3.10] gh-90622: Do not spawn ProcessPool workers on demand via fork …
b795376
miss-islington
added a commit that references this issue on May 8, 2022
[3.10] pythongh-90622: Do not spawn ProcessPool workers on demand via…
ecfa914
gpshead
added a commit that references this issue on May 8, 2022
gh-90622: Do not spawn ProcessPool workers on demand via fork method. (…
4270b79
gpshead
linked a pull request that will close this issue[3.9] [3.10] gh-90622: Do not spawn ProcessPool workers on demand via fork method. (GH-91598) (GH-92497) #92499 on May 8, 2022
gpshead
reopened this on May 8, 2022
gpshead
added a commit that references this issue on May 8, 2022
[3.10] gh-90622: Do not spawn ProcessPool workers on demand via fork …
8e523c1
gpshead
closed this as completed on May 8, 2022
gpshead
linked a pull request that will close this issue[3.11] gh-90622: Do not spawn ProcessPool workers on demand via fork method. (GH-91598) #92495 on May 8, 2022
gpshead
linked a pull request that will close this issue[3.10] gh-90622: Do not spawn ProcessPool workers on demand via the "fork" spawn method (GH-91598) #92497 on May 8, 2022
gpshead commented on May 8, 2022
gpshead
on May 8, 2022
Member
More actions
Note that there is still one deadlock possibility that I left as a comment in the code. If the "fork" mp_context spawn method is used and one of the worker processes dies, a new one will be forked after multiprocessing has already started its executor manager thread. mp_context "forkserver" and "spawn" methods avoid that.
hello-adam
added a commit that references this issue on Jun 2, 2022
[3.10] pythongh-90622: Do not spawn ProcessPool workers on demand via…
78e11e4
ay-bh
added 2 commits that reference this issue on Oct 26, 2026
judges/screening: harden parallel prepare after review
47de453
judges/screening: build prepare windows in parallel worker processes (#9
28139c9

Metadata

Assignees

gpshead

Labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used #90622

Description

Activity

yilei commented on Jan 21, 2022

gpshead commented on Jan 21, 2022

gpshead commented on Jan 24, 2022

14 remaining items

gpshead commented on May 8, 2022

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions