GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Hard to debug subprocess crashes when using concurrent.futures.ProcessPoolExecutor · Issue #139462 · python/cpython

Hard to debug subprocess crashes when using concurrent.futures.ProcessPoolExecutor #139462

New issue
Copy link
New issue
Copy link
Closed
Closed
Hard to debug subprocess crashes when using concurrent.futures.ProcessPoolExecutor#139462
Copy link
Assignees
gpshead
Labels
stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytopic-multiprocessingtype-featureA feature request or enhancementA feature request or enhancement

Description

@jberg5
jberg5
opened on Sep 30, 2025
Issue body actions

Feature or enhancement

Proposal:

Background

When using a concurrent.futures.ProcessPoolExecutor, if a subprocess crashes/terminates, the library gives you an exception like:

concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

This doesn't tell you two very important bits of information:

  • Which subprocess terminated
  • What exit code it returned

When running a nontrivial application, odds are that you'll run into a BrokenProcessPool, whether because a subprocess segfaulted, OOMed, or something else, and having some basic information about what exactly died and why would be incredibly helpful for correlating crashes with logs/telemetry/etc and aiding further investigation.

As a possible but suboptimal workaround, you can inspect the private _processes list upon failure to try and figure out what went wrong, e.g.

from concurrent.futures import as_completed, ProcessPoolExecutor
from concurrent.futures.process import BrokenProcessPool
import ctypes
import os
import traceback

def worker_ok() -> str:
    print(f"worker_ok {os.getpid()=}", flush=True)
    return "ok"

def worker_segfault():
    print(f"worker_segfault {os.getpid()=}", flush=True)
    ctypes.string_at(0)

if __name__ == "__main__":
    with ProcessPoolExecutor(max_workers=2) as executor:
        futures = [
            executor.submit(worker_ok),
            executor.submit(worker_segfault),
        ]
        for future in as_completed(futures):
            try:
                future.result()
            except BrokenProcessPool as e:
                # Need to look at private _processes to figure out what died and why
                for pid, p in executor._processes.items():
                    if p.exitcode:
                        print(f"Process {pid} failed with exitcode {p.exitcode}")
                traceback.print_exception(e)
worker_ok os.getpid()=37334
worker_segfault os.getpid()=37335
Process 37334 failed with exitcode -15
Process 37335 failed with exitcode -11
Traceback (most recent call last):
  File "/users/me/projects/repo/example.py", line 26, in <module>
    future.result()
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Accessing _processes works, but it means reading a nonpublic part of the api, and writing slightly nontrivial exception handling code. It would be much better if the ProcessPoolExecutor automatically reported this kind of crash info when raising a BrokenProcessPool exception.

Proposal

There is already a concept in the codebase of a bpe.__cause__, which contains the cause of the BrokenProcessPool if one is known. When _ExecutorManagerThread._terminate_broken is called, it should loop through the processes and report any nonzero exit codes.

I've implemented the above approach, and running the example script above with my change, I now see:

worker_ok os.getpid()=37202
worker_segfault os.getpid()=37202
Lib.concurrent.futures.process._RemoteTraceback: Process 37202 terminated abruptly with exit code -11

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/jberg/Projects/cpython/example.py", line 26, in <module>
    future.result()
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
Lib.concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Note the Lib.concurrent.futures.process._RemoteTraceback above - imo this is a much more informative traceback.

If this looks good to everyone, I'll raise a PR shortly.

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs

  • gh-139462: Make the ProcessPoolExecutor BrokenProcessPool exception report which child process terminated #139486
Reacted by Gregory P. Smith

Activity

  1. jberg5
    added
    type-featureA feature request or enhancementA feature request or enhancement
    on Sep 30, 2025
  2. ZeroIntensity
    added
    stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory
    topic-multiprocessing
    on Oct 1, 2025
  3. ZeroIntensity commented on Oct 1, 2025

    @ZeroIntensity
    ZeroIntensity
    on Oct 1, 2025
    Member
    More actions

    cc @gpshead

    I think this is a reasonable addition, depending on the complexity of the change.

    Reacted by Gregory P. Smith
  4. jberg5 commented on Oct 1, 2025

    @jberg5
    jberg5
    on Oct 1, 2025
    ContributorAuthor
    More actions

    Thanks @ZeroIntensity - it's a pretty small diff: https://github.com/python/cpython/pull/139486/files

    Let me know what you think!

  5. gpshead
    self-assigned this
    on Oct 1, 2025
  6. gpshead
    added a commit that references this issue on Nov 11, 2025

    gh-139462: Make the ProcessPoolExecutor BrokenProcessPool exception r…

    9e7340c
  7. gpshead commented on Nov 11, 2025

    @gpshead
    gpshead
    on Nov 11, 2025
    Member
    More actions

    merged, thanks!

    Reacted by J Berg
  8. gpshead
    closed this as completedon Nov 11, 2025
  9. StanFromIreland
    added a commit that references this issue on Dec 6, 2025

    pythongh-139462: Make the ProcessPoolExecutor BrokenProcessPool excep…

    76a1b47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

  • @gpshead
    gpshead

Labels

stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytopic-multiprocessingtype-featureA feature request or enhancementA feature request or enhancement

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

    You are visitor 01402503 — GeezerVison does not store the pages you look up.
    Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.