GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-134634: Use a pipe instead of a queue for Pool's change notifier by yonatan-genai · Pull Request #148050 · python/cpython

Conversation

@yonatan-genai

@yonatan-genai yonatan-genai commented Apr 3, 2026 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

Summary

Pool.__init__ used a multiprocessing.SimpleQueue for its internal _change_notifier, a mechanism that wakes up the worker handler thread when the pool state changes (close, terminate, cache empty). SimpleQueue creates two Lock objects backed by POSIX named semaphores via sem_open(), which requires /dev/shm to be mounted as tmpfs.

The change notifier never crosses a process boundary. It is only accessed by threads within the parent process (the worker handler, result handler, and main thread). Named semaphores were never needed here.

This replaces the SimpleQueue with a _ChangeNotifier class backed by multiprocessing.connection.Pipe (which wraps os.pipe()). It provides the same interface (put, get, empty, _reader) without requiring sem_open().

Impact

Both multiprocessing.Pool and multiprocessing.pool.ThreadPool fail on platforms without /dev/shm, even though ThreadPool only uses threads and never needs inter-process synchronization. Affected platforms include AWS Lambda (no /dev/shm), Android and iOS (sem_open() unavailable), and Docker containers with restricted shared memory.

Changes

  • Lib/multiprocessing/pool.py: Add _ChangeNotifier class, replace the one line in Pool.__init__ that created the SimpleQueue
  • Lib/test/_test_multiprocessing.py: Three new tests — notifier type verification + end-to-end pool.map, notification drain under load, worker process recovery after SIGKILL
  • Misc/NEWS.d/next/Library/: NEWS entry

Testing

Full test_multiprocessing_spawn suite: 447 tests passed, 0 failures. Reference leak check passed.

  • Issue: Some multiprocessing APIs should be possible to import or use even when multiprocessing itself isn't available #134634

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@yonatan-genai
gh-134634: Use a pipe instead of a queue for Pool's change notifier
e136267
Pool.__init__ used a multiprocessing.SimpleQueue for its internal
_change_notifier. SimpleQueue creates Lock objects backed by POSIX
named semaphores (sem_open), which requires /dev/shm. The change
notifier never crosses a process boundary, so named semaphores were
never needed.

Replace it with a _ChangeNotifier class backed by
multiprocessing.connection.Pipe (os.pipe). Same interface, no
sem_open dependency.

<claude>
@bedevere-app bedevere-app Bot added the awaiting review label Apr 3, 2026
@yonatan-genai
yonatan-genai requested a review from gpshead as a code owner April 3, 2026 18:23
@python-cla-bot

python-cla-bot Bot commented Apr 3, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot mentioned this pull request Apr 3, 2026
Some multiprocessing APIs should be possible to import or use even when multiprocessing itself isn't available #134634
Open
yonatan-genai and others added 2 commits April 3, 2026 21:25
@yonatan-genai
Merge branch 'main' into gh-134634-pool-pipe-notifier
ab7fd0d
@yonatan-genai
Add missing fork deprecation warning decorator to new tests
3995086
The new tests need @warnings_helper.ignore_fork_in_thread_deprecation_warnings()
to avoid DeprecationWarning when creating Pool with the fork start method
in a multi-threaded test process (Python 3.14+ strictness).

<claude>
@yonatan-genai

yonatan-genai commented Apr 3, 2026

Copy link
Copy Markdown
Author

The macOS / build and test (macos-26) failure is unrelated to this change — it's test_perf_profiler.TestPerfTrampoline.test_trampoline_works_with_forks failing with a non-zero child exit code. This test has a known flakiness issue related to fork child finalization (see gh-144766). I'll submit a separate fix for it.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@yonatan-genai yonatan-genai mentioned this pull request Apr 3, 2026
gh-144766: Fix flaky test_trampoline_works_with_forks #148056
Closed
@gpshead

gpshead commented Apr 3, 2026

Copy link
Copy Markdown
Member

https://devguide.python.org/getting-started/generative-ai/

I see no evidence of a person being involved here and thus doubt the CLA was signed by a human with legal authority to do so.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gpshead gpshead closed this Apr 3, 2026
@yonatan-genai

yonatan-genai commented Apr 4, 2026 via email

Copy link
Copy Markdown
Author
My username has been this way for several years… before Claude Code, OpenClaw, etc. I signed the CLA.
…
On Sat, 4 Apr 2026 at 7:23 Gregory P. Smith ***@***.***> wrote: *gpshead* left a comment (python/cpython#148050) <#148050 (comment)> https://devguide.python.org/getting-started/generative-ai/ I see no evidence of a person being involved here and thus doubt the CLA was signed by a human with legal authority to do so. — Reply to this email directly, view it on GitHub <#148050 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ARWBSXZ3SZD4PFGDOAM7UGL4UBBWFAVCNFSM6AAAAACXMA72ZCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DCOBVGU4TANBWGU> . You are receiving this because you authored the thread.Message ID: ***@***.***>
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@yonatan-genai

yonatan-genai commented Apr 4, 2026

Copy link
Copy Markdown
Author

I’d appreciate it if you could consider reopening…

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@serhiy-storchaka
serhiy-storchaka self-requested a review April 4, 2026 13:11
@yonatan-genai

yonatan-genai commented Apr 17, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Author

@serhiy-storchaka @gpshead do you guys intend to review this? 🙏

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gpshead gpshead Awaiting requested review from gpshead gpshead is a code owner
@serhiy-storchaka serhiy-storchaka Awaiting requested review from serhiy-storchaka

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@yonatan-genai @gpshead
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.