GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-96471: Add multiprocessing.Queue.shutdown() by YvesDup · Pull Request #138828 · python/cpython

Conversation

@YvesDup

@YvesDup YvesDup commented Sep 12, 2025 •
edited by gpshead
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

This adds shutdown(immediate=False) to multiprocessing.Queue and multiprocessing.JoinableQueue, with the behaviour of queue.Queue.shutdown() (gh-104750) and asyncio.Queue.shutdown() (gh-104228), applied in every process which uses the queue.

Behaviour

After shutdown():

  • put() raises queue.ShutDown. Callers blocked in put() are unblocked and raise it too.
  • get() returns the items already put, including those which a process has put but which its feeder thread has not yet written to the pipe. Once the queue is empty, get() raises queue.ShutDown, and callers blocked in get() are unblocked.
  • With immediate=True, the remaining items are discarded instead, in the pipe and in every process's feeder thread buffer. For a JoinableQueue, the unfinished task count goes down by the number of items discarded, so tasks which a worker has already taken still need their task_done().
  • shutdown() may be called more than once, and an immediate shutdown may follow a graceful one.
Implementation details from Claude

Two constraints shape it: ctypes is not available on every platform, so
no multiprocessing.Value, and sem_getvalue() is broken on macOS, so a
semaphore can only be tested for zero. The state is four SemLocks:

  • a flag for "shut down" and a flag for "shut down immediately";
  • a count of the items accepted by put() in any process and not yet
    received. This is what tells a getter that the queue is really empty,
    rather than that the pipe happens to be empty;
  • a lock. On macOS the zero test acquires and releases the semaphore, so two
    tests at the same moment can give a wrong answer. Every test whose answer
    matters is made under this lock, as is the "check the flag, count the
    item" step of put().

Nothing needs to know how many processes are waiting:

  • Getters. shutdown() sends one empty message through the pipe, which
    can never be a valid pickle. The getter which reads it sends it back for
    the next one, and raises ShutDown only if nothing is outstanding;
    otherwise it waits for the item which is still on its way.
  • Putters. shutdown() releases the queue's semaphore once if the queue
    is full. A putter which acquires it and finds the queue shut down releases
    it again, which wakes the next one.
  • Immediate. The caller discards its own buffer and what is in the pipe.
    Feeder threads in other processes discard their buffers when they see the
    flag. On Unix the caller holds the write lock while it empties the pipe
    and leaves the marker in it, so an item which a feeder thread had already
    decided to send lands behind the marker and is thrown away by the next
    getter.

A queue which never calls shutdown() pays for one lock and two semaphore
operations per put(), and one semaphore operation per get(). A
subclass which overrides put() without counting its items keeps working.

Limitations (documented)

  • A getter is woken by the getter before it. If a process is killed, or exits without flushing its feeder thread, while it is being woken, the getters behind it stay blocked.
  • If a process is killed, or exits without flushing its feeder thread, after putting items, the queue may never be considered empty and get()
    may block instead of raising ShutDown.
  • On Windows there is no write lock, so an item which another process is writing while shutdown(immediate=True) runs may still be returned by a later get().

Performance

Cost per item for a queue which is never shut down, producer process to consumer process, small items:

main this PR
Linux 1.95 µs 2.1–2.3 µs
macOS 3.5 µs 4.6 µs
  • Issue: Queue shutdown #96471

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@YvesDup
Initial commits
6a5fc24
@YvesDup
YvesDup requested a review from gpshead as a code owner September 12, 2025 11:59
@bedevere-app bedevere-app Bot added the awaiting review label Sep 12, 2025
@bedevere-app bedevere-app Bot mentioned this pull request Sep 12, 2025
Queue shutdown #96471
Open
@blurb-it
📜🤖 Added by blurb_it.
7a98c3a
@YvesDup YvesDup mentioned this pull request Sep 12, 2025
gh-96471: Add multiprocessing queue shutdown #104230
Draft
YvesDup added 2 commits September 12, 2025 14:21
@YvesDup
Fix nits
297a460
Docs is not updated, so ref to `shutdown`method is wrong.
@YvesDup
Merge branch 'main' into queue-shutdown-multiprocessing
603264c
@gvanrossum

gvanrossum commented Sep 14, 2025

Copy link
Copy Markdown
Member

Thank you! Unfortunately I don't know the multiprocessing code at all and I can't offer to review this. Maybe you can ask for help in the core development category on discuss.python.org.

YvesDup reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

EpicWink
EpicWink reviewed Sep 15, 2025
View reviewed changes
Comment thread Lib/multiprocessing/queues.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/multiprocessing/queues.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

YvesDup added 7 commits September 18, 2025 15:51
@YvesDup
Merge branch 'main' into queue-shutdown-multiprocessing
9b226c0
@YvesDup
Merge branch 'main' into queue-shutdown-multiprocessing
4ff001b
@YvesDup
Merge branch 'main' into queue-shutdown-multiprocessing
bab07dd
@YvesDup
Merge branch 'main' into queue-shutdown-multiprocessing
243068f
@YvesDup
Merge branch 'python:main' into queue-shutdown-multiprocessing
a2d25a4
@YvesDup
Merge branch 'main' into queue-shutdown-multiprocessing
148ffaf
@YvesDup
Merge branch 'main' into queue-shutdown-multiprocessing
bc57173
@YvesDup YvesDup mentioned this pull request Apr 1, 2026
Indicate that multiprocessing.Queue does not implement shutdown() #147965
Closed
YvesDup added 6 commits April 5, 2026 18:28
@YvesDup
Merge branch 'main' into queue-shutdown-multiprocessing
247f7a9
@YvesDup
Merge branch 'main' into queue-shutdown-multiprocessing
293c2e3
@YvesDup
Merge branch 'main' into queue-shutdown-multiprocessing
9b7d141
@YvesDup
Merge branch 'main' into queue-shutdown-multiprocessing
93fe64b
@YvesDup
Merge branch 'main' into queue-shutdown-multiprocessing
5981aaf
@YvesDup
Add Queue.ShutDown documentation.
7c6f69b
+ add ``Queue.shutdown` method,
+ update ``Queue.get``and ``Queue.put`` methods
+ add versionadded to ``get``and ``put``
@read-the-docs-community

read-the-docs-community Bot commented Jun 21, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

Documentation build overview

📚 cpython-previews | 🛠️ Build #35827780 | 📁 Comparing a5bf5f3 against main (8270ae5)

  🔍 Preview build  

460 files changed · + 13 added · ± 446 modified · - 1 deleted

+ Added

  • builtins/constants.html
  • builtins/exceptions.html
  • builtins/functions.html
  • builtins/index.html
  • builtins/stdtypes.html
  • builtins/threadsafety.html
  • builtins/time-complexity.html
  • c-api/pyexpat.html
  • library/asyncio-tools.html
  • library/time-complexity.html
  • and 3 more...

± Modified

  • glossary.html
  • index.html
  • license.html
  • py-modindex.html
  • _static/tachyon-example-heatmap.html
  • c-api/arg.html
  • c-api/bool.html
  • c-api/buffer.html
  • c-api/bytearray.html
  • c-api/bytes.html
  • and 436 more...

- Deleted

  • library/asyncio-policy.html
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

YvesDup added 5 commits June 21, 2026 19:44
@YvesDup
Remove inappropiate assertion
640798f
@YvesDup
Merge branch 'main' into queue-shutdown-multiprocessing
537fd85
@YvesDup
Refactor two tests
3d47640
@YvesDup
Merge branch 'main' into queue-shutdown-multiprocessing
1ad46c7
@YvesDup
Merge branch 'main' into queue-shutdown-multiprocessing
ccdf821
@YvesDup
Merge branch 'main' into queue-shutdown-multiprocessing
54d1ae9
@bedevere-app bedevere-app Bot added the type-feature A feature request or enhancement label Oct 6, 2026
YvesDup added 3 commits October 7, 2026 17:31
@YvesDup
Merge branch 'main' into queue-shutdown-multiprocessing
2859437
@YvesDup
Merge branch 'main' into queue-shutdown-multiprocessing
a20465a
@YvesDup
Merge branch 'main' into queue-shutdown-multiprocessing
b1b1748
smontanaro
smontanaro reviewed Oct 26, 2026
View reviewed changes
Comment thread Doc/library/multiprocessing.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

gpshead
gpshead reviewed Nov 11, 2026
View reviewed changes
Comment thread Doc/library/multiprocessing.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/library/multiprocessing.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/library/multiprocessing.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

gpshead and others added 5 commits November 11, 2026 14:46
@gpshead
s/3.16/next/
e540ce3
@gpshead
s/3.16/next/
f71d8bc
@gpshead
retrieved per smontanaro
2297146
@gpshead
Merge branch 'main' into queue-shutdown-multiprocessing
11d4089
@gpshead @claude
pythongh-96471: Reimplement multiprocessing.Queue.shutdown()
a5bf5f3
Replace the pending-getter and pending-putter counters with a design
which does not need to know who is waiting:

* A count of the items accepted by put() in any process and not yet
  received tells a getter whether the queue is really empty, including
  the items still in another process's feeder thread buffer.
* shutdown() writes a single empty message to the pipe.  The getter
  which reads it writes it back, so every getter is woken in its turn,
  and raises ShutDown only when nothing is outstanding.
* shutdown() releases the queue's semaphore once if the queue is full.
  A putter which acquires it and finds the queue shut down releases it
  again, waking the next one.
* shutdown(immediate=True) discards the items in the pipe and in every
  feeder thread buffer, accounting for each of them.  The unfinished
  task count of a JoinableQueue only goes down by the number of items
  discarded, as in queue.Queue.

Items put just before shutdown() are no longer lost, getters and
shutdown() itself no longer hang, and calling shutdown() again is
allowed, as in queue.Queue.

The tests are rewritten.  Most of them also run on queue.Queue, through
the 'threads' and 'manager' types, to check the behaviour is the same.

Co-Authored-By: Claude Fable 5.2 <[email protected]>
@gpshead
gpshead requested a review from AA-Turner as a code owner November 12, 2026 02:04
@gpshead gpshead self-assigned this Nov 12, 2026
@gpshead
gpshead self-requested a review November 12, 2026 02:07
@gpshead gpshead changed the title gh-96471: Add multiprocessing queue shutdown gh-96471: Add multiprocessing.Queue.shutdown() Nov 12, 2026
@gpshead

gpshead commented Nov 12, 2026

Copy link
Copy Markdown
Member

Thanks for your patience on this one.

While reviewing I found a few cases the pending-getter/putter counters could not handle, for example:

  • q.put('x'); q.shutdown(); q.get() could raise ShutDown and lose 'x';
  • a consumer already blocked in get() lost the items put just before shutdown();
  • several getters draining a shut-down queue, or shutdown(immediate=True) with a waiting getter, could hang.

These came from the design rather than from individual bugs, so rather than ask you for another round I pushed a5bf5f30ffc to your branch, which reimplements shutdown() on top of your work (thanks Claude Fable 5.2!); the PR description was updated to detail the approach.

Your docs, and NEWS entry are kept, the tests are rewritten to also cover the cases above.

Reviews welcome.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner is a code owner
@gpshead gpshead Awaiting requested review from gpshead gpshead is a code owner
3 more reviewers
@smontanaro smontanaro smontanaro left review comments
@EpicWink EpicWink EpicWink left review comments
@gvanrossum-ms gvanrossum-ms gvanrossum-ms left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@gpshead gpshead

Labels

awaiting review type-feature A feature request or enhancement

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@YvesDup @gvanrossum @gpshead @smontanaro @EpicWink @gvanrossum-ms
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.