GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-83069: Use efficient event-driven `subprocess.Popen.wait()` on Linux / macOS / BSD by giampaolo · Pull Request #144047 · python/cpython

Conversation

@giampaolo

@giampaolo giampaolo commented Jan 19, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

This patch replaces the busy-loop mechanism used by subprocess.Popen.wait() on Linux, BSD, and macOS when a timeout is specified:

  • Linux >= 5.3: uses os.pidfd_open() + select.poll()
  • macOS, BSD: uses select.kqueue() + KQ_FILTER_PROC + KQ_NOTE_EXIT

The kernel wakes us up exactly when the PID terminates or the timeout expires. If the fast-wait mechanisms is not available or fails (e.g. "too many open files"), the function falls back to the traditional busy loop. A similar approach was recently implemented in psutil (see giampaolo/psutil#2706). While working on that change, I noticed that the subprocess module used essentially the same busy-loop strategy, which motivated this proposal.

EDIT: I wrote a blog post about this: https://gmpy.dev/blog/2026/event-driven-process-waiting

  • Issue: Popen should use pidfd_open to implement a non-busy wait #83069

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

soltanoff, 5j9, saisua, and kaizen-cmd reacted with heart emoji mbeijen, takluyver, and kaizen-cmd reacted with rocket emoji
All reactions
  • ❤️ 4 reactions
  • 🚀 3 reactions
giampaolo added 30 commits January 18, 2026 20:51
@giampaolo
Use pidfd_open() and save the FD as an attribute
288e4db
@giampaolo
Create _busy_wait() and move code in there
6e494ca
@giampaolo
Move existing code in new _blocking_wait() method
aaf193c
@giampaolo
Use _wait_pidfd()
7cbb0ad
@giampaolo
Use pidfd_open() not at class level but at method level
d9760c3
@giampaolo
Add kqueue() implementation for macOS and BSD
5fd8eec
@giampaolo
Add docstrings
100b111
@giampaolo
Be conservative and check for specific error codes
74ac2f4
@giampaolo
Document possible failures
fc0cfd6
@giampaolo
Add missing import
f282459
@giampaolo
Write test for pidfd_open() failing
6125976
@giampaolo
Write test case for pidfd_open() / kqueue failing. Assert fallback is…
41dc6c0
… used.
@giampaolo
Write test case for kqueue failing. Assert fallback is used.
73ba380
@giampaolo
Move tests in their own class
3c8e603
@giampaolo
Add test for terminated PID race
932ae58
@giampaolo
Add test_kqueue_race()
bb5080a
@giampaolo
Add test_kqueue_control_error
f067073
@giampaolo
Add docstring
4d96c2f
@giampaolo
Guard against possible slow test
fe05acc
@giampaolo
Timeout: use math.ceil to avoid truncation
adb444e
@giampaolo
Remove unused exception var
4ec17c1
@giampaolo
Timeout: use math.ceil to avoid truncation
e807ba9
@giampaolo
Replace _can_use_kqueue() -> _CAN_USE_KQUEUE
1ddc52b
@giampaolo
Shorten code
645ef6c
@giampaolo
Use waitpid() + WNOHANG even if process exited to avoid rare race
6ee771b
@giampaolo
Revert prev change
61c6b99
@giampaolo
Use ceil(timeout) to avoid truncation
527646d
@giampaolo
Remove check for timeout < 0 + rm test case which didn't make sense.
0a8a1b2
@giampaolo
Don't use _blocking_wait() as it has a while loop
73b97dc
@giampaolo
Add docstring
2d3c3f7
@giampaolo giampaolo mentioned this pull request Jan 20, 2026
[Linux, macOS, BSD] speedup Process.wait() giampaolo/psutil#2705
Closed
vstinner
vstinner reviewed Jan 21, 2026
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice change! I like it. Here is a first review.

cc @gpshead

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/subprocess.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/subprocess.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/library/subprocess.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_subprocess.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_subprocess.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_subprocess.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_subprocess.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/whatsnew/3.15.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

giampaolo added 5 commits January 21, 2026 16:57
@giampaolo
Small refact
525c047
@giampaolo
Address review comments (bare assert, don't use ceil())
2b931b8
@giampaolo
Address review comments (use test class VARIABLES)
d1c6e91
@giampaolo
Address review comments (add periods in doc)
3e9d303
@giampaolo
Merge branch 'main' into subprocess-fast-wait
a44b2d7
@giampaolo

giampaolo commented Jan 21, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review @vstinner. I addressed your comments.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner
vstinner reviewed Jan 21, 2026
View reviewed changes
Comment thread Lib/subprocess.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

giampaolo added 4 commits January 21, 2026 17:59
@giampaolo
Remove weird unicode char from doc added by accident
b4f6020
@giampaolo
Don't import select on Windows
4e17856
@giampaolo
Make kq.control() invocation clearer by passing kwargs
6f6600d
@giampaolo
Revert previous commit
37288c6
vstinner
vstinner approved these changes Jan 22, 2026
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for the updates!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/test/test_subprocess.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_subprocess.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Jan 22, 2026
giampaolo added 2 commits January 22, 2026 12:22
@giampaolo
Remove bare assert statements
300692e
@giampaolo
Merge branch 'main' into subprocess-fast-wait
7bee1e9
@giampaolo

giampaolo commented Jan 22, 2026

Copy link
Copy Markdown
Contributor Author

Whops! I fixed the remaining bare assert statements.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Jan 22, 2026

Copy link
Copy Markdown
Member

@gpshead: Do you want to double check the change?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@giampaolo giampaolo mentioned this pull request Jan 24, 2026
Extend selectors module to support platform-specific events #144211
Open
@giampaolo

giampaolo commented Jan 24, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

In the hope of adding BSD / macOS support for asyncio in the future, I have opened #144211.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner
vstinner merged commit 9b154ab into python:main Jan 28, 2026
47 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the awaiting merge label Jan 28, 2026
@vstinner

vstinner commented Jan 28, 2026

Copy link
Copy Markdown
Member

Merged, thank you for this nice optimization!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@giampaolo
giampaolo deleted the subprocess-fast-wait branch January 28, 2026 18:19
thunder-coding pushed a commit to thunder-coding/cpython that referenced this pull request Feb 15, 2026
@giampaolo @thunder-coding
pythongh-83069: Use efficient event-driven subprocess.Popen.wait() …
368281f
…on Linux / macOS / BSD (python#144047)
@giampaolo giampaolo mentioned this pull request Feb 20, 2026
Move inactive experts to emeritus status python/devguide#1635
Merged
This was referenced Jul 28, 2026
3.15 regression: Popen.wait(timeout=float("inf")) raises TypeError: timeout must be a real number or None on macOS/BSD (new kqueue wait path) #154836
Closed
gh-154836: Fix Popen.wait() with very large timeouts on the pidfd/kqueue wait paths #154837
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner vstinner approved these changes
@gpshead gpshead Awaiting requested review from gpshead gpshead is a code owner
@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner is a code owner
@benjaminp benjaminp Awaiting requested review from benjaminp

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@giampaolo @vstinner
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.