GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-94182: run the PidfdChildWatcher on the running loop by graingert · Pull Request #94184 · python/cpython

Conversation

@graingert

@graingert graingert commented Jun 23, 2022 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor
  • Issue: PidfdChildWatcher runs callbacks on the main thread eventloop and not the currently running loop #94182

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@graingert
run the PidfdChildWatcher on the running loop
e49365f
@bedevere-bot bedevere-bot added the awaiting review label Jun 23, 2022
@graingert
enable GenericWatcherTests and add a test for pidfd watcher run in a …
49cfa81
…thread
graingert
graingert commented Jun 23, 2022
View reviewed changes
Comment thread Lib/test/test_asyncio/test_subprocess.py Outdated


class GenericWatcherTests:
class GenericWatcherTests(test_utils.TestCase):

@graingert graingert Jun 23, 2022

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this test wasn't enabled?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@graingert graingert Jul 19, 2022

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I split this fix out, so this PR is easier to review: #95009

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@graingert
fix authspec typo
3a3d5e2
graingert
graingert commented Jun 23, 2022
View reviewed changes
Comment thread Lib/test/test_asyncio/test_subprocess.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@graingert
Update Lib/test/test_asyncio/test_subprocess.py
60dafa1
graingert
graingert commented Jun 23, 2022
View reviewed changes
Comment thread Lib/test/test_asyncio/test_subprocess.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@graingert
Update Lib/test/test_asyncio/test_subprocess.py
16f1c4e
graingert
graingert commented Jun 23, 2022
View reviewed changes
Comment thread Lib/test/test_asyncio/test_subprocess.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@graingert
Update Lib/test/test_asyncio/test_subprocess.py
c8ae89d
graingert
graingert commented Jun 23, 2022
View reviewed changes
Comment thread Lib/test/test_asyncio/test_subprocess.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

graingert added 5 commits June 24, 2022 00:21
@graingert
Update Lib/test/test_asyncio/test_subprocess.py
8229efc
@graingert
Merge branch 'main' into run-pidfdchildwatcher-on-the-running-loop
6473022
@graingert
AbstractChildWatcher is called as a context manager
02190f2
test_create_subprocess_fails_with_inactive_watcher was only setting
the is_active() method on the context manager not the context
manager result
@graingert
move GenericWatcherTests into the other unix tests
9607d8e
@graingert
Merge branch 'main' into run-pidfdchildwatcher-on-the-running-loop
dad94d9
graingert
graingert commented Jun 24, 2022
View reviewed changes
Comment thread Lib/test/test_asyncio/test_subprocess.py Outdated

async def execute():
watcher = mock.create_authspec(asyncio.AbstractChildWatcher)
watcher.is_active.return_value = False

@graingert graingert Jun 24, 2022 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was patching the wrong attribute - asyncio calls asyncio.get_child_watcher().__enter__().is_active()

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
graingert and others added 2 commits June 24, 2022 09:10
@graingert
check that asyncio.set_event_loop() was not called
6e56155
otherwise it invalidates test_create_subprocess_with_pidfd
@blurb-it
📜🤖 Added by blurb_it.
1037078
@graingert
graingert marked this pull request as ready for review June 24, 2022 08:50
@graingert
graingert requested review from 1st1 and asvetlov as code owners June 24, 2022 08:50
graingert added 3 commits June 24, 2022 11:34
@graingert
Merge branch 'main' into run-pidfdchildwatcher-on-the-running-loop
4e69faf
@graingert
Merge branch 'main' into run-pidfdchildwatcher-on-the-running-loop
740e0c2
@graingert
Merge branch 'main' into run-pidfdchildwatcher-on-the-running-loop
7721ea1
@graingert

graingert commented Jul 6, 2022

Copy link
Copy Markdown
Contributor Author

@asvetlov can I get a review on this please?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

graingert added 3 commits July 7, 2022 10:06
@graingert
Merge branch 'main' of github.com:python/cpython into run-pidfdchildw…
f4f6fc5
…atcher-on-the-running-loop
@graingert
Merge branch 'main' into run-pidfdchildwatcher-on-the-running-loop
cad5231
@graingert
asyncio.run sets up the policy system asyncio.Runner with a loop_fact…
5647edb
…ory does not
@gvanrossum
gvanrossum self-requested a review as a code owner October 7, 2022 18:55
gvanrossum
gvanrossum approved these changes Oct 7, 2022
View reviewed changes

@gvanrossum gvanrossum left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. We'll probably merge this ASAP.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/test/test_asyncio/test_subprocess.py


@functools.cache
def _has_pidfd_support():

@gvanrossum gvanrossum Oct 7, 2022

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This same function is also introduced by @kumaraditya303's PR GH-98024 (which uses PidfdChildWatcher when supported). I propose to refactor things so that everything uses Kumar's version. We can do that in Kumar's PR once this PR is merged.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/asyncio/unix_events.py
def _do_wait(self, pid):
pidfd, callback, args = self._callbacks.pop(pid)
self._loop._remove_reader(pidfd)
def _do_wait(self, pid, pidfd, callback, args):

@gvanrossum gvanrossum Oct 7, 2022

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious why we can't pass the loop as an argument as well? (Though maybe that creates an unhealthy self-reference to the loop.)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting core review labels Oct 7, 2022
@gvanrossum

gvanrossum commented Oct 7, 2022

Copy link
Copy Markdown
Member

(Waiting for @1st1 review and passing tests.)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gvanrossum gvanrossum mentioned this pull request Oct 7, 2022
GH-98023: Change default child watcher to PidfdChildWatcher on supported systems #98024
Merged
@gvanrossum
Merge branch 'main' into run-pidfdchildwatcher-on-the-running-loop
a4fac11
1st1
1st1 approved these changes Oct 7, 2022
View reviewed changes
@gvanrossum

gvanrossum commented Oct 7, 2022

Copy link
Copy Markdown
Member

Not sure what's wrong with the docs build.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

1st1
1st1 reviewed Oct 7, 2022
View reviewed changes
Comment thread Misc/NEWS.d/next/Library/2022-06-24-08-49-47.gh-issue-94182.Wknau0.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Yury Selivanov
Update Misc/NEWS.d/next/Library/2022-06-24-08-49-47.gh-issue-94182.Wk…
fc2e400
…nau0.rst
@1st1

1st1 commented Oct 7, 2022

Copy link
Copy Markdown
Member

Not sure what's wrong with the docs build.

I've fixed it.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

1st1
1st1 reviewed Oct 7, 2022
View reviewed changes
Comment thread Misc/NEWS.d/next/Library/2022-06-24-08-49-47.gh-issue-94182.Wknau0.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Yury Selivanov
Update Misc/NEWS.d/next/Library/2022-06-24-08-49-47.gh-issue-94182.Wk…
65a2db6
…nau0.rst
@1st1

1st1 commented Oct 7, 2022

Copy link
Copy Markdown
Member

I've fixed it.

Reverted the "fix" because apparently it was caused by an unrelated commit in main.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@JelleZijlstra JelleZijlstra self-assigned this Oct 7, 2022
@JelleZijlstra

JelleZijlstra commented Oct 7, 2022

Copy link
Copy Markdown
Member

I can take care of getting CI green and hitting the merge button here.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@1st1

1st1 commented Oct 7, 2022

Copy link
Copy Markdown
Member

I can take care of getting CI green and hitting the merge button here.

Please do. You might need to rebase this PR to the latest main.

JelleZijlstra reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@JelleZijlstra
Merge branch 'main' into run-pidfdchildwatcher-on-the-running-loop
d25f852
@gvanrossum

gvanrossum commented Oct 8, 2022

Copy link
Copy Markdown
Member

The macOS build hit some GitHub Actions limit. I'm going to merge without waiting for it.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gvanrossum
gvanrossum merged commit 3d8b224 into python:main Oct 8, 2022
@bedevere-bot bedevere-bot removed the awaiting merge label Oct 8, 2022
carljm added a commit to carljm/cpython that referenced this pull request Oct 8, 2022
@carljm
Merge branch 'main' into typewatch
d67d6fd
* main:
  pythongh-68686: Retire eptag ptag scripts (python#98064)
  pythongh-97922: Run the GC only on eval breaker (python#97920)
  GitHub Workflows security hardening (python#96492)
  Add `@ezio-melotti` as codeowner for `.github/`. (python#98079)
  pythongh-97913 Docs: Add walrus operator to the index (python#97921)
  [doc] Fix broken links to C extensions accelerating stdlib modules (python#96914)
  pythongh-97822: Fix http.server documentation reference to test() function (python#98027)
  pythongh-91052: Add PyDict_Unwatch for unwatching a dictionary (python#98055)
  pythonGH-98023: Change default child watcher to PidfdChildWatcher on supported systems (python#98024)
  pythonGH-94182: Run the PidfdChildWatcher on the running loop (python#94184)
carljm added a commit to carljm/cpython that referenced this pull request Oct 9, 2022
@carljm
Merge branch 'main' into urllibnonet
132c32e
* main: (5519 commits)
  Minor edits to the Descriptor HowTo Guide (pythonGH-24901)
  Fix link to Lifecycle of a Pull Request in CONTRIBUTING (python#98102)
  pythonGH-94597: deprecate `SafeChildWatcher`, `FastChildWatcher` and `MultiLoopChildWatcher` child watchers  (python#98089)
  Auto-cancel old builds when new commit pushed to branch (python#98009)
  pythongh-95011: Migrate syslog module to Argument Clinic (pythonGH-95012)
  pythongh-68686: Retire eptag ptag scripts (python#98064)
  pythongh-97922: Run the GC only on eval breaker (python#97920)
  GitHub Workflows security hardening (python#96492)
  Add `@ezio-melotti` as codeowner for `.github/`. (python#98079)
  pythongh-97913 Docs: Add walrus operator to the index (python#97921)
  [doc] Fix broken links to C extensions accelerating stdlib modules (python#96914)
  pythongh-97822: Fix http.server documentation reference to test() function (python#98027)
  pythongh-91052: Add PyDict_Unwatch for unwatching a dictionary (python#98055)
  pythonGH-98023: Change default child watcher to PidfdChildWatcher on supported systems (python#98024)
  pythonGH-94182: Run the PidfdChildWatcher on the running loop (python#94184)
  pythongh-92886: make test_ast pass with -O (assertions off) (pythonGH-98058)
  pythongh-92886: make test_coroutines pass with -O (assertions off) (pythonGH-98060)
  pythongh-57179: Add note on symlinks for os.walk (python#94799)
  pythongh-94808: Fix regex on exotic platforms (python#98036)
  pythongh-90085: Remove vestigial -t and -c timeit options (python#94941)
  ...
mpage pushed a commit to mpage/cpython that referenced this pull request Oct 11, 2022
@graingert Yury Selivanov
@JelleZijlstra @mpage
pythonGH-94182: Run the PidfdChildWatcher on the running loop (python…
6cd489d
…#94184)

There is no reason for this watcher to be attached to any particular loop.
This should make it safe to use regardless of the lifetime of the event loop running in the main thread
(relative to other loops).

Co-authored-by: Yury Selivanov <[email protected]>
Co-authored-by: Jelle Zijlstra <[email protected]>
bdraco added a commit to bdraco/home-assistant that referenced this pull request Feb 12, 2023
@bdraco
Use PidfdChildWatcher by default when available
29f4931
This is a backport from cpython 3.12

https://docs.python.org/3/library/asyncio-policy.html
> PidfdChildWatcher is a “Goldilocks” child watcher implementation. It doesn’t require signals or threads, doesn’t interfere with any processes launched outside the event loop, and scales linearly with the number of subprocesses launched by the event loop. The main disadvantage is that pidfds are specific to Linux, and only work on recent (5.3+) kernels.

python/cpython#98024
There are some additional fixes in cpython 3.12
in python/cpython#94184
when there is no event loop running in the main thread
but this is not a problem we have
@bdraco bdraco mentioned this pull request Feb 12, 2023
Use PidfdChildWatcher by default when available home-assistant/core#87951
Merged
2 of 19 tasks
balloob pushed a commit to home-assistant/core that referenced this pull request Feb 12, 2023
@bdraco
Use PidfdChildWatcher by default when available (#87951)
d1e1734
This is a backport from cpython 3.12

https://docs.python.org/3/library/asyncio-policy.html
> PidfdChildWatcher is a “Goldilocks” child watcher implementation. It doesn’t require signals or threads, doesn’t interfere with any processes launched outside the event loop, and scales linearly with the number of subprocesses launched by the event loop. The main disadvantage is that pidfds are specific to Linux, and only work on recent (5.3+) kernels.

python/cpython#98024
There are some additional fixes in cpython 3.12
in python/cpython#94184
when there is no event loop running in the main thread
but this is not a problem we have
AlePerla pushed a commit to AlePerla/homeassistant_core that referenced this pull request Feb 17, 2023
@bdraco @AlePerla
Use PidfdChildWatcher by default when available (home-assistant#87951)
638c431
This is a backport from cpython 3.12

https://docs.python.org/3/library/asyncio-policy.html
> PidfdChildWatcher is a “Goldilocks” child watcher implementation. It doesn’t require signals or threads, doesn’t interfere with any processes launched outside the event loop, and scales linearly with the number of subprocesses launched by the event loop. The main disadvantage is that pidfds are specific to Linux, and only work on recent (5.3+) kernels.

python/cpython#98024
There are some additional fixes in cpython 3.12
in python/cpython#94184
when there is no event loop running in the main thread
but this is not a problem we have
@graingert
graingert deleted the run-pidfdchildwatcher-on-the-running-loop branch October 11, 2023 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@1st1 1st1 1st1 approved these changes
@gvanrossum gvanrossum gvanrossum approved these changes
@asvetlov asvetlov Awaiting requested review from asvetlov asvetlov is a code owner
+1 more reviewer
@agronholm agronholm agronholm requested changes
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@JelleZijlstra JelleZijlstra

Labels

3.10 (EOL) end of life 3.11 only security fixes 3.12 only security fixes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@graingert @gvanrossum @1st1 @JelleZijlstra @agronholm @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.