GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-88050: Fix asyncio subprocess kill process cleanly when process is blocked by kumaraditya303 · Pull Request #32073 · python/cpython

Conversation

@kumaraditya303

@kumaraditya303 kumaraditya303 commented Mar 23, 2022 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

This PR fixes the issue by calling the waiters after all the callbacks are executed so that no callbacks are scheduled after the method returns.

  • Issue: Cannot cleanly kill a subprocess using high-level asyncio APIs #88050

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot bedevere-bot added the awaiting review label Mar 23, 2022
@the-knights-who-say-ni the-knights-who-say-ni added the CLA signed label Mar 23, 2022
@kumaraditya303
bpo-43884: Fix asyncio subprocess kill process cleanly when process i…
485b26b
…s blocked
@kumaraditya303
kumaraditya303 force-pushed the fix-asyncio-subprocess branch from 98f282f to 485b26b Compare March 23, 2022 09:36
@kumaraditya303
kumaraditya303 marked this pull request as ready for review March 23, 2022 10:26
@kumaraditya303
kumaraditya303 requested review from 1st1 and asvetlov as code owners March 23, 2022 10:26
@asvetlov asvetlov changed the title bpo-43884: Fix asyncio subprocess kill process cleanly when process i… bpo-43884: Fix asyncio subprocess kill process cleanly when process is blocked Mar 23, 2022
@asvetlov

asvetlov commented Mar 23, 2022

Copy link
Copy Markdown
Contributor

What's wrong with Windows?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

kumaraditya303 added 2 commits March 24, 2022 08:03
@kumaraditya303
Merge branch 'main' of https://github.com/python/cpython into fix-asy…
7c10817
…ncio-subprocess
@kumaraditya303
rework
0b87f8d
@kumaraditya303 kumaraditya303 closed this Mar 28, 2022
@kumaraditya303 kumaraditya303 reopened this Mar 28, 2022
kumaraditya303 added 5 commits March 28, 2022 10:19
@kumaraditya303
start from scratch
2bd08cf
@kumaraditya303
Merge branch 'main' of https://github.com/python/cpython into fix-asy…
6afa6df
…ncio-subprocess
@kumaraditya303
rework
a2a3a1d
@kumaraditya303
remove script
e325b6c
@kumaraditya303
Merge branch 'main' into fix-asyncio-subprocess
6a37501
@kumaraditya303 kumaraditya303 changed the title bpo-43884: Fix asyncio subprocess kill process cleanly when process is blocked gh-88050: Fix asyncio subprocess kill process cleanly when process is blocked Apr 17, 2022
@kumaraditya303
Merge branch 'main' into fix-asyncio-subprocess
5fa81d6
@kumaraditya303 kumaraditya303 closed this May 4, 2022
@kumaraditya303 kumaraditya303 reopened this May 4, 2022
vstinner
vstinner reviewed May 4, 2022
View reviewed changes
Comment thread Lib/asyncio/base_subprocess.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

kumaraditya303 added 2 commits May 27, 2022 20:05
@kumaraditya303
Merge branch 'main' into fix-asyncio-subprocess
74e42a7
@kumaraditya303
another try
7946267
@rabraham rabraham mannequin mentioned this pull request May 31, 2022
Cannot cleanly kill a subprocess using high-level asyncio APIs #88050
Closed
kumaraditya303 and others added 3 commits July 7, 2022 19:09
@kumaraditya303
Merge branch 'main' into fix-asyncio-subprocess
4f18b7d
@kumaraditya303
skip unless sleep
d896e0b
@blurb-it
📜🤖 Added by blurb_it.
b5de374
@kumaraditya303 kumaraditya303 added the topic-asyncio label Jul 8, 2022
kumaraditya303 added 2 commits October 5, 2022 12:18
@kumaraditya303
Merge branch 'main' into fix-asyncio-subprocess
4ac1184
@kumaraditya303
fixup test as Guido suggested
69a9fda
@kumaraditya303
kumaraditya303 requested a review from gvanrossum October 5, 2022 07:02
kumaraditya303 added 3 commits October 5, 2022 12:55
@kumaraditya303
Merge branch 'main' into fix-asyncio-subprocess
678cc76
@kumaraditya303
fix killing on windows
ba0382d
@kumaraditya303
Merge branch 'fix-asyncio-subprocess' of https://github.com/kumaradit…
90e7ac5
…ya303/cpython into fix-asyncio-subprocess
gvanrossum
gvanrossum approved these changes Oct 5, 2022
View reviewed changes

@gvanrossum gvanrossum left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, thanks for the summary! This looks like a clear improvement so let's merge it.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

kumaraditya303 reacted with heart emoji
All reactions
  • ❤️ 1 reaction
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting review labels Oct 5, 2022
@gvanrossum
gvanrossum merged commit 7015e13 into python:main Oct 5, 2022
@miss-islington

miss-islington commented Oct 5, 2022

Copy link
Copy Markdown
Contributor

Thanks @kumaraditya303 for the PR, and @gvanrossum for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10, 3.11.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed awaiting merge needs backport to 3.11 only security fixes labels Oct 5, 2022
@bedevere-bot

bedevere-bot commented Oct 5, 2022

Copy link
Copy Markdown

GH-97915 is a backport of this pull request to the 3.11 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Oct 5, 2022
@kumaraditya303 @miss-islington
pythongh-88050: Fix asyncio subprocess to kill process cleanly when p…
494942e
…rocess is blocked (pythonGH-32073)

(cherry picked from commit 7015e13)

Co-authored-by: Kumar Aditya <[email protected]>
@bedevere-bot

bedevere-bot commented Oct 5, 2022

Copy link
Copy Markdown

GH-97916 is a backport of this pull request to the 3.10 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the needs backport to 3.10 label Oct 5, 2022
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Oct 5, 2022
@kumaraditya303 @miss-islington
pythongh-88050: Fix asyncio subprocess to kill process cleanly when p…
ea840e3
…rocess is blocked (pythonGH-32073)

(cherry picked from commit 7015e13)

Co-authored-by: Kumar Aditya <[email protected]>
@kumaraditya303
kumaraditya303 deleted the fix-asyncio-subprocess branch October 5, 2022 17:16
@kumaraditya303 kumaraditya303 mentioned this pull request Oct 5, 2022
bpo-43884: Fix asyncio subprocess kill process cleanly when process is blocked #31611
Closed
miss-islington added a commit that referenced this pull request Oct 5, 2022
@miss-islington @kumaraditya303
gh-88050: Fix asyncio subprocess to kill process cleanly when process…
aec1333
… is blocked (GH-32073)

(cherry picked from commit 7015e13)

Co-authored-by: Kumar Aditya <[email protected]>
carljm added a commit to carljm/cpython that referenced this pull request Oct 6, 2022
@carljm
Merge branch 'main' into dictwatch
4bb5d9d
* main: (66 commits)
  pythongh-65961: Raise `DeprecationWarning` when `__package__` differs from `__spec__.parent` (python#97879)
  docs(typing): add "see PEP 675" to LiteralString (python#97926)
  pythongh-97850: Remove all known instances of module_repr() (python#97876)
  I changed my surname early this year (python#96671)
  pythongh-93738: Documentation C syntax (:c:type:<C type> -> :c:expr:<C type>) (python#97768)
  pythongh-91539: improve performance of get_proxies_environment  (python#91566)
  build(deps): bump actions/stale from 5 to 6 (python#97701)
  pythonGH-95172 Make the same version `versionadded` oneline (python#95172)
  pythongh-88050: Fix asyncio subprocess to kill process cleanly when process is blocked (python#32073)
  pythongh-93738: Documentation C syntax (Function glob patterns -> literal markup) (python#97774)
  pythongh-93357: Port test cases to IsolatedAsyncioTestCase, part 2 (python#97896)
  pythongh-95196: Disable incorrect pickling of the C implemented classmethod descriptors (pythonGH-96383)
  pythongh-97758: Fix a crash in getpath_joinpath() called without arguments (pythonGH-97759)
  pythongh-74696: Pass root_dir to custom archivers which support it (pythonGH-94251)
  pythongh-97661: Improve accuracy of sqlite3.Cursor.fetchone docs (python#97662)
  pythongh-87092: bring compiler code closer to a preprocessing-opt-assembler organisation (pythonGH-97644)
  pythonGH-96704: Add {Task,Handle}.get_context(), use it in call_exception_handler() (python#96756)
  pythongh-93738: Documentation C syntax (:c:type:`PyTypeObject*` -> :c:expr:`PyTypeObject*`) (python#97778)
  pythongh-97825: fix AttributeError when calling subprocess.check_output(input=None) with encoding or errors args (python#97826)
  Add re.VERBOSE flag documentation example (python#97678)
  ...
carljm added a commit to carljm/cpython that referenced this pull request Oct 8, 2022
@carljm
Merge branch 'main' into typewatch
1af2a98
* main: (53 commits)
  pythongh-94808: Coverage: Test that maximum indentation level is handled (python#95926)
  pythonGH-88050: fix race in closing subprocess pipe in asyncio  (python#97951)
  pythongh-93738: Disallow pre-v3 syntax in the C domain (python#97962)
  pythongh-95986: Fix the example using match keyword (python#95989)
  pythongh-97897: Prevent os.mkfifo and os.mknod segfaults with macOS 13 SDK (pythonGH-97944)
  pythongh-94808: Cover `PyUnicode_Count` in CAPI (python#96929)
  pythongh-94808: Cover `PyObject_PyBytes` case with custom `__bytes__` method (python#96610)
  pythongh-95691: Doc BufferedWriter and BufferedReader (python#95703)
  pythonGH-88968: Add notes about socket ownership transfers (python#97936)
  pythongh-96865: [Enum] fix Flag to use CONFORM boundary (pythonGH-97528)
  pythongh-65961: Raise `DeprecationWarning` when `__package__` differs from `__spec__.parent` (python#97879)
  docs(typing): add "see PEP 675" to LiteralString (python#97926)
  pythongh-97850: Remove all known instances of module_repr() (python#97876)
  I changed my surname early this year (python#96671)
  pythongh-93738: Documentation C syntax (:c:type:<C type> -> :c:expr:<C type>) (python#97768)
  pythongh-91539: improve performance of get_proxies_environment  (python#91566)
  build(deps): bump actions/stale from 5 to 6 (python#97701)
  pythonGH-95172 Make the same version `versionadded` oneline (python#95172)
  pythongh-88050: Fix asyncio subprocess to kill process cleanly when process is blocked (python#32073)
  pythongh-93738: Documentation C syntax (Function glob patterns -> literal markup) (python#97774)
  ...
mpage pushed a commit to mpage/cpython that referenced this pull request Oct 11, 2022
@kumaraditya303 @mpage
pythongh-88050: Fix asyncio subprocess to kill process cleanly when p…
129a24f
…rocess is blocked (python#32073)
@vstinner

vstinner commented Oct 18, 2022

Copy link
Copy Markdown
Member

AMD64 Ubuntu Shared 3.x buildbot started to fail recently with Out-of-Memory (OOM) errors: Python test processes are killed with SIGKILL by the Linux kernel.

David Bolen, owner of the buildbot worker, wrote that there's currently an awful lot (340) of processes like: (...)/python -c import time; time.sleep(100000000). I guess that these processes come from test_kill_issue43884() of test_asyncio.test_subprocess (Lib/test/test_asyncio/test_subprocess.py). He added: "Looks like they started accumulating sporadically on Oct 2".

Maybe this change supposed to kill asyncio processes more cleanly is now making the situation worse? Or maybe it's a different regression?

I ran the following command in 3 terminals on Linux to stress test the test:

./python -m test -m test_kill_issue43884 -j40 test_asyncio test_asyncio  test_asyncio  test_asyncio  test_asyncio  test_asyncio  test_asyncio  test_asyncio  test_asyncio  test_asyncio  test_asyncio test_asyncio  test_asyncio  test_asyncio  test_asyncio  test_asyncio  test_asyncio  test_asyncio  test_asyncio  test_asyncio  test_asyncio test_asyncio  test_asyncio  test_asyncio  test_asyncio  test_asyncio  test_asyncio  test_asyncio  test_asyncio  test_asyncio  test_asyncio test_asyncio  test_asyncio  test_asyncio  test_asyncio  test_asyncio  test_asyncio  test_asyncio  test_asyncio  test_asyncio

While tests are running, I see Python processes running time.sleep(100000000). But once tests complete, I don't see these processes anymore.

Maybe the issue was fixed in the meanwhile? I don't know.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@db3l

db3l commented Oct 18, 2022

Copy link
Copy Markdown
Contributor

So, there are already a new set of hung processes on my buildbot since I killed everything a short while ago. There's an active 3.11 build going on, but the asyncio test already completed, so it sounds like those processes should be gone. There are two clusters (3 processes) about 16 minutes apart, so I'm guessing one from the last build and one from this one. test_asyncio did run near the end of the last build and near the beginning of the current one.

As soon as I have some free cycles, I'll try some manual tests on the buildbot to see if I can reproduce. But in the meantime I'll monitor to ensure they don't accumulate enough to interfere again with subsequent builds.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner vstinner mentioned this pull request Oct 18, 2022
AMD64 Ubuntu Shared 3.x: python processes killed with SIGKILL by Linux Out-of-Memory (OOM), maybe related to test_asyncio #98407
Closed
@vstinner

vstinner commented Oct 18, 2022

Copy link
Copy Markdown
Member

I created issue #98407: "AMD64 Ubuntu Shared 3.x: python processes killed with SIGKILL by Linux Out-of-Memory (OOM), maybe related to test_asyncio".

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@benfdking benfdking mentioned this pull request Nov 2, 2025
ci: fix benchmark running SQLMesh/sqlmesh#5570
Merged
@tapetersen tapetersen mentioned this pull request Jun 10, 2026
asyncio proc.kill() and proc.wait() are counter intuitive #119710
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner vstinner left review comments
@gvanrossum gvanrossum gvanrossum approved these changes
@1st1 1st1 Awaiting requested review from 1st1 1st1 is a code owner
@asvetlov asvetlov Awaiting requested review from asvetlov asvetlov is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

topic-asyncio

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

9 participants

@kumaraditya303 @asvetlov @bedevere-bot @gvanrossum @miss-islington @vstinner @db3l @the-knights-who-say-ni @ezio-melotti
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.