GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-96522: Fix deadlock in pty.spawn by zhangyoufu · Pull Request #96639 · python/cpython

Conversation

@zhangyoufu

@zhangyoufu zhangyoufu commented Sep 7, 2022 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor
  • Issue: pty.spawn deadlock #96522

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@bedevere-bot bedevere-bot added the awaiting review label Sep 7, 2022
@bedevere-bot

bedevere-bot commented Sep 7, 2022

Copy link
Copy Markdown

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ghost

ghost commented Sep 7, 2022 •
edited by ghost
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.
CLA signed
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
ambv
ambv reviewed Oct 7, 2022
View reviewed changes
Comment thread Lib/pty.py

def _read(fd):
"""Default read function."""
return os.read(fd, 1024)

@ambv ambv Oct 7, 2022

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be okay to replace high_waterlevel and this 1024 above here with a PIPE_BUF global set to 4096 so it's clear where it comes from? This also makes the deadlock in line 164 (master_read) more apparent.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@zhangyoufu zhangyoufu Oct 10, 2022 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_copy accept caller provided master_read (may be something other than pty._read). It does not support "read at most n bytes" unfortunately. The actual buffer size may exceed 4096 here. high_waterlevel serves as a threshold for whether we should continue reading into our buffer, it can not limit maximum buffer size. A name like PIPE_BUF is not appropriate per my understanding.

I do want to increase the read size from 1024 to 4096, but I'm afraid of unexpected changes to application behavior. As long as we didn't pass lots of data via pty, 1024 should serve us well. So I didn't change it.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
ambv
ambv reviewed Oct 7, 2022
View reviewed changes
Comment thread Lib/pty.py Outdated
fds = [master_fd, STDIN_FILENO]
while fds:
rfds, _wfds, _xfds = select(fds, [], [])
os.set_blocking(master_fd, False)

@ambv ambv Oct 7, 2022

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you mind commenting why this is necessary?

And also, even though _copy is a private function here, it is for sure used by end users in their programs (per Hyrum's Law) so it would be good to unset non-blocking when exiting _copy so that backward's compatibility is preserved.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@zhangyoufu zhangyoufu Oct 10, 2022

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed as requested. PTAL

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@zhangyoufu
Fix deadlock in pty.spawn
e819a1c 
Fix gh-96522
@bedevere-bot bedevere-bot mentioned this pull request Feb 15, 2023
pty.spawn deadlock #96522
Closed
@zhangyoufu
zhangyoufu requested a review from ambv February 15, 2023 06:37
@zhangyoufu

zhangyoufu commented Apr 20, 2023

Copy link
Copy Markdown
Contributor Author

@ambv PTAL, thanks.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ambv ambv added the needs backport to 3.11 only security fixes label May 19, 2023
@ambv
ambv enabled auto-merge (squash) May 19, 2023 13:22
@ambv
ambv merged commit 9c5aa89 into python:main May 19, 2023
@miss-islington

miss-islington commented May 19, 2023

Copy link
Copy Markdown
Contributor

Thanks @zhangyoufu for the PR, and @ambv for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the awaiting review label May 19, 2023
@miss-islington

miss-islington commented May 19, 2023

Copy link
Copy Markdown
Contributor

Sorry @zhangyoufu and @ambv, I had trouble checking out the 3.11 backport branch.
Please retry by removing and re-adding the "needs backport to 3.11" label.
Alternatively, you can backport using cherry_picker on the command line.
cherry_picker 9c5aa8967bd7c1b02fb1da055c6b3afcccbbb251 3.11
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@miss-islington miss-islington assigned ambv May 19, 2023
@ambv ambv added needs backport to 3.11 only security fixes and removed needs backport to 3.11 only security fixes labels May 19, 2023
@miss-islington

miss-islington commented May 19, 2023

Copy link
Copy Markdown
Contributor

Thanks @zhangyoufu for the PR, and @ambv for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@miss-islington

miss-islington commented May 19, 2023

Copy link
Copy Markdown
Contributor

Sorry, @zhangyoufu and @ambv, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 9c5aa8967bd7c1b02fb1da055c6b3afcccbbb251 3.11
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
ambv pushed a commit to ambv/cpython that referenced this pull request May 19, 2023
@zhangyoufu @ambv
[3.11] pythongh-96522: Fix deadlock in pty.spawn (pythonGH-96639)
38b5d74 
(cherry picked from commit 9c5aa89)

Co-authored-by: Youfu Zhang <[email protected]>
@bedevere-bot

bedevere-bot commented May 19, 2023

Copy link
Copy Markdown

GH-104655 is a backport of this pull request to the 3.11 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the needs backport to 3.11 only security fixes label May 19, 2023
ambv added a commit that referenced this pull request May 19, 2023
@ambv @zhangyoufu
[3.11] gh-96522: Fix deadlock in pty.spawn (GH-96639) (#104655)
702ce87 
(cherry picked from commit 9c5aa89)

Co-authored-by: Youfu Zhang <[email protected]>
@zhangyoufu
zhangyoufu deleted the fix-pty-deadlock branch May 20, 2023 02:54
carljm added a commit to gsallam/cpython_with_perfmap_apii that referenced this pull request May 20, 2023
@carljm
Merge branch 'main' into gsallam/main
ea5965a 
* main: (30 commits)
  pythongh-103987: fix several crashes in mmap module (python#103990)
  docs: fix wrong indentation causing rendering error in dis page (python#104661)
  pythongh-94906: Support multiple steps in math.nextafter (python#103881)
  pythongh-104472: Skip `test_subprocess.ProcessTestCase.test_empty_env` if ASAN is enabled (python#104667)
  pythongh-103839: Allow building Tkinter against Tcl 8.7 without external libtommath (pythonGH-103842)
  pythongh-85984: New additions and improvements to the tty library. (python#101832)
  pythongh-104659: Consolidate python examples in enum documentation (python#104665)
  pythongh-92248: Deprecate `type`, `choices`, `metavar` parameters of `argparse.BooleanOptionalAction` (python#103678)
  pythongh-104645: fix error handling in marshal tests (python#104646)
  pythongh-104600: Make type.__type_params__ writable (python#104634)
  pythongh-104602: Add additional test for listcomp with lambda (python#104639)
  pythongh-104640: Disallow walrus in comprehension within type scopes (python#104641)
  pythongh-103921: Rename "type" header in argparse docs (python#104654)
  Improve readability of `typing._ProtocolMeta.__instancecheck__` (python#104649)
  pythongh-96522: Fix deadlock in pty.spawn (python#96639)
  pythonGH-102818: Do not call `PyTraceBack_Here` in sys.settrace trampoline.  (pythonGH-104579)
  pythonGH-103545: Add macOS specific constants for ``os.setpriority`` to ``os`` (python#104606)
  pythongh-104623: Update macOS installer to SQLite 3.42.0 (pythonGH-104624)
  pythongh-104619: never leak comprehension locals to outer locals() (python#104637)
  pythongh-104602: ensure all cellvars are known up front (python#104603)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ambv ambv Awaiting requested review from ambv

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@ambv ambv

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@zhangyoufu @bedevere-bot @miss-islington @ambv
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.