GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — bpo-35537: subprocess can use posix_spawn with pipes by vstinner · Pull Request #11575 · python/cpython

Conversation

@vstinner

@vstinner vstinner commented Jan 16, 2019 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member
  • subprocess.Popen can now also use os.posix_spawn() with pipes if
    pipe file descriptors are greater than 2.
  • Fix Popen._posix_spawn(): set _child_created to True.
  • Add Popen._close_pipe_fds() helper function to factorize the code.

https://bugs.python.org/issue35537

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@vstinner
vstinner requested a review from gpshead as a code owner January 16, 2019 00:20
@the-knights-who-say-ni the-knights-who-say-ni added the CLA signed label Jan 16, 2019
@bedevere-bot bedevere-bot added the awaiting merge label Jan 16, 2019
@vstinner
vstinner requested a review from giampaolo January 16, 2019 00:20
@vstinner vstinner added the skip news label Jan 16, 2019
@vstinner

vstinner commented Jan 16, 2019

Copy link
Copy Markdown
Member Author

Follow-up of PR #11452.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner vstinner mentioned this pull request Jan 16, 2019
bpo-35537: subprocess uses os.posix_spawn in some cases #11452
Merged
@vstinner

vstinner commented Jan 16, 2019

Copy link
Copy Markdown
Member Author

cc @izbyshev, @pablogsal, @nanjekyejoannah

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Jan 16, 2019

Copy link
Copy Markdown
Member Author

PR rebased on top of PR #11579 (commit 0785889).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal
pablogsal self-requested a review January 16, 2019 14:34
izbyshev
izbyshev reviewed Jan 16, 2019
View reviewed changes
Comment thread Lib/subprocess.py Outdated

@izbyshev izbyshev Jan 16, 2019

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically, there is no need to explicitly close parent pipe ends because they have O_CLOEXEC. One small advantage of doing that is to reduce the number of descriptors before the following dup2, which could increase their number (in a corner case when any of descriptors 0, 1, 2 is closed in the parent process) and potentially hit resource limits.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@vstinner vstinner Jan 16, 2019

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically, there is no need, but _posixsubprocess does that. In case of doubt, I prefer to mimick _posixsubprocess behavior.

    /* Close parent's pipe ends. */
    if (p2cwrite != -1)
        POSIX_CALL(close(p2cwrite));
    if (c2pread != -1)
        POSIX_CALL(close(c2pread));
    if (errread != -1)
        POSIX_CALL(close(errread));
    POSIX_CALL(close(errpipe_read));

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@vstinner vstinner Jan 16, 2019

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we decide to skip the close(), I would prefer to also modify _posixsubprocess. But here I only want to use posix_spawn() in more cases, so I prefer to not modify "unrelated" changes :-)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
izbyshev
izbyshev reviewed Jan 16, 2019
View reviewed changes
Comment thread Lib/subprocess.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@vstinner
bpo-35537: subprocess can use posix_spawn with pipes
9bc8adf
* subprocess.Popen can now also use os.posix_spawn() with pipes if
  pipe file descriptors are greater than 2.
* Fix Popen._posix_spawn(): set _child_created to True.
* Add Popen._close_pipe_fds() helper function to factorize the code.
@vstinner

vstinner commented Jan 16, 2019

Copy link
Copy Markdown
Member Author

PR rebased on top of PR #11579 (commit 0785889).

Since PR #11579, I rebased this PR again :-)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Jan 16, 2019

Copy link
Copy Markdown
Member Author

@gpshead @giampaolo: would you mind to review this change? It allows to use posix_spawn() in subprocess in much more cases.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Jan 22, 2019

Copy link
Copy Markdown
Member Author

If I don't get any feedback before the end of the week, I will just merge my PR. I prefer to merge it early during the 3.8 development cycle, to get more time to fix it if something goes wrong :)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

giampaolo
giampaolo requested changes Jan 23, 2019
View reviewed changes
Comment thread Lib/subprocess.py
os.close(errwrite)

if devnull_fd is not None:
os.close(devnull_fd)

@giampaolo giampaolo Jan 23, 2019

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case of premature failure on X.Close() or os.close(X) the remaining pipes/fds will remain "open". Perhaps it makes sense to use contextlib.ExitStack.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@vstinner vstinner Jan 23, 2019

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it a regression caused by my PR? Or a general remark on existing code? My PR just moves code, no?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@giampaolo giampaolo Jan 23, 2019

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, it doesn't look like a regression introduced by you (was already there).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@vstinner vstinner Jan 23, 2019

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At least, I factorized the code so it should help to enhance the code ;-) Maybe open an issue if you want to work on that? I don't see an obvious pattern to ensure that all file descriptors are properly closed.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@giampaolo giampaolo Jan 23, 2019

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, makes sense. I bet there are many other places which can benefit from ExitStack in a similar manner.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot

bedevere-bot commented Jan 23, 2019

Copy link
Copy Markdown

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot added awaiting changes and removed awaiting merge labels Jan 23, 2019
vstinner
vstinner commented Jan 23, 2019
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@giampaolo: So what do you think of the overall change? :-)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/subprocess.py
os.close(errwrite)

if devnull_fd is not None:
os.close(devnull_fd)

@vstinner vstinner Jan 23, 2019

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At least, I factorized the code so it should help to enhance the code ;-) Maybe open an issue if you want to work on that? I don't see an obvious pattern to ensure that all file descriptors are properly closed.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
giampaolo
giampaolo approved these changes Jan 23, 2019
View reviewed changes

@giampaolo giampaolo left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting changes labels Jan 23, 2019
@vstinner
vstinner merged commit f6243ac into python:master Jan 23, 2019
@bedevere-bot bedevere-bot removed the awaiting merge label Jan 23, 2019
@vstinner
vstinner deleted the subprocess_spawn2 branch January 23, 2019 18:00
@vstinner

vstinner commented Jan 23, 2019

Copy link
Copy Markdown
Member Author

@giampaolo: "LGTM"

Yay! Thanks for the approval! I merged my PR.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Jan 23, 2019

Copy link
Copy Markdown
Member Author

@giampaolo: As I wrote, please open a new issue if you would like to enhance the new _close_pipe_fds() helper method.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@giampaolo giampaolo mentioned this pull request Jan 27, 2019
subprocess: close pipes/fds by using ExitStack #11686
Merged
@giampaolo

giampaolo commented Jan 27, 2019

Copy link
Copy Markdown
Contributor

@vstinner here it is: #11686

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

gpshead pushed a commit that referenced this pull request Jan 29, 2019
@giampaolo @gpshead
subprocess: close pipes/fds by using ExitStack (GH-11686)
bafa848
Close pipes/fds in subprocess by using ExitStack.

"In case of premature failure on X.Close() or os.close(X) the remaining pipes/fds will remain "open". Perhaps it makes sense to use contextlib.ExitStack."
- Rationale: #11575 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@giampaolo giampaolo giampaolo approved these changes
@gpshead gpshead Awaiting requested review from gpshead
@pablogsal pablogsal Awaiting requested review from pablogsal
+1 more reviewer
@izbyshev izbyshev izbyshev left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@vstinner @bedevere-bot @giampaolo @izbyshev @the-knights-who-say-ni
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.