GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-104372: Drop the GIL around the vfork() call. by gpshead · Pull Request #104782 · python/cpython

Conversation

@gpshead

@gpshead gpshead commented May 23, 2023 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

On Linux where the subprocess module can use the vfork syscall for faster spawning, prevent the parent process from blocking other threads by dropping the GIL while it waits for the vfork'ed child process exec outcome. This prevents spawning a binary from a slow filesystem from blocking the rest of the application.

Fixes #104372.

  • Issue: subprocess can block all Python threads when using vfork() until the child process exec() succeeds or fails. #104372

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@gpshead
pythongh-104372: Drop the GIL around the vfork() call.
2d54d8e 
On Linux where the `subprocess` module can use the `vfork` syscall for
faster spawning, prevent the parent process from blocking other threads
by dropping the GIL while it waits for the vfork'ed child process `exec`
outcome.  This prevents spawning a binary from a slow filesystem from
blocking the rest of the application.

Fixes python#104372.
@gpshead gpshead self-assigned this May 23, 2023
@bedevere-bot bedevere-bot mentioned this pull request May 23, 2023
subprocess can block all Python threads when using vfork() until the child process exec() succeeds or fails. #104372
Closed
@gpshead

gpshead commented May 23, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

This PR is preferred over #104515 as it does not rely on undefined behavior magic cross-process address space sharing shenanigans.

Pre-requisites: #104518 + #104697 to make the post vfork code safe for use without the GIL.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@gpshead gpshead added needs backport to 3.11 only security fixes needs backport to 3.12 only security fixes labels May 23, 2023
gpshead and others added 3 commits May 23, 2023 22:56
@gpshead
Simplify: avoid -Wclobbered the compiler warning.
eb38573 
Don't write to the vfork_tstate_save pointer local after vfork.
@gpshead
Docs: clarify that timeout= can't cover process creation.
9197a4f
@gpshead
Merge branch 'main' into subprocess-drop-gil-before-vfork
02a8538
@ghost

ghost commented May 24, 2023 •
edited by ghost
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.
CLA signed
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@gpshead
gpshead requested review from Yhg1s and vstinner May 24, 2023 08:13
@gpshead
gpshead marked this pull request as ready for review May 24, 2023 08:14
@bedevere-bot bedevere-bot added the awaiting core review label May 24, 2023
Yhg1s
Yhg1s approved these changes May 24, 2023
View reviewed changes
Comment thread Modules/_posixsubprocess.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting core review labels May 24, 2023
@gpshead
update the comment
df7f7ab
@gpshead
gpshead enabled auto-merge (squash) May 25, 2023 19:53
@gpshead
gpshead merged commit d086792 into python:main May 25, 2023
@miss-islington

miss-islington commented May 25, 2023

Copy link
Copy Markdown
Contributor

Thanks @gpshead for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11, 3.12.
🐍🍒⛏🤖
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the awaiting merge label May 25, 2023
@miss-islington

miss-islington commented May 25, 2023

Copy link
Copy Markdown
Contributor

Sorry, @gpshead, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker d08679212d9af52dd074cd4a6abb440edb944c9c 3.11
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request May 25, 2023
@gpshead @miss-islington
pythongh-104372: Drop the GIL around the vfork() call. (pythonGH-104782)
302217d 
On Linux where the `subprocess` module can use the `vfork` syscall for
faster spawning, prevent the parent process from blocking other threads
by dropping the GIL while it waits for the vfork'ed child process `exec`
outcome.  This prevents spawning a binary from a slow filesystem from
blocking the rest of the application.

Fixes pythonGH-104372.
(cherry picked from commit d086792)

Co-authored-by: Gregory P. Smith <[email protected]>
@bedevere-bot

bedevere-bot commented May 25, 2023

Copy link
Copy Markdown

GH-104942 is a backport of this pull request to the 3.12 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the needs backport to 3.12 only security fixes label May 25, 2023
@gpshead
gpshead deleted the subprocess-drop-gil-before-vfork branch May 25, 2023 20:15
gpshead added a commit that referenced this pull request May 25, 2023
@miss-islington @gpshead
[3.12] gh-104372: Drop the GIL around the vfork() call. (GH-104782) (#…
5c2971b 
…104942)

gh-104372: Drop the GIL around the vfork() call. (GH-104782)

On Linux where the `subprocess` module can use the `vfork` syscall for
faster spawning, prevent the parent process from blocking other threads
by dropping the GIL while it waits for the vfork'ed child process `exec`
outcome.  This prevents spawning a binary from a slow filesystem from
blocking the rest of the application.

Fixes GH-104372.
(cherry picked from commit d086792)

Co-authored-by: Gregory P. Smith <[email protected]>
gpshead added a commit to gpshead/cpython that referenced this pull request May 26, 2023
@gpshead
pythongh-104372: Drop the GIL around the vfork() call. (python#104782)
75f7969 
On Linux where the `subprocess` module can use the `vfork` syscall for
faster spawning, prevent the parent process from blocking other threads
by dropping the GIL while it waits for the vfork'ed child process `exec`
outcome.  This prevents spawning a binary from a slow filesystem from
blocking the rest of the application.

Fixes python#104372.

(cherry picked from commit d086792)
@bedevere-bot

bedevere-bot commented May 26, 2023

Copy link
Copy Markdown

GH-104958 is a backport of this pull request to the 3.11 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the needs backport to 3.11 only security fixes label May 26, 2023
@erlend-aasland erlend-aasland mentioned this pull request Jun 6, 2023
3.12 backport gh 105236 #105358
Closed
gpshead added a commit that referenced this pull request Sep 1, 2023
@gpshead
[3.11] gh-104372: Drop the GIL around the vfork() call. (#104782) (#1…
6ba1234 
…04958)

gh-104372: Drop the GIL around the vfork() call. (#104782)

On Linux where the `subprocess` module can use the `vfork` syscall for
faster spawning, prevent the parent process from blocking other threads
by dropping the GIL while it waits for the vfork'ed child process `exec`
outcome.  This prevents spawning a binary from a slow filesystem from
blocking the rest of the application.

Fixes #104372.

(cherry picked from commit d086792)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Yhg1s Yhg1s Yhg1s approved these changes
@vstinner vstinner Awaiting requested review from vstinner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@gpshead gpshead

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

subprocess can block all Python threads when using vfork() until the child process exec() succeeds or fails.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@gpshead @miss-islington @bedevere-bot @Yhg1s
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.