GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — [3.12] gh-114440: Close writer pipe in multiprocessing.Queue, not concurrent.futures by encukou · Pull Request #114489 · python/cpython

Conversation

@encukou

@encukou encukou commented Jan 23, 2024 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

This was left out of the 3.12 backport for three related issues:

  • test_concurrent_futures.test_deadlock: test_crash_big_data() hangs randomly on Windows #107219 (which adds self.call_queue._writer.close() to _ExecutorManagerThread in concurrent.futures)
  • Unexpected traceback output in test_concurrent_futures and crash #109370 (which changes this to be only called on Windows)
  • test_concurrent_futures.test_shutdown: test_interpreter_shutdown() fails randomly (race condition) #109047 (which moves the call to multiprocessing.Queue's _terminate_broken)

Without this change, ProcessPoolExecutor sometimes hangs on Windows when a worker process is terminated.

Co-authored-by: Victor Stinner [email protected]
Co-authored-by: Serhiy Storchaka [email protected]

  • Issue: test_deadlock & test_threads sometimes deadlock on Windows refleaks buildbot #114440

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@encukou @vstinner
[3.12] pythongh-114440: Close writer pipe in multiprocessing.Queue, n…
188228f 
…ot concurrent.futures

This was left out of the 3.12 backport for three related issues:
- pythongh-107219 (which adds `self.call_queue._writer.close()` to `_ExecutorManagerThread` in `concurrent.futures`)
- pythongh-109370 (which changes this to be only called on Windows)
- pythongh-109047 (which moves the call to `multiprocessing.Queue`'s `_terminate_broken`)

Without this change, ProcessPoolExecutor sometimes hangs on Windows
when a worker process is terminated.

Co-authored-by: Victor Stinner <[email protected]>
Co-authored-by:	Serhiy Storchaka <[email protected]>
@encukou
encukou requested review from serhiy-storchaka and vstinner January 23, 2024 15:52
@encukou
encukou requested a review from rhettinger as a code owner January 23, 2024 15:52
@bedevere-app bedevere-app Bot added the awaiting core review label Jan 23, 2024
@bedevere-app bedevere-app Bot mentioned this pull request Jan 23, 2024
test_deadlock & test_threads sometimes deadlock on Windows refleaks buildbot #114440
Closed
vstinner
vstinner approved these changes Jan 23, 2024
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

If I recall correctly, @serhiy-storchaka had doubts about this backport to stable branches, and wanted extra changes. But his PR to make extra changes didn't land. And apparently, then we forgot to finish these backports.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Jan 23, 2024
serhiy-storchaka
serhiy-storchaka reviewed Jan 23, 2024
View reviewed changes
Comment thread Misc/NEWS.d/next/Library/2023-09-25-02-11-14.gh-issue-114440.b2TrqG.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.
@serhiy-storchaka
Update Misc/NEWS.d/next/Library/2023-09-25-02-11-14.gh-issue-114440.b…
62af0b7 
…2TrqG.rst
serhiy-storchaka
serhiy-storchaka approved these changes Jan 23, 2024
View reviewed changes

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@encukou

encukou commented Jan 23, 2024

Copy link
Copy Markdown
Member Author

!buildbot AMD64 Windows11 Refleaks 3.12
encukou reacted with confused emoji
All reactions
  • 😕 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot

bedevere-bot commented Jan 23, 2024

Copy link
Copy Markdown

The regex 'AMD64 Windows11 Refleaks 3.12' did not match any buildbot builder. Is the requested builder in the list of stable builders?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@serhiy-storchaka

serhiy-storchaka commented Jan 23, 2024

Copy link
Copy Markdown
Member

!buildbot AMD64 Windows11 Refleaks
encukou reacted with heart emoji
All reactions
  • ❤️ 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot

bedevere-bot commented Jan 23, 2024

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @serhiy-storchaka for commit 62af0b7 🤖

The command will test the builders whose names match following regular expression: AMD64 Windows11 Refleaks

The builders matched are:

  • AMD64 Windows11 Refleaks PR
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@rhettinger
rhettinger removed their request for review January 24, 2024 02:02
@encukou
encukou merged commit c912bc3 into python:3.12 Jan 24, 2024
@bedevere-app bedevere-app Bot removed the awaiting merge label Jan 24, 2024
@encukou
encukou deleted the 114440-3.12 branch January 24, 2024 12:21
@vstinner

vstinner commented Jan 24, 2024

Copy link
Copy Markdown
Member

This was left out of the 3.12 backport for three related issues

Should this change be backported to 3.11?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
naveen521kk pushed a commit to naveen521kk/cpython that referenced this pull request Feb 19, 2024
@encukou @vstinner
@serhiy-storchaka @naveen521kk
[3.12] pythongh-114440: Close writer pipe in multiprocessing.Queue, n…
c4877f2 
…ot concurrent.futures (pythonGH-114489)

This was left out of the 3.12 backport for three related issues:
- pythongh-107219 (which adds `self.call_queue._writer.close()` to `_ExecutorManagerThread` in `concurrent.futures`)
- pythongh-109370 (which changes this to be only called on Windows)
- pythongh-109047 (which moves the call to `multiprocessing.Queue`'s `_terminate_broken`)

Without this change, ProcessPoolExecutor sometimes hangs on Windows
when a worker process is terminated.

Co-authored-by: Victor Stinner <[email protected]>
Co-authored-by: Serhiy Storchaka <[email protected]>
naveen521kk pushed a commit to naveen521kk/cpython that referenced this pull request Feb 19, 2024
@encukou @vstinner
@serhiy-storchaka @naveen521kk
[3.12] pythongh-114440: Close writer pipe in multiprocessing.Queue, n…
fd8463b 
…ot concurrent.futures (pythonGH-114489)

This was left out of the 3.12 backport for three related issues:
- pythongh-107219 (which adds `self.call_queue._writer.close()` to `_ExecutorManagerThread` in `concurrent.futures`)
- pythongh-109370 (which changes this to be only called on Windows)
- pythongh-109047 (which moves the call to `multiprocessing.Queue`'s `_terminate_broken`)

Without this change, ProcessPoolExecutor sometimes hangs on Windows
when a worker process is terminated.

Co-authored-by: Victor Stinner <[email protected]>
Co-authored-by: Serhiy Storchaka <[email protected]>
naveen521kk pushed a commit to naveen521kk/cpython that referenced this pull request Feb 19, 2024
@encukou @vstinner
@serhiy-storchaka @naveen521kk
[3.12] pythongh-114440: Close writer pipe in multiprocessing.Queue, n…
df2288a 
…ot concurrent.futures (pythonGH-114489)

This was left out of the 3.12 backport for three related issues:
- pythongh-107219 (which adds `self.call_queue._writer.close()` to `_ExecutorManagerThread` in `concurrent.futures`)
- pythongh-109370 (which changes this to be only called on Windows)
- pythongh-109047 (which moves the call to `multiprocessing.Queue`'s `_terminate_broken`)

Without this change, ProcessPoolExecutor sometimes hangs on Windows
when a worker process is terminated.

Co-authored-by: Victor Stinner <[email protected]>
Co-authored-by: Serhiy Storchaka <[email protected]>
naveen521kk pushed a commit to naveen521kk/cpython that referenced this pull request Feb 19, 2024
@encukou @vstinner
@serhiy-storchaka @naveen521kk
[3.12] pythongh-114440: Close writer pipe in multiprocessing.Queue, n…
df5f451 
…ot concurrent.futures (pythonGH-114489)

This was left out of the 3.12 backport for three related issues:
- pythongh-107219 (which adds `self.call_queue._writer.close()` to `_ExecutorManagerThread` in `concurrent.futures`)
- pythongh-109370 (which changes this to be only called on Windows)
- pythongh-109047 (which moves the call to `multiprocessing.Queue`'s `_terminate_broken`)

Without this change, ProcessPoolExecutor sometimes hangs on Windows
when a worker process is terminated.

Co-authored-by: Victor Stinner <[email protected]>
Co-authored-by: Serhiy Storchaka <[email protected]>
naveen521kk pushed a commit to naveen521kk/cpython that referenced this pull request Feb 21, 2024
@encukou @vstinner
@serhiy-storchaka @naveen521kk
[3.12] pythongh-114440: Close writer pipe in multiprocessing.Queue, n…
69198de 
…ot concurrent.futures (pythonGH-114489)

This was left out of the 3.12 backport for three related issues:
- pythongh-107219 (which adds `self.call_queue._writer.close()` to `_ExecutorManagerThread` in `concurrent.futures`)
- pythongh-109370 (which changes this to be only called on Windows)
- pythongh-109047 (which moves the call to `multiprocessing.Queue`'s `_terminate_broken`)

Without this change, ProcessPoolExecutor sometimes hangs on Windows
when a worker process is terminated.

Co-authored-by: Victor Stinner <[email protected]>
Co-authored-by: Serhiy Storchaka <[email protected]>
naveen521kk pushed a commit to naveen521kk/cpython that referenced this pull request Jul 11, 2024
@encukou @vstinner
@serhiy-storchaka @naveen521kk
[3.12] pythongh-114440: Close writer pipe in multiprocessing.Queue, n…
6a3e15b 
…ot concurrent.futures (pythonGH-114489)

This was left out of the 3.12 backport for three related issues:
- pythongh-107219 (which adds `self.call_queue._writer.close()` to `_ExecutorManagerThread` in `concurrent.futures`)
- pythongh-109370 (which changes this to be only called on Windows)
- pythongh-109047 (which moves the call to `multiprocessing.Queue`'s `_terminate_broken`)

Without this change, ProcessPoolExecutor sometimes hangs on Windows
when a worker process is terminated.

Co-authored-by: Victor Stinner <[email protected]>
Co-authored-by: Serhiy Storchaka <[email protected]>
naveen521kk pushed a commit to naveen521kk/cpython that referenced this pull request Jul 11, 2024
@encukou @vstinner
@serhiy-storchaka @naveen521kk
[3.12] pythongh-114440: Close writer pipe in multiprocessing.Queue, n…
23a82b7 
…ot concurrent.futures (pythonGH-114489)

This was left out of the 3.12 backport for three related issues:
- pythongh-107219 (which adds `self.call_queue._writer.close()` to `_ExecutorManagerThread` in `concurrent.futures`)
- pythongh-109370 (which changes this to be only called on Windows)
- pythongh-109047 (which moves the call to `multiprocessing.Queue`'s `_terminate_broken`)

Without this change, ProcessPoolExecutor sometimes hangs on Windows
when a worker process is terminated.

Co-authored-by: Victor Stinner <[email protected]>
Co-authored-by: Serhiy Storchaka <[email protected]>
naveen521kk pushed a commit to naveen521kk/cpython that referenced this pull request Jul 11, 2024
@encukou @vstinner
@serhiy-storchaka @naveen521kk
[3.12] pythongh-114440: Close writer pipe in multiprocessing.Queue, n…
701b465 
…ot concurrent.futures (pythonGH-114489)

This was left out of the 3.12 backport for three related issues:
- pythongh-107219 (which adds `self.call_queue._writer.close()` to `_ExecutorManagerThread` in `concurrent.futures`)
- pythongh-109370 (which changes this to be only called on Windows)
- pythongh-109047 (which moves the call to `multiprocessing.Queue`'s `_terminate_broken`)

Without this change, ProcessPoolExecutor sometimes hangs on Windows
when a worker process is terminated.

Co-authored-by: Victor Stinner <[email protected]>
Co-authored-by: Serhiy Storchaka <[email protected]>
naveen521kk pushed a commit to msys2-contrib/cpython-mingw that referenced this pull request Aug 5, 2024
@encukou @vstinner
@serhiy-storchaka @naveen521kk
[3.12] pythongh-114440: Close writer pipe in multiprocessing.Queue, n…
2ce1791 
…ot concurrent.futures (pythonGH-114489)

This was left out of the 3.12 backport for three related issues:
- pythongh-107219 (which adds `self.call_queue._writer.close()` to `_ExecutorManagerThread` in `concurrent.futures`)
- pythongh-109370 (which changes this to be only called on Windows)
- pythongh-109047 (which moves the call to `multiprocessing.Queue`'s `_terminate_broken`)

Without this change, ProcessPoolExecutor sometimes hangs on Windows
when a worker process is terminated.

Co-authored-by: Victor Stinner <[email protected]>
Co-authored-by: Serhiy Storchaka <[email protected]>
naveen521kk pushed a commit to naveen521kk/cpython that referenced this pull request Sep 4, 2024
@encukou @vstinner
@serhiy-storchaka @naveen521kk
[3.12] pythongh-114440: Close writer pipe in multiprocessing.Queue, n…
1ca1659 
…ot concurrent.futures (pythonGH-114489)

This was left out of the 3.12 backport for three related issues:
- pythongh-107219 (which adds `self.call_queue._writer.close()` to `_ExecutorManagerThread` in `concurrent.futures`)
- pythongh-109370 (which changes this to be only called on Windows)
- pythongh-109047 (which moves the call to `multiprocessing.Queue`'s `_terminate_broken`)

Without this change, ProcessPoolExecutor sometimes hangs on Windows
when a worker process is terminated.

Co-authored-by: Victor Stinner <[email protected]>
Co-authored-by: Serhiy Storchaka <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner vstinner approved these changes
@serhiy-storchaka serhiy-storchaka serhiy-storchaka approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@encukou @bedevere-bot @serhiy-storchaka @vstinner
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.