GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — GH-110829: Ensure Thread.join() joins the OS thread by pitrou · Pull Request #110848 · python/cpython

Conversation

@pitrou

@pitrou pitrou commented Oct 13, 2023 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member
  • Issue: threading Thread.join should call the OS join API #110829

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ambv reacted with rocket emoji
All reactions
  • 🚀 1 reaction
@bedevere-app bedevere-app Bot mentioned this pull request Oct 13, 2023
threading Thread.join should call the OS join API #110829
Closed
@pitrou

pitrou commented Oct 13, 2023

Copy link
Copy Markdown
Member Author

@gpshead This should work at least on Linux, hopefully on Windows if I'm lucky.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pitrou
pitrou force-pushed the gh-110829-join branch 2 times, most recently from ef411ef to 41b9cba Compare October 13, 2023 23:23
@pitrou pitrou added type-feature A feature request or enhancement 🔨 test-with-buildbots Test PR w/ buildbots; report in status section interpreter-core (Objects, Python, Grammar, and Parser dirs) labels Oct 14, 2023
@bedevere-bot

bedevere-bot commented Oct 14, 2023

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @pitrou for commit 6c594ce 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 14, 2023
@pitrou
pitrou marked this pull request as ready for review October 14, 2023 07:47
@bedevere-app bedevere-app Bot added the awaiting core review label Oct 14, 2023
@pitrou
pitrou requested a review from gpshead October 14, 2023 07:47
@pitrou

pitrou commented Oct 14, 2023

Copy link
Copy Markdown
Member Author

@zooba I think my use of the Windows APIs is correct here, but you might want to take a look.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

pitrou
pitrou commented Oct 14, 2023
View reviewed changes
Comment thread Modules/_threadmodule.c Outdated
Comment on lines +55 to +59
unsigned long long ull_handle = PyLong_AsUnsignedLongLong(handle_obj);
if ((*ident == (unsigned long) -1 || ull_handle == (unsigned long long) -1)
&& PyErr_Occurred()) {
// This should not occur as we control the contents of state->joinable_dict
return -1;
}
*handle = (Py_uintptr_t) ull_handle;

@pitrou pitrou Oct 14, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I filed #110863 to ease this a bit.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
pitrou
pitrou commented Oct 14, 2023
View reviewed changes
Comment thread Lib/test/_test_multiprocessing.py

def test_terminate(self):
if self.TYPE == 'threads':
self.skipTest("Threads cannot be terminated")

@pitrou pitrou Oct 14, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note the changes in this file are probably not necessary, it's just that without them the threads emulation of processes takes a very long time to test. The reason is simple and not related to this PR: while you can terminate a process early, you cannot do that on a thread, so joining a sleeping thread has to wait for the sleep to finish.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ericsnowcurrently and gpshead reacted with thumbs up emoji
All reactions
  • 👍 2 reactions

@encukou encukou Jan 17, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use a shorter wait time rather than skip the test entirely: #114186

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
pitrou
pitrou commented Oct 14, 2023
View reviewed changes
Comment thread Modules/_threadmodule.c Outdated
} else {
PyThread_detach_thread((Py_uintptr_t) handle);
}
}

@pitrou pitrou Oct 14, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might also want to emit a ResourceWarning here, what do you think @vstinner ?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@pitrou pitrou Oct 14, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking about it, I'd rather defer this to a later PR (if at all).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
vstinner
vstinner reviewed Oct 14, 2023
View reviewed changes
Comment thread Include/pythread.h Resolved

Uh oh!

There was an error while loading. Please reload this page.

@vstinner vstinner mentioned this pull request Oct 14, 2023
Explain why the current state of the API is inconsistent capi-workgroup/api-evolution#30
Open
@pitrou

pitrou commented Oct 14, 2023

Copy link
Copy Markdown
Member Author

!buildbot wasm

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot

bedevere-bot commented Oct 14, 2023

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @pitrou for commit 26e868c 🤖

The command will test the builders whose names match following regular expression: wasm

The builders matched are:

  • wasm32-emscripten browser (dynamic linking, no tests) PR
  • wasm32-emscripten node (pthreads) PR
  • wasm32-emscripten node (dynamic linking) PR
  • wasm32-wasi PR
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pitrou

pitrou commented Oct 14, 2023

Copy link
Copy Markdown
Member Author

Most buildbots are green and the few failures look unrelated:
https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F110848%2Fmerge

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pitrou pitrou mentioned this pull request Oct 15, 2023
GH-109978: Allow multiprocessing finalizers to run on a separate thread #110510
Closed
zooba
zooba reviewed Oct 16, 2023
View reviewed changes
Comment thread Python/thread_nt.h Resolved

Uh oh!

There was an error while loading. Please reload this page.

@pitrou
pitrou force-pushed the gh-110829-join branch from 26e868c to e2f440f Compare October 16, 2023 21:01
@pitrou pitrou added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 16, 2023
@bedevere-bot

bedevere-bot commented Oct 16, 2023

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @pitrou for commit e2f440f 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 16, 2023
@pitrou

pitrou commented Oct 17, 2023

Copy link
Copy Markdown
Member Author

I formally announce that there are no buildbot failures :-)

I realize that this PR lacks some docs for the _thread additions, but before that I would like someone to vet the overall approach. @gpshead Are you available for a review?

ericsnowcurrently reacted with hooray emoji
All reactions
  • 🎉 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gpshead

gpshead commented Oct 25, 2023

Copy link
Copy Markdown
Member

@gpshead Are you available for a review?

I'll take a look this week, I was on vacation.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gpshead gpshead self-assigned this Oct 25, 2023
pitrou added 2 commits November 4, 2023 11:48
@pitrou
Introduce PyThread_ident_t, PyThread_handle_t
420ece6
@pitrou
Address compiler warnings
96849d1
@pitrou
pitrou force-pushed the gh-110829-join branch from 2546653 to 96849d1 Compare November 4, 2023 11:05
@pitrou
Fix merge error
5cf2684
@pitrou pitrou added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 4, 2023
@bedevere-bot

bedevere-bot commented Nov 4, 2023

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @pitrou for commit 5cf2684 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 4, 2023
@blurb-it
📜🤖 Added by blurb_it.
fa11244
@pitrou
pitrou enabled auto-merge (squash) November 4, 2023 13:46
@pitrou
pitrou merged commit 0e9c364 into python:main Nov 4, 2023
@bedevere-app bedevere-app Bot removed the awaiting merge label Nov 4, 2023
@pitrou
pitrou deleted the gh-110829-join branch November 4, 2023 14:14
@pitrou

pitrou commented Nov 4, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

In an ironic (and depressing) twist, it seems that even after pthread_join returned successfully, the given thread can still appear in /proc/self/task. Forking should be safe in such a situation, but our fork() wrapper prints a warning nevertheless....

gpshead reacted with confused emoji
All reactions
  • 😕 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gpshead

gpshead commented Nov 4, 2023

Copy link
Copy Markdown
Member

doh! I guess I can understand why, the thread won't be executing anymore so join succeeded but the in kernel data structure cleanup will take its own sweet time. :/

I still think it is a valuable addition that we wait for the OS join to happen (thanks for all this work, reviews & merge) but I understand it is annoying given one of the motivating reasons.

ericsnowcurrently reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

encukou added a commit to encukou/cpython that referenced this pull request Jan 17, 2024
@encukou
pythongh-113205: test_multiprocessing.test_terminate: Test the API wo…
f34cfdb
…rks on threadpools

Threads can't be forced to terminate (without potentially corrupting too much
state), so the  expected behaviour of `ThreadPool.terminate` is to wait for
the currently executing tasks to finish.

The entire test was skipped in pythonGH-110848 (0e9c364).
Instead of skipping it entirely, we should ensure the API eventually succeeds:
use a shorter timeout.

For the record: on my machine, when the test is un-skipped, the task manages to
start in about 1.5% cases.
@encukou encukou mentioned this pull request Jan 17, 2024
gh-113205: test_multiprocessing.test_terminate: Test the API on threadpools #114186
Merged
gpshead pushed a commit that referenced this pull request Jan 18, 2024
@encukou
gh-113205: test_multiprocessing.test_terminate: Test the API on threa…
c1db960
…dpools (#114186)

gh-113205: test_multiprocessing.test_terminate: Test the API works on threadpools

Threads can't be forced to terminate (without potentially corrupting too much
state), so the  expected behaviour of `ThreadPool.terminate` is to wait for
the currently executing tasks to finish.

The entire test was skipped in GH-110848 (0e9c364).
Instead of skipping it entirely, we should ensure the API eventually succeeds:
use a shorter timeout.

For the record: on my machine, when the test is un-skipped, the task manages to
start in about 1.5% cases.
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jan 18, 2024
@encukou @miss-islington
pythongh-113205: test_multiprocessing.test_terminate: Test the API on…
09cd0dc
… threadpools (pythonGH-114186)

pythongh-113205: test_multiprocessing.test_terminate: Test the API works on threadpools

Threads can't be forced to terminate (without potentially corrupting too much
state), so the  expected behaviour of `ThreadPool.terminate` is to wait for
the currently executing tasks to finish.

The entire test was skipped in pythonGH-110848 (0e9c364).
Instead of skipping it entirely, we should ensure the API eventually succeeds:
use a shorter timeout.

For the record: on my machine, when the test is un-skipped, the task manages to
start in about 1.5% cases.
(cherry picked from commit c1db960)

Co-authored-by: Petr Viktorin <[email protected]>
@miss-islington miss-islington mentioned this pull request Jan 18, 2024
[3.12] gh-113205: test_multiprocessing.test_terminate: Test the API on threadpools (GH-114186) #114222
Merged
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jan 18, 2024
@encukou @miss-islington
pythongh-113205: test_multiprocessing.test_terminate: Test the API on…
db5921b
… threadpools (pythonGH-114186)

pythongh-113205: test_multiprocessing.test_terminate: Test the API works on threadpools

Threads can't be forced to terminate (without potentially corrupting too much
state), so the  expected behaviour of `ThreadPool.terminate` is to wait for
the currently executing tasks to finish.

The entire test was skipped in pythonGH-110848 (0e9c364).
Instead of skipping it entirely, we should ensure the API eventually succeeds:
use a shorter timeout.

For the record: on my machine, when the test is un-skipped, the task manages to
start in about 1.5% cases.
(cherry picked from commit c1db960)

Co-authored-by: Petr Viktorin <[email protected]>
@miss-islington miss-islington mentioned this pull request Jan 18, 2024
[3.11] gh-113205: test_multiprocessing.test_terminate: Test the API on threadpools (GH-114186) #114223
Merged
kulikjak pushed a commit to kulikjak/cpython that referenced this pull request Jan 22, 2024
@encukou @kulikjak
pythongh-113205: test_multiprocessing.test_terminate: Test the API on…
d03347f
… threadpools (python#114186)

pythongh-113205: test_multiprocessing.test_terminate: Test the API works on threadpools

Threads can't be forced to terminate (without potentially corrupting too much
state), so the  expected behaviour of `ThreadPool.terminate` is to wait for
the currently executing tasks to finish.

The entire test was skipped in pythonGH-110848 (0e9c364).
Instead of skipping it entirely, we should ensure the API eventually succeeds:
use a shorter timeout.

For the record: on my machine, when the test is un-skipped, the task manages to
start in about 1.5% cases.
aisk pushed a commit to aisk/cpython that referenced this pull request Feb 11, 2024
@pitrou @blurb-it @aisk
pythonGH-110829: Ensure Thread.join() joins the OS thread (python#110848
a9d9778
)

Joining a thread now ensures the underlying OS thread has exited. This is required for safer fork() in multi-threaded processes.

---------

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
aisk pushed a commit to aisk/cpython that referenced this pull request Feb 11, 2024
@encukou @aisk
pythongh-113205: test_multiprocessing.test_terminate: Test the API on…
2d366e1
… threadpools (python#114186)

pythongh-113205: test_multiprocessing.test_terminate: Test the API works on threadpools

Threads can't be forced to terminate (without potentially corrupting too much
state), so the  expected behaviour of `ThreadPool.terminate` is to wait for
the currently executing tasks to finish.

The entire test was skipped in pythonGH-110848 (0e9c364).
Instead of skipping it entirely, we should ensure the API eventually succeeds:
use a shorter timeout.

For the record: on my machine, when the test is un-skipped, the task manages to
start in about 1.5% cases.
@ericsnowcurrently ericsnowcurrently mentioned this pull request Mar 15, 2024
gh-114271: Fix race in Thread.join() #114839
Merged
Glyphack pushed a commit to Glyphack/cpython that referenced this pull request Sep 2, 2024
@pitrou @blurb-it @Glyphack
pythonGH-110829: Ensure Thread.join() joins the OS thread (python#110848
ed1118f
)

Joining a thread now ensures the underlying OS thread has exited. This is required for safer fork() in multi-threaded processes.

---------

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Glyphack pushed a commit to Glyphack/cpython that referenced this pull request Sep 2, 2024
@encukou @Glyphack
pythongh-113205: test_multiprocessing.test_terminate: Test the API on…
ddf3788
… threadpools (python#114186)

pythongh-113205: test_multiprocessing.test_terminate: Test the API works on threadpools

Threads can't be forced to terminate (without potentially corrupting too much
state), so the  expected behaviour of `ThreadPool.terminate` is to wait for
the currently executing tasks to finish.

The entire test was skipped in pythonGH-110848 (0e9c364).
Instead of skipping it entirely, we should ensure the API eventually succeeds:
use a shorter timeout.

For the record: on my machine, when the test is un-skipped, the task manages to
start in about 1.5% cases.
This was referenced Feb 17, 2025
Active thread list may be inaccurate due to data type mismatch #130115
Closed
gh-130115: fix thread identifiers for 32-bit musl #130391
Merged
rafa-rrayes
rafa-rrayes reviewed Mar 18, 2025
View reviewed changes

@rafa-rrayes rafa-rrayes left a comment

Copy link
Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner vstinner left review comments
@encukou encukou encukou left review comments
@colesbury colesbury colesbury left review comments
@zooba zooba zooba left review comments
@ericsnowcurrently ericsnowcurrently ericsnowcurrently approved these changes
@gpshead gpshead Awaiting requested review from gpshead gpshead is a code owner
+1 more reviewer
@rafa-rrayes rafa-rrayes rafa-rrayes left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@gpshead gpshead

Labels

interpreter-core (Objects, Python, Grammar, and Parser dirs) type-feature A feature request or enhancement

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

9 participants

@pitrou @bedevere-bot @gpshead @vstinner @encukou @colesbury @ericsnowcurrently @zooba @rafa-rrayes
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.