GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-113884: Make queue.SimpleQueue thread-safe in --disable-gil builds by mpage · Pull Request #114161 · python/cpython

Conversation

@mpage

@mpage mpage commented Jan 17, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Make queue.SimpleQueue thread-safe when the GIL is disabled. Queue state is protected by the per-object lock; thread suspension now uses the low-level PyParkingLot abstraction.

  • Issue: Make queue.SimpleQueue thread-safe in --disable-gil builds #113884

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@bedevere-app bedevere-app Bot mentioned this pull request Jan 17, 2024
Make queue.SimpleQueue thread-safe in --disable-gil builds #113884
Closed
@mpage
mpage marked this pull request as ready for review January 17, 2024 01:21
@mpage
mpage requested a review from rhettinger as a code owner January 17, 2024 01:21
@bedevere-app bedevere-app Bot added the awaiting review label Jan 17, 2024
erlend-aasland
erlend-aasland reviewed Jan 17, 2024
View reviewed changes
Comment thread Misc/NEWS.d/next/Core and Builtins/2024-01-17-00-52-57.gh-issue-113884.CvEjUE.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.
@erlend-aasland

erlend-aasland commented Jan 17, 2024

Copy link
Copy Markdown
Contributor

Could you split it up so the ring buffer refactoring is done first, then a follow-up PR to make it thread-safe?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@mpage

mpage commented Jan 17, 2024

Copy link
Copy Markdown
Contributor Author

Could you split it up so the ring buffer refactoring is done first, then a follow-up PR to make it thread-safe?

@erlend-aasland I think it makes the most sense for these commits to be merged together as part of a single PR. The commits in this PR are structured in a way that they can be reviewed independently. How about we start by reviewing the first commit, which is the ring buffer refactoring, then go from there?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@mpage

mpage commented Jan 17, 2024

Copy link
Copy Markdown
Contributor Author

Tagging @colesbury
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
colesbury
colesbury reviewed Jan 17, 2024
View reviewed changes

@colesbury colesbury left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other than the threshold for shrinking and a few minor comments, this looks good to me.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Modules/_queuemodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Modules/_queuemodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Modules/_queuemodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Modules/_queuemodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Modules/_queuemodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Modules/_queuemodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
colesbury
colesbury reviewed Jan 17, 2024
View reviewed changes
Comment thread Modules/_queuemodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
@rhettinger
rhettinger removed their request for review January 18, 2024 00:53
@mpage
mpage requested a review from colesbury January 18, 2024 02:45
@erlend-aasland

erlend-aasland commented Jan 18, 2024

Copy link
Copy Markdown
Contributor

I think it makes the most sense for these commits to be merged together as part of a single PR. The commits in this PR are structured in a way that they can be reviewed independently. How about we start by reviewing the first commit, which is the ring buffer refactoring, then go from there?

AFAICS, the ring buffer refactoring makes for a nice stand-alone PR. It will result in a cleaner Git history, and it will make bisecting easier when hunting for bugs.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@mpage

mpage commented Jan 18, 2024

Copy link
Copy Markdown
Contributor Author

AFAICS, the ring buffer refactoring makes for a nice stand-alone PR.

I disagree but will split the PR in the interest of moving things forward.

Do you have any suggestions for the appropriate "size" of a PR? This will help me avoid having to split PRs in the future. I've been operating under the assumption that a PR should roughly correspond to a single, self-contained "feature" that stands on its own. I don't think the refactoring fits this model, as it wouldn't make sense to do without the other changes in the PR.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@mpage

mpage commented Jan 18, 2024

Copy link
Copy Markdown
Contributor Author

Split the ring buffer refactoring out into #114259. Putting this in draft until that PR is merged.
erlend-aasland reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@mpage
mpage marked this pull request as draft January 18, 2024 21:01
@bedevere-app bedevere-app Bot removed the awaiting review label Jan 18, 2024
@erlend-aasland

erlend-aasland commented Jan 19, 2024

Copy link
Copy Markdown
Contributor

Do you have any suggestions for the appropriate "size" of a PR? This will help me avoid having to split PRs in the future. I've been operating under the assumption that a PR should roughly correspond to a single, self-contained "feature" that stands on its own. I don't think the refactoring fits this model, as it wouldn't make sense to do without the other changes in the PR.

We normally do not mix refactorings and features. Most refactorings are done in order to make the code more readable, and/or structure the code in a more maintainable way. Those are qualities that stand on their own, and can justify a single (or multiple) PRs.
mpage reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@erlend-aasland

erlend-aasland commented Jan 19, 2024

Copy link
Copy Markdown
Contributor

As for PR size, I think it is good to try and keep the diff as small as possible (we don't like churn), and keep the number of changed lines within a couple of hundred lines. (Of course, generated files do not count.)
mpage reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
mpage and others added 6 commits January 19, 2024 14:12
@mpage
Use ParkingLot to manage waiting threads
bc7a640
@mpage
Make SimpleQueue thread-safe with the GIL disabled
65c0b90 
Methods on SimpleQueue are protected with the per-object lock.
@blurb-it @mpage
📜🤖 Added by blurb_it.
d999dd6
@erlend-aasland @mpage
Update Misc/NEWS.d/next/Core and Builtins/2024-01-17-00-52-57.gh-issu…
511f59f 
…e-113884.CvEjUE.rst
@mpage
Remove unnecessary overflow check
ad5217a 
We're not using `PyThread_acquire_lock_timed`.
@mpage
Use a clearer name for field indicating whether or not threads are wa…
d3b5547 
…iting
@mpage
mpage force-pushed the gh-113884-ft-simplequeue branch from cdf0917 to d3b5547 Compare January 19, 2024 22:30
@mpage

mpage commented Jan 19, 2024

Copy link
Copy Markdown
Contributor Author

I've rebased this against main after the ring buffer refactoring was merged. @colesbury and @erlend-aasland would you please have another look?
erlend-aasland reacted with thumbs up emoji erlend-aasland reacted with rocket emoji
All reactions
  • 👍 1 reaction
  • 🚀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
colesbury
colesbury reviewed Jan 19, 2024
View reviewed changes

@colesbury colesbury left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mpage - you probably want to mark the PR as "Ready for review" (i.e., not a "Draft")

This LGTM other than one minor comment.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
mpage and erlend-aasland reacted with thumbs up emoji
All reactions
  • 👍 2 reactions
Comment thread Modules/_queuemodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
@mpage
mpage marked this pull request as ready for review January 19, 2024 23:53
@bedevere-app bedevere-app Bot added the awaiting review label Jan 19, 2024
@mpage
Fix type
6fef770
@erlend-aasland
erlend-aasland self-requested a review January 20, 2024 10:17
erlend-aasland
erlend-aasland reviewed Jan 22, 2024
View reviewed changes

@erlend-aasland erlend-aasland left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. The parking lot API still feels little bit uncomfortable for me, but AFAICS, this looks good. Perhaps a comment explaining how the hand-off data structure relates to the parking lot API could be beneficial for future readers of the code. I also find the maybe_unparked_thread oddly named. OTOH, I don't have a better suggestion :)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Modules/_queuemodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Modules/_queuemodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Modules/_queuemodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Misc/NEWS.d/next/Core and Builtins/2024-01-17-00-52-57.gh-issue-113884.CvEjUE.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.
erlend-aasland
erlend-aasland reviewed Jan 23, 2024
View reviewed changes
Comment thread Modules/_queuemodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
@erlend-aasland
Apply suggestions from code review
98dfcf6
@erlend-aasland

erlend-aasland commented Jan 23, 2024

Copy link
Copy Markdown
Contributor

@mpage, I applied my minor suggestions. Let me know what you think of #114161 (comment).
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
erlend-aasland
erlend-aasland reviewed Jan 23, 2024
View reviewed changes
Comment thread Modules/_queuemodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
erlend-aasland and others added 2 commits January 23, 2024 12:05
@erlend-aasland
Fix suggestion
60fd0ec
@mpage
Address review comments
80200f8 
Rename parking lot callback to better reflect what it does.
More precise type for whether handoff occurred.
@mpage

mpage commented Jan 23, 2024

Copy link
Copy Markdown
Contributor Author

@erlend-aasland - I think I've addressed all the comments now, can you take another look?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
erlend-aasland
erlend-aasland approved these changes Jan 23, 2024
View reviewed changes

@erlend-aasland erlend-aasland left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you so much, this is very nice!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
mpage reacted with heart emoji
All reactions
  • ❤️ 1 reaction
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting review labels Jan 23, 2024
@erlend-aasland
erlend-aasland merged commit 925907e into python:main Jan 23, 2024
@bedevere-app bedevere-app Bot removed the awaiting merge label Jan 23, 2024
@mpage
mpage deleted the gh-113884-ft-simplequeue branch January 29, 2024 18:06
aisk pushed a commit to aisk/cpython that referenced this pull request Feb 11, 2024
@mpage @erlend-aasland @aisk
pythongh-113884: Make queue.SimpleQueue thread-safe when the GIL is d…
1b9b471 
…isabled (python#114161)

* use the ParkingLot API to manage waiting threads
* use Argument Clinic's critical section directive to protect queue methods
* remove unnecessary overflow check

Co-authored-by: Erlend E. Aasland <[email protected]>
kumaraditya303
kumaraditya303 reviewed Jul 25, 2024
View reviewed changes
Comment thread Modules/_queuemodule.c
@@ -164,7 +167,7 @@ RingBuf_Put(RingBuf *buf, PyObject *item)
return -1;

@kumaraditya303 kumaraditya303 Jul 25, 2024

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to decref item here isn't it?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@colesbury colesbury Jul 25, 2024

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think so

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Glyphack pushed a commit to Glyphack/cpython that referenced this pull request Sep 2, 2024
@mpage @erlend-aasland @Glyphack
pythongh-113884: Make queue.SimpleQueue thread-safe when the GIL is d…
2adfdfc 
…isabled (python#114161)

* use the ParkingLot API to manage waiting threads
* use Argument Clinic's critical section directive to protect queue methods
* remove unnecessary overflow check

Co-authored-by: Erlend E. Aasland <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@colesbury colesbury colesbury left review comments
@kumaraditya303 kumaraditya303 kumaraditya303 left review comments
@erlend-aasland erlend-aasland erlend-aasland approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@mpage @erlend-aasland @colesbury @kumaraditya303
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.