GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-116738: Make _heapq module thread-safe by yoney · Pull Request #135036 · python/cpython

Conversation

@yoney

@yoney yoney commented Jun 2, 2025 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

This uses critical sections to make heapq methods that update the heap thread-safe when the GIL is disabled. This is accomplished by using the @critical_section clinic directive.

cc: @mpage @colesbury

  • Issue: Audit all built-in modules for thread safety #116738

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@yoney
pythongh-116738: Make _heapq module thread-safe
ec3cade
@python-cla-bot

python-cla-bot Bot commented Jun 2, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot mentioned this pull request Jun 2, 2025
Audit all built-in modules for thread safety #116738
Open
@colesbury
colesbury requested review from colesbury and mpage June 2, 2025 16:06
StanFromIreland
StanFromIreland reviewed Jun 2, 2025
View reviewed changes

@StanFromIreland StanFromIreland left a comment •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a lot of duplication in tests because of min/max heaps, why not organize with subtests?

Also for testing the heap, it is probably best to reuse the existing methods from test_heapq

cpython/Lib/test/test_heapq.py

Lines 106 to 116 in 0558275

def check_invariant(self, heap):
# Check the heap invariant.
for pos, item in enumerate(heap):
if pos: # pos 0 has no parent
parentpos = (pos-1) >> 1
self.assertTrue(heap[parentpos] <= item)
def check_max_invariant(self, heap):
for pos, item in enumerate(heap[1:], start=1):
parentpos = (pos - 1) >> 1
self.assertGreaterEqual(heap[parentpos], item)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@ZeroIntensity ZeroIntensity added the topic-free-threading label Jun 2, 2025
@ZeroIntensity
ZeroIntensity self-requested a review June 2, 2025 17:10
ZeroIntensity
ZeroIntensity reviewed Jun 2, 2025
View reviewed changes

@ZeroIntensity ZeroIntensity left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would also be a good idea to get rid of the borrowed reference usage here.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

yoney reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
Comment thread Modules/_heapqmodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@yoney
yoney marked this pull request as ready for review June 2, 2025 17:38
@yoney
yoney requested a review from rhettinger as a code owner June 2, 2025 17:38
@bedevere-app bedevere-app Bot added the awaiting review label Jun 2, 2025
@colesbury colesbury added the needs backport to 3.14 bugs and security fixes label Jun 2, 2025
@colesbury

colesbury commented Jun 2, 2025

Copy link
Copy Markdown
Contributor

The Modules/_heapqmodule.c change looks good to me. I haven't looked through the tests yet. I don't think should change the implementation to avoid borrowed references: I'd rather keep the change small and limited to the thread safety fix rather than try to "clean" things up, and I'm not really convinced that avoiding borrowed references here would make things better.

@yoney - would you please add add a NEWS entry via blurb. You can use uvx blurb, if you have the uv tools, or pip install it, or blurb-it.

yoney reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ZeroIntensity

ZeroIntensity commented Jun 2, 2025

Copy link
Copy Markdown
Member

I'd rather keep the change small and limited to the thread safety fix rather than try to "clean" things up, and I'm not really convinced that avoiding borrowed references here would make things better.

Ok, but we should definitely do this in a follow-up (possibly only for 3.15). There are definitely some things here that aren't safe. For example:

lastelt = PyList_GET_ITEM(heap, n-1) ;
Py_INCREF(lastelt);
if (PyList_SetSlice(heap, n-1, n, NULL)) {
    Py_DECREF(lastelt);
    return NULL;
}
n--;

if (!n)
    return lastelt;
returnitem = PyList_GET_ITEM(heap, 0);

A finalizer could either release the critical section or explicitly clear the list, which could cause that PyList_GET_ITEM call to return NULL. I guess that's not related to borrowed references, though--more of a problem with PyList_GET_ITEM not doing validation.

There's also some incredibly horrible things going on, like this:

returnitem = PyList_GET_ITEM(heap, 0);
PyList_SET_ITEM(heap, 0, lastelt);

returnitem starts out as a borrowed reference, but then has the ownership implicitly handed off through PyList_SET_ITEM, which doesn't decref it.

yoney reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@yoney
pythongh-116738: Add news entry in Misc/NEWS/next
01b2be4
@yoney

yoney commented Jun 2, 2025

Copy link
Copy Markdown
Contributor Author

Ok, but we should definitely do this in a follow-up (possibly only for 3.15). There are definitely some things here that aren't safe.

@ZeroIntensity I agree that there are things we should follow up on. I initially tried to address some of them as part of the free-threading change, but it introduces complexity to the review and makes the free-threading change harder to review, so I decided to follow up on those issues separately.

ZeroIntensity reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ZeroIntensity
ZeroIntensity reviewed Jun 2, 2025
View reviewed changes

@ZeroIntensity ZeroIntensity left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpicks

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/test/test_free_threading/test_heapq.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_free_threading/test_heapq.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_free_threading/test_heapq.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_free_threading/test_heapq.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Misc/NEWS.d/next/Core_and_Builtins/2025-06-02-13-57-40.gh-issue-116738.ycJsL8.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

@rhettinger
rhettinger removed their request for review June 3, 2025 05:20
mpage
mpage reviewed Jun 3, 2025
View reviewed changes
Comment thread Modules/_heapqmodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

yoney added 2 commits June 3, 2025 13:05
@yoney
pythongh-116738: Address the review comments
68f3a26
@yoney
pythongh-116738: Address the review comments
41d145a
@yoney

yoney commented Jun 3, 2025

Copy link
Copy Markdown
Contributor Author

There is a lot of duplication in tests because of min/max heaps, why not organize with subtests?

@StanFromIreland Thank you so much for your review! I've already refactored the code and moved some repeated parts into separate functions while addressing the other comments. I'm not sure if subtests will provide more code reuse here. What do you think?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

StanFromIreland
StanFromIreland reviewed Jun 4, 2025
View reviewed changes
Comment thread Lib/test/test_free_threading/test_heapq.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@yoney
pythongh-116738: Remove type hints
a22da8f
ZeroIntensity
ZeroIntensity approved these changes Jun 5, 2025
View reviewed changes

@ZeroIntensity ZeroIntensity left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with two tiny complaints

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/test/test_free_threading/test_heapq.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_free_threading/test_heapq.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot added awaiting core review and removed awaiting review labels Jun 5, 2025
@yoney
pythongh-116738: Address the review comments
75a1d3a
ZeroIntensity
ZeroIntensity reviewed Jun 5, 2025
View reviewed changes
Comment thread Modules/_heapqmodule.c Outdated
if (PyList_Append(heap, item))
// In a free-threaded build, the heap is locked at this point.
// Therefore, calling _PyList_AppendTakeRef() is safe and no overhead.
if (_PyList_AppendTakeRef((PyListObject *)heap, Py_XNewRef(item)))

@ZeroIntensity ZeroIntensity Jun 5, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait, why is this now XNewRef? item can never be NULL.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@yoney yoney Jun 5, 2025

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn't sure about "item can never be NULL" and in the previous version, PyList_Append() had a NULL check. That's why I thought this would be safe. However, _PyList_AppendTakeRef() expects a non-NULL value for item so it doesn't really make it safer.

Can I assume that "item can never be NULL" because _heapq_heappush_impl() is called from clinic-generated code?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@mpage mpage Jun 5, 2025

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it might be possible for item to be NULL if you vectorcall this from a C extension? If we want to mirror the previous implementation we should add a check that item is not NULL:

if (newitem == NULL) {
    PyErr_BadInternalCall();
    return NULL
}

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

yoney reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

@ZeroIntensity ZeroIntensity Jun 5, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it might be possible for item to be NULL if you vectorcall this from a C extension?

Well, that would definitely be the user's fault--you can't pass NULL to a vectorcall arg. I think adding the NULL check here is additional work that isn't necessary. We don't do this for anything else that uses AC, right?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@mpage mpage Jun 5, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think adding the NULL check here is additional work that isn't necessary. We don't do this for anything else that uses AC, right?

The previous implementation that used PyList_Append performed the NULL check (in PyList_Append). If we want to match the existing behavior we should include the NULL check here as well. I would err on the side of being conservative and include it unless we are guaranteed to never see a NULL item (for example if something higher up in the call stack already checks for it).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@yoney yoney Jun 5, 2025

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mpage @ZeroIntensity
Thank you!
I've added the NULL check, which aligns better with the previous version. However, if we decide that we can assume the value cannot be NULL, I can remove it.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
mpage
mpage reviewed Jun 5, 2025
View reviewed changes
Comment thread Lib/test/test_free_threading/test_heapq.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Modules/_heapqmodule.c Outdated
if (PyList_Append(heap, item))
// In a free-threaded build, the heap is locked at this point.
// Therefore, calling _PyList_AppendTakeRef() is safe and no overhead.
if (_PyList_AppendTakeRef((PyListObject *)heap, Py_XNewRef(item)))

@mpage mpage Jun 5, 2025

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it might be possible for item to be NULL if you vectorcall this from a C extension? If we want to mirror the previous implementation we should add a check that item is not NULL:

if (newitem == NULL) {
    PyErr_BadInternalCall();
    return NULL
}

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

yoney reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
Comment thread Modules/_heapqmodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

yoney added 3 commits June 5, 2025 14:29
@yoney
pythongh-116738: Add NULL check for the item arg in heappush()
e8138be
@yoney
pythongh-116738: Fix typo
3f09689
@yoney
pythongh-116738: Use invariant checks from heapq test
7ffdb6d
mpage
mpage approved these changes Jun 6, 2025
View reviewed changes

@mpage mpage left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Will merge on Monday.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

yoney reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Jun 6, 2025
@mikeshardmind mikeshardmind mentioned this pull request Jun 8, 2025
Native code due to freethreading mikeshardmind/async-utils#21
Closed
@yoney
pythongh-116738: Remove forgotten type hint
c2225a5
@yoney

yoney commented Jun 9, 2025

Copy link
Copy Markdown
Contributor Author

Looks great! Will merge on Monday.

Just FYI: Made a minor update to remove the forgotten type hint.

Thank you!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mpage
Merge branch 'main' into ft_heapq
a196a00
@mpage
mpage merged commit a58026a into python:main Jun 9, 2025
@bedevere-app bedevere-app Bot removed the awaiting merge label Jun 9, 2025
@miss-islington-app

miss-islington-app Bot commented Jun 9, 2025

Copy link
Copy Markdown

Thanks @yoney for the PR, and @mpage for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jun 9, 2025
@yoney @mpage @miss-islington
pythongh-116738: Make _heapq module thread-safe (pythonGH-135036)
cece3b8
Use critical sections to make heapq methods that update the heap thread-safe when the GIL is disabled.

---------
(cherry picked from commit a58026a)

Co-authored-by: Alper <[email protected]>
Co-authored-by: mpage <[email protected]>
@bedevere-app

bedevere-app Bot commented Jun 9, 2025

Copy link
Copy Markdown

GH-135309 is a backport of this pull request to the 3.14 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label Jun 9, 2025
@yoney
yoney deleted the ft_heapq branch June 16, 2025 05:59
@kumaraditya303 kumaraditya303 mentioned this pull request Jun 16, 2025
data race between lock free list reads and heapq #135557
Closed
colesbury pushed a commit that referenced this pull request Jun 16, 2025
@miss-islington @yoney @mpage
[3.14] gh-116738: Make _heapq module thread-safe (GH-135036) (gh-135309)
964c29d
Use critical sections to make heapq methods that update the heap thread-safe when the GIL is disabled.
(cherry picked from commit a58026a)

Co-authored-by: Alper <[email protected]>
Co-authored-by: mpage <[email protected]>
lkollar pushed a commit to lkollar/cpython that referenced this pull request Jun 19, 2025
@yoney @mpage @lkollar
pythongh-116738: Make _heapq module thread-safe (python#135036)
1c5f6b1
Use critical sections to make heapq methods that update the heap thread-safe when the GIL is disabled.

---------

Co-authored-by: mpage <[email protected]>
Pranjal095 pushed a commit to Pranjal095/cpython that referenced this pull request Jul 12, 2025
@yoney @mpage @Pranjal095
pythongh-116738: Make _heapq module thread-safe (python#135036)
da8c56f
Use critical sections to make heapq methods that update the heap thread-safe when the GIL is disabled.

---------

Co-authored-by: mpage <[email protected]>
taegyunkim pushed a commit to taegyunkim/cpython that referenced this pull request Aug 4, 2025
@yoney @mpage @taegyunkim
pythongh-116738: Make _heapq module thread-safe (python#135036)
621b723
Use critical sections to make heapq methods that update the heap thread-safe when the GIL is disabled.

---------

Co-authored-by: mpage <[email protected]>
Agent-Hellboy pushed a commit to Agent-Hellboy/cpython that referenced this pull request Aug 19, 2025
@yoney @mpage @Agent-Hellboy
pythongh-116738: Make _heapq module thread-safe (python#135036)
2afb521
Use critical sections to make heapq methods that update the heap thread-safe when the GIL is disabled.

---------

Co-authored-by: mpage <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ZeroIntensity ZeroIntensity ZeroIntensity approved these changes
@AlexWaygood AlexWaygood AlexWaygood left review comments
@StanFromIreland StanFromIreland StanFromIreland left review comments
@mpage mpage mpage approved these changes
@colesbury colesbury Awaiting requested review from colesbury

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

topic-free-threading

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@yoney @colesbury @ZeroIntensity @mpage @AlexWaygood @StanFromIreland
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.