GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-133164: Add `PyUnstable_Object_IsUniqueReferencedTemporary` C API by colesbury · Pull Request #133170 · python/cpython

Conversation

@colesbury

@colesbury colesbury commented Apr 29, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

After gh-130704, the interpreter replaces some uses of LOAD_FAST with LOAD_FAST_BORROW which avoid incref/decrefs by "borrowing" references on the interpreter stack when the bytecode compiler can determine that its safe.

This change broke some checks in C API extensions that relied on Py_REFCNT() of 1 to determine if it's safe to modify an object in-place. Objects may have a reference count of one, but still be referenced further up the interpreter stack due to borrowing of references.

This provides a replacement function for those checks. PyUnstable_Object_IsUniqueReferencedTemporary is more conservative: it checks that the object has a reference count of one and that it exists as a unique strong reference in the interpreter's stack of temporary variables in the top most frame.

See also:

  • BUG: temporary elision heuristics are broken on Python 3.14 numpy/numpy#28681
  • Issue: Add a C API function to detect temporaries #133164

📚 Documentation preview 📚:

  • https://cpython-previews--133170.org.readthedocs.build/en/133170/c-api/object.html#c.PyUnstable_Object_IsUniqueReferencedTemporary
  • https://cpython-previews--133170.org.readthedocs.build/en/133170/whatsnew/3.14.html#incompatible-changes
  • https://cpython-previews--133170.org.readthedocs.build/en/133170/whatsnew/3.14.html#whatsnew314-refcount

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@colesbury
pythongh-133164: Add PyUnstable_Object_IsUniqueTemporary C API
32b6cd0
After pythongh-130704, the interpreter replaces some uses of `LOAD_FAST` with
`LOAD_FAST_BORROW` which avoid incref/decrefs by "borrowing" references
on the interpreter stack when the bytecode compiler can determine that
its safe.

This change broke some checks in C API extensions that relied on
`Py_REFCNT()` of `1` to determine if it's safe to modify an object
in-place. Objects may have a reference count of one, but still be
referenced further up the interpreter stack due to borrowing of
references.

This provides a replacement function for those checks.
`PyUnstable_Object_IsUniqueTemporary` is more conservative: it checks
that the object has a reference count of one and that it exists as a
unique strong reference in the interpreter's stack of temporary
variables in the top most frame.

See also:

* numpy/numpy#28681
@bedevere-app bedevere-app Bot added the awaiting core review label Apr 29, 2025
@bedevere-app bedevere-app Bot mentioned this pull request Apr 29, 2025
Add a C API function to detect temporaries #133164
Closed
@colesbury
Make the asserts use consistent APIs
479f185
ZeroIntensity
ZeroIntensity reviewed Apr 29, 2025
View reviewed changes
Comment thread Doc/c-api/object.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Misc/NEWS.d/next/C_API/2025-04-29-19-39-16.gh-issue-133164.W-XTU7.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Objects/object.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

colesbury added 2 commits April 29, 2025 20:26
@colesbury
Minor edits
4a53a98
@colesbury
Add what's new
fdc438a
@colesbury
colesbury requested review from Yhg1s, mpage and pablogsal April 29, 2025 21:09
@colesbury

colesbury commented Apr 29, 2025

Copy link
Copy Markdown
Contributor Author

@ngoldbaum - I can't add you as a reviewer, but I'd appreciate it if you could take a look at this when you get a chance.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

mpage
mpage approved these changes Apr 29, 2025
View reviewed changes
Comment thread Doc/whatsnew/3.14.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Apr 29, 2025
eendebakpt
eendebakpt reviewed Apr 29, 2025
View reviewed changes
Comment thread Doc/c-api/object.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

eendebakpt
eendebakpt reviewed Apr 29, 2025
View reviewed changes
Comment thread Doc/c-api/object.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

eendebakpt
eendebakpt reviewed Apr 29, 2025
View reviewed changes
Comment thread Include/cpython/object.h Resolved

Uh oh!

There was an error while loading. Please reload this page.

Yhg1s
Yhg1s reviewed Apr 29, 2025
View reviewed changes
Comment thread Doc/c-api/object.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/c-api/object.rst Outdated
temporary object.

If an object is a unique temporary, it is guaranteed that the reference
count is ``1`` and it may be safe to modify the object in-place becuase

@Yhg1s Yhg1s Apr 29, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think "may be safe" doesn't quite convey what we want here. We don't want people to be in doubt about whether it's safe from the interpreter's point of view. Maybe something along the lines of "it is guaranteed that the current code owns the only reference to the object"?

It's probably also worth mentioning why a refcount of 1 isn't enough to guarantee it's a unique temporary, something about the interpreter borrowing references internally, similar to the whatsnew entry.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

colesbury reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
Comment thread Lib/test/test_capi/test_object.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

colesbury and others added 2 commits April 29, 2025 20:04
@colesbury @eendebakpt
@Yhg1s @mpage
Apply suggestions from code review
b6508df
Co-authored-by: Pieter Eendebak <[email protected]>
Co-authored-by: T. Wouters <[email protected]>
Co-authored-by: mpage <[email protected]>
@colesbury
Edit docs
07ba540
markshannon
markshannon requested changes Apr 30, 2025
View reviewed changes

@markshannon markshannon left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer PyUnstable_Object_IsUniqueReferencedTemporary as it is not the object that is unique, but the reference to it.

The code seems correct, but could be more efficient. If the total explicit refcount is 1, then the first non-borrowed reference to the object must be the only reference.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Doc/c-api/object.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/c-api/object.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Objects/object.c
int
PyUnstable_Object_IsUniqueTemporary(PyObject *op)
{
if (!_PyObject_IsUniquelyReferenced(op)) {

@markshannon markshannon Apr 30, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this just demonstrates that _PyObject_IsUniquelyReferenced is badly named, otherwise we wouldn't need a new function.

We need to check that the following are true:

  • The object has an explicit (not borrowed) reference count of 1
  • It is mortal
  • It does not support deferred reclamation.

I think _PyObject_IsUniquelyReferenced does that, but it isn't clear from the name.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@colesbury colesbury Apr 30, 2025

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's discuss the name for _PyObject_IsUniquelyReferenced in #133144

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ZeroIntensity reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
Comment thread Objects/object.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Objects/object.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Objects/object.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot added awaiting changes and removed awaiting merge labels Apr 30, 2025
@bedevere-app

bedevere-app Bot commented Apr 30, 2025

Copy link
Copy Markdown

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ZeroIntensity ZeroIntensity mentioned this pull request Apr 30, 2025
Adding an unstable C API for unique references #133140
Closed
@colesbury @markshannon
Apply suggestions from code review
99e3265
Co-authored-by: Mark Shannon <[email protected]>
@colesbury colesbury changed the title gh-133164: Add PyUnstable_Object_IsUniqueTemporary C API gh-133164: Add PyUnstable_Object_IsUniqueReferencedTemporary C API Apr 30, 2025
colesbury added 2 commits April 30, 2025 14:20
@colesbury
Rename function and simplify implementation
0e0e674
@colesbury
Merge branch 'main' into pythongh-133164-unique-temporary
07f3b23
@colesbury
colesbury requested a review from markshannon April 30, 2025 14:38
ngoldbaum
ngoldbaum reviewed Apr 30, 2025
View reviewed changes

@ngoldbaum ngoldbaum left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As expected, the NumPy tests pass if I replace the python internals usage with this function:

Details
diff --git a/numpy/_core/src/multiarray/temp_elide.c b/numpy/_core/src/multiarray/temp_elide.c
index 667ba2b443..ee1194b3a0 100644
--- a/numpy/_core/src/multiarray/temp_elide.c
+++ b/numpy/_core/src/multiarray/temp_elide.c
@@ -62,13 +62,6 @@
 
 #include <feature_detection_misc.h>
 
-#if PY_VERSION_HEX >= 0x030E00A7 && !defined(PYPY_VERSION)
-#define Py_BUILD_CORE
-#include "internal/pycore_frame.h"
-#include "internal/pycore_interpframe.h"
-#undef Py_BUILD_CORE
-#endif
-
 /* 1 prints elided operations, 2 prints stacktraces */
 #define NPY_ELIDE_DEBUG 0
 #define NPY_MAX_STACKSIZE 10
@@ -124,28 +117,7 @@ check_unique_temporary(PyObject *lhs)
     // that an object is a unique temporary variable. We scan the top of the
     // interpreter stack to check if the object exists as an "owned" reference
     // in the temporary stack.
-    PyFrameObject *frame = PyEval_GetFrame();
-    if (frame == NULL) {
-        return 0;
-    }
-    _PyInterpreterFrame *f = frame->f_frame;
-    _PyStackRef *base = _PyFrame_Stackbase(f);
-    _PyStackRef *stackpointer = f->stackpointer;
-    int found_once = 0;
-    while (stackpointer > base) {
-        stackpointer--;
-        PyObject *obj = PyStackRef_AsPyObjectBorrow(*stackpointer);
-        if (obj == lhs) {
-            if (!found_once && PyStackRef_IsHeapSafe(*stackpointer)) {
-                found_once = 1;
-            }
-            else {
-                return 0;
-            }
-        }
-        return found_once;
-    }
-    return 0;
+    return PyUnstable_Object_IsUniqueReferencedTemporary(lhs);
 #else
     return 1;
 #endif

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

corona10 and vstinner reacted with thumbs up emoji
All reactions
  • 👍 2 reactions
Comment thread Doc/whatsnew/3.14.rst Outdated
compared to previous Python versions. C API extensions that checked
:c:func:`Py_REFCNT` of ``1`` to determine if an function argument is not
referenced by any other code should instead use
:c:func:`PyUnstable_Object_IsUniqueTemporary` as a safer replacement.

@ngoldbaum ngoldbaum Apr 30, 2025

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fallout of this is a bit wider than the need for this function, e.g. the Pandas check that broke was looking at if sys.refcount(obj) <= 3. There were also some NumPy and nanobind tests that failed because the absolute values of reference counts changed.

https://github.com/pandas-dev/pandas/blob/main/pandas/core/frame.py#L4156-L4161

(c.f. pandas-dev/pandas#61368).

I'm actually not totally sure why Pandas was checking for <= 3 in particular and if there's a way for them to implement that check in pure Python still.

In NumPy we switched all the refcount tests to check changes in refcounts before and after an operation rather than absolute refcount values. That works across Python versions.

Maybe a separate paragraph advising against using the absolute values of reference counts in tests or runtime sanity checks? Could be a followup too.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@colesbury colesbury May 2, 2025

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think we should do another pass over this and the Py_REFCNT docs after we add PyUnstable_Object_IsUniquelyReferenced in #133144

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ngoldbaum reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
pablogsal
pablogsal reviewed Apr 30, 2025
View reviewed changes
Comment thread Doc/whatsnew/3.14.rst
:mod:`multiprocessing` or :mod:`concurrent.futures`, see the
:ref:`forkserver restrictions <multiprocessing-programming-forkserver>`.

The interpreter avoids some reference count modifications internally when

@pablogsal pablogsal Apr 30, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👌

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@nascheme nascheme mentioned this pull request Apr 30, 2025
use Py_REFCNT(obj) instead of obj->ob_refcnt (allows building in nogil) indygreg/python-zstandard#201
Closed
@colesbury

colesbury commented May 1, 2025

Copy link
Copy Markdown
Contributor Author

@markshannon - would you please take another look at this?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner
vstinner reviewed May 1, 2025
View reviewed changes
Comment thread Doc/c-api/object.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/c-api/object.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

colesbury and others added 3 commits May 1, 2025 10:48
@colesbury @vstinner
Update Doc/c-api/object.rst
bba589f
Co-authored-by: Victor Stinner <[email protected]>
@colesbury
Add reference to Py_REFCNT
f23de33
@colesbury
Add link to PyUnstable_Object_IsUniqueReferencedTemporary in Py_REFCNT
6efe2fb
markshannon
markshannon approved these changes May 2, 2025
View reviewed changes

@markshannon markshannon left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting changes labels May 2, 2025
@colesbury colesbury changed the title gh-133164: Add PyUnstable_Object_IsUniqueReferencedTemporary C API gh-133164: Add PyUnstable_Object_IsUniqueReferencedTemporary C API May 2, 2025
@colesbury
Merge branch 'main' into pythongh-133164-unique-temporary
f30ccf4
@colesbury
colesbury enabled auto-merge (squash) May 2, 2025 13:01
@colesbury
colesbury merged commit f237953 into python:main May 2, 2025
@bedevere-app bedevere-app Bot removed the awaiting merge label May 2, 2025
@ngoldbaum

ngoldbaum commented May 2, 2025

Copy link
Copy Markdown
Contributor

Awesome! As soon as beta 1 is available for CI I’ll drop support for the 3.14 alpha releases and update NumPy to use this.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Pranjal095 pushed a commit to Pranjal095/cpython that referenced this pull request Jul 12, 2025
@colesbury @eendebakpt
@Yhg1s @mpage @markshannon @vstinner @Pranjal095
pythongh-133164: Add PyUnstable_Object_IsUniqueReferencedTemporary …
ba5b945
…C API (pythongh-133170)

After pythongh-130704, the interpreter replaces some uses of `LOAD_FAST` with
`LOAD_FAST_BORROW` which avoid incref/decrefs by "borrowing" references
on the interpreter stack when the bytecode compiler can determine that
it's safe.

This change broke some checks in C API extensions that relied on
`Py_REFCNT()` of `1` to determine if it's safe to modify an object
in-place. Objects may have a reference count of one, but still be
referenced further up the interpreter stack due to borrowing of
references.

This provides a replacement function for those checks.
`PyUnstable_Object_IsUniqueReferencedTemporary` is more conservative:
it checks that the object has a reference count of one and that it exists as a
unique strong reference in the interpreter's stack of temporary
variables in the top most frame.

See also:

* numpy/numpy#28681

Co-authored-by: Pieter Eendebak <[email protected]>
Co-authored-by: T. Wouters <[email protected]>
Co-authored-by: mpage <[email protected]>
Co-authored-by: Mark Shannon <[email protected]>
Co-authored-by: Victor Stinner <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner vstinner left review comments
@eendebakpt eendebakpt eendebakpt left review comments
@ngoldbaum ngoldbaum ngoldbaum left review comments
@Yhg1s Yhg1s Yhg1s left review comments
@pablogsal pablogsal pablogsal left review comments
@ZeroIntensity ZeroIntensity ZeroIntensity left review comments
@mpage mpage mpage approved these changes
@markshannon markshannon markshannon approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

9 participants

@colesbury @ngoldbaum @vstinner @mpage @eendebakpt @Yhg1s @markshannon @pablogsal @ZeroIntensity
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.