GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — GH-104142: Fix `_Py_RefcntAdd` to respect immortality by brandtbucher · Pull Request #104143 · python/cpython

Conversation

@brandtbucher

@brandtbucher brandtbucher commented May 3, 2023 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Also, make the existing immortality tests a bit stricter and more comprehensive.

  • Issue: _Py_RefcntAdd doesn't respect immortality #104142

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
eduardo-elizondo reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
brandtbucher added 3 commits May 2, 2023 18:27
@brandtbucher
Add failing regression tests
e24771b
@brandtbucher
Fix _Py_RefcntAdd
9338eea
@brandtbucher
blurb add
be3b103
@brandtbucher brandtbucher added interpreter-core (Objects, Python, Grammar, and Parser dirs) release-blocker labels May 3, 2023
@brandtbucher
brandtbucher requested a review from ericsnowcurrently May 3, 2023 21:39
@brandtbucher brandtbucher self-assigned this May 3, 2023
@bedevere-bot bedevere-bot mentioned this pull request May 3, 2023
_Py_RefcntAdd doesn't respect immortality #104142
Closed
@bedevere-bot bedevere-bot added the awaiting core review label May 3, 2023
@brandtbucher brandtbucher added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 3, 2023
@bedevere-bot

bedevere-bot commented May 3, 2023

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @brandtbucher for commit be3b103 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 3, 2023
@ericsnowcurrently

ericsnowcurrently commented May 4, 2023

Copy link
Copy Markdown
Member

CC @eduardo-elizondo
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@brandtbucher

brandtbucher commented May 4, 2023

Copy link
Copy Markdown
Member Author

(Not sure why buildbots aren't running...)
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
corona10
corona10 previously approved these changes May 4, 2023
View reviewed changes
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting core review labels May 4, 2023
@corona10

corona10 commented May 4, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

@brandtbucher Ah would you like to add the C API test in here too?

cpython/Modules/_testcapi/immortal.c

Lines 34 to 38 in 9885677

static PyMethodDef test_methods[] = {
{"test_immortal_builtins", test_immortal_builtins, METH_NOARGS},
{"test_immortal_small_ints", test_immortal_small_ints, METH_NOARGS},
{NULL},
};

Adding a code line somewhere of here would be enough.

cpython/Modules/_testcapi/immortal.c

Lines 5 to 10 in 9885677

assert(_Py_IsImmortal(object));
Py_ssize_t old_count = Py_REFCNT(object);
for (int j = 0; j < 10000; j++) {
Py_DECREF(object);
}
Py_ssize_t current_count = Py_REFCNT(object);

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@corona10
corona10 dismissed their stale review May 4, 2023 02:11

#104143 (comment)

@bedevere-bot bedevere-bot added awaiting review and removed awaiting merge labels May 4, 2023
@corona10

corona10 commented May 4, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Ah sorry please ignore #104143 (comment)
it's not public API...
brandtbucher reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
corona10
corona10 approved these changes May 4, 2023
View reviewed changes
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting review labels May 4, 2023
eduardo-elizondo
eduardo-elizondo reviewed May 4, 2023
View reviewed changes
Comment thread Include/internal/pycore_object.h
Comment on lines +61 to +63
if (_Py_IsImmortal(op)) {
return;
}

@eduardo-elizondo eduardo-elizondo May 4, 2023

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One quick comment here is that if this were to be in a hot path, I would recommend using the same technique that we use for Py_INCREF to minimize the number of generated instructions to do the check and add: https://github.com/python/cpython/blob/main/Include/object.h#L620-L634

Now, given that it's only in use for sq_repeat in tuple and list, I don't think this is perf sensitive so keeping the code as you have it here should be good.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
brandtbucher and itamaro reacted with thumbs up emoji
All reactions
  • 👍 2 reactions
eduardo-elizondo
eduardo-elizondo reviewed May 4, 2023
View reviewed changes
Comment thread Lib/test/test_builtin.py
self.assertEqual(sys.getrefcount(immortal), self.IMMORTAL_REFCOUNT)

def test_immortals(self):
for immortal in self.IMMORTALS:

@eduardo-elizondo eduardo-elizondo May 4, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small nit: I generally prefer rolling out assertions in unit tests to make it easier to isolate by line-number in the place where the test failed. But I don't think there's a standard in the code base so I'm fine either way

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
brandtbucher reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

@brandtbucher brandtbucher May 4, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I'd like comprehensive testing of these immortals though, and I don't want to roll out ~250 different assertions for each test.

I figured making each one a subTest in assert_immortal is a pretty good compromise.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@eduardo-elizondo

eduardo-elizondo commented May 4, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Nice catch, I could have sworn that I had added the check to _Py_RefcntAdd as well but I somehow missed it. I added two comments but they are minor so this LGTM.

Thanks for the fix!
brandtbucher reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
eduardo-elizondo
eduardo-elizondo approved these changes May 4, 2023
View reviewed changes
@sunmy2019

sunmy2019 commented May 4, 2023

Copy link
Copy Markdown
Member

LGTM, all increments to ob->ob_refcnt should be guarded now.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@JelleZijlstra

JelleZijlstra commented May 4, 2023

Copy link
Copy Markdown
Member

There is a USan failure:

Include/object.h:227:12: runtime error: member access within address 0x7fff7ae78b80 with insufficient space for an object of type 'PyObject' (aka 'struct _object')
0x7fff7ae78b80: note: pointer points here
 b8 7f 00 00  00 47 a8 c2 97 55 00 00  00 b5 59 7e 78 cd 65 f5  00 8a ea c2 b8 7f 00 00  39 7a 90 c0
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Include/object.h:227:12 in 
make: *** [Makefile:1106: checksharedmods] Error 1

Not sure what that means, but the line is in _Py_IsImmortal (https://github.com/brandtbucher/cpython/blob/py-refcnt-add-immortal/Include/object.h#L227), so it may be real.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@JelleZijlstra
Merge branch 'main' into py-refcnt-add-immortal
1471b39
@JelleZijlstra JelleZijlstra added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 4, 2023
@bedevere-bot

bedevere-bot commented May 4, 2023

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @JelleZijlstra for commit 1471b39 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 4, 2023
@JelleZijlstra

JelleZijlstra commented May 4, 2023

Copy link
Copy Markdown
Member

(Trying again just in case it was something flaky.)
brandtbucher reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@brandtbucher

brandtbucher commented May 4, 2023

Copy link
Copy Markdown
Member Author

Grrr... it failed again.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@brandtbucher

brandtbucher commented May 4, 2023

Copy link
Copy Markdown
Member Author

I honestly don't know what to do here.

It feels like a bug in USan, since I honestly don't get how the new code could cause that failure. But it's reproducible, and indeed seems to indeed be related to my change here.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@brandtbucher

brandtbucher commented May 4, 2023

Copy link
Copy Markdown
Member Author

Ah, wait, the same buildbot is also failing on main: https://buildbot.python.org/all/#/builders/719/builds/2611

So it's an existing issue (but one that probably should still be looked into).
JelleZijlstra reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@brandtbucher
brandtbucher merged commit ce871fd into python:main May 5, 2023
@bedevere-bot bedevere-bot removed the awaiting merge label May 5, 2023
@sunmy2019

sunmy2019 commented May 5, 2023

Copy link
Copy Markdown
Member

So it's an existing issue (but one that probably should still be looked into).

See #104190
eduardo-elizondo reacted with heart emoji
All reactions
  • ❤️ 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@brandtbucher

brandtbucher commented May 5, 2023

Copy link
Copy Markdown
Member Author

Nice catch!
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
carljm added a commit to carljm/cpython that referenced this pull request May 5, 2023
@carljm
Merge branch 'main' into inlinecomp2
8a78a36 
* main: (61 commits)
  pythongh-64595: Argument Clinic: Touch source file if any output file changed (python#104152)
  pythongh-64631: Test exception messages in cloned Argument Clinic funcs (python#104167)
  pythongh-68395: Avoid naming conflicts by mangling variable names in Argument Clinic (python#104065)
  pythongh-64658: Expand Argument Clinic return converter docs (python#104175)
  pythonGH-103092: port `_asyncio` freelist to module state (python#104196)
  pythongh-104051: fix crash in test_xxtestfuzz with -We (python#104052)
  pythongh-104190: fix ubsan crash (python#104191)
  pythongh-104106: Add gcc fallback of mkfifoat/mknodat for macOS (pythongh-104129)
  pythonGH-104142: Fix _Py_RefcntAdd to respect immortality (pythonGH-104143)
  pythongh-104112: link from cached_property docs to method-caching FAQ (python#104113)
  pythongh-68968: Correcting message display issue with assertEqual (python#103937)
  pythonGH-103899: Provide a hint when accidentally calling a module (pythonGH-103900)
  pythongh-103963: fix 'make regen-opcode' in out-of-tree builds (python#104177)
  pythongh-102500: Add PEP 688 and 698 to the 3.12 release highlights (python#104174)
  pythonGH-81079: Add case_sensitive argument to `pathlib.Path.glob()` (pythonGH-102710)
  pythongh-91896: Deprecate collections.abc.ByteString (python#102096)
  pythongh-99593: Add tests for Unicode C API (part 2) (python#99868)
  pythongh-102500: Document PEP 688 (python#102571)
  pythongh-102500: Implement PEP 688 (python#102521)
  pythongh-96534: socketmodule: support FreeBSD divert(4) socket (python#96536)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@corona10 corona10 corona10 approved these changes
@ericsnowcurrently ericsnowcurrently Awaiting requested review from ericsnowcurrently
+1 more reviewer
@eduardo-elizondo eduardo-elizondo eduardo-elizondo approved these changes
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@brandtbucher brandtbucher

Labels

interpreter-core (Objects, Python, Grammar, and Parser dirs) release-blocker

Projects

Release and Deferred blockers 🚫
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

7 participants

@brandtbucher @bedevere-bot @ericsnowcurrently @corona10 @eduardo-elizondo @sunmy2019 @JelleZijlstra
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.