GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-104252: Immortalize Py_EMPTY_KEYS by ericsnowcurrently · Pull Request #104253 · python/cpython

Conversation

@ericsnowcurrently

@ericsnowcurrently ericsnowcurrently commented May 6, 2023 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

This was missed in gh-19474. It matters for with a per-interpreter GIL since PyDictKeysObject.dk_refcnt breaks isolation and leads to races.

cc @eduardo-elizondo

  • Issue: Immortalize Py_EMPTY_KEYS #104252

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
ericsnowcurrently added 2 commits May 6, 2023 17:55
@ericsnowcurrently
Factor out _dict_state_INIT.
5bc2469
@ericsnowcurrently
Immortalize Py_EMPTY_KEYS.
e472d94
@ericsnowcurrently ericsnowcurrently added the skip news label May 6, 2023
@ericsnowcurrently
ericsnowcurrently requested a review from markshannon May 6, 2023 23:59
@ericsnowcurrently
ericsnowcurrently requested a review from methane as a code owner May 6, 2023 23:59
@bedevere-bot bedevere-bot mentioned this pull request May 6, 2023
Immortalize Py_EMPTY_KEYS #104252
Closed
@bedevere-bot bedevere-bot added the awaiting core review label May 6, 2023
@ericsnowcurrently
ericsnowcurrently requested review from methane and removed request for methane May 6, 2023 23:59
sunmy2019
sunmy2019 requested changes May 7, 2023
View reviewed changes

@sunmy2019 sunmy2019 left a comment •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are other places with direct changes to dk_refcnt. They are not guarded.

cpython/Objects/dictobject.c

Line 639 in 4729383

dk->dk_refcnt = 1;

cpython/Objects/dictobject.c

Lines 822 to 828 in 4729383

/* Since we copied the keys table we now have an extra reference
in the system. Manually call increment _Py_RefTotal to signal that
we have it now; calling dictkeys_incref would be an error as
keys->dk_refcnt is already set to 1 (after memcpy). */
#ifdef Py_REF_DEBUG
_Py_IncRefTotal(_PyInterpreterState_GET());
#endif

cpython/Objects/dictobject.c

Lines 1530 to 1538 in 4729383

// We can not use free_keys_object here because key's reference
// are moved already.
#ifdef Py_REF_DEBUG
_Py_DecRefTotal(_PyInterpreterState_GET());
#endif
if (oldkeys == Py_EMPTY_KEYS) {
oldkeys->dk_refcnt--;
assert(oldkeys->dk_refcnt > 0);
}

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Objects/dictobject.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
@ericsnowcurrently

ericsnowcurrently commented May 8, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

@sunmy2019, thanks for pointing out the other uses of dk_refcnt. I'll fix that.

[done]
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
ericsnowcurrently added 2 commits May 8, 2023 07:57
@ericsnowcurrently
Handle other uses of dk_refcnt.
87351e0
@ericsnowcurrently
Add a note.
f5da088
methane
methane reviewed May 8, 2023
View reviewed changes
Comment thread Objects/dictobject.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
sunmy2019
sunmy2019 reviewed May 8, 2023
View reviewed changes
Comment thread Include/internal/pycore_dict_state.h Resolved

Uh oh!

There was an error while loading. Please reload this page.
ericsnowcurrently added 2 commits May 8, 2023 13:25
@ericsnowcurrently
Drop an old line.
db8612f
@ericsnowcurrently
Drop an unnecessary condition.
58585a9
methane
methane reviewed May 9, 2023
View reviewed changes
Comment thread Objects/dictobject.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
methane
methane reviewed May 9, 2023
View reviewed changes
Comment thread Objects/dictobject.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
@methane

methane commented May 9, 2023

Copy link
Copy Markdown
Member

Please remove these lines too:

cpython/Objects/dictobject.c

Line 849 in 58585a9

dictkeys_incref(Py_EMPTY_KEYS);

cpython/Objects/dictobject.c

Line 1347 in 58585a9

dictkeys_decref(interp, Py_EMPTY_KEYS);

And Move _Py_DecRefTotal() into if (oldkeys != Py_EMPTY_KEYS) block.

cpython/Objects/dictobject.c

Lines 1545 to 1548 in 58585a9

#ifdef Py_REF_DEBUG
_Py_DecRefTotal(_PyInterpreterState_GET());
#endif
if (oldkeys != Py_EMPTY_KEYS) {

ericsnowcurrently reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ericsnowcurrently

ericsnowcurrently commented May 9, 2023

Copy link
Copy Markdown
Member Author

Thanks for the feedback. I'll clean things up tomorrow.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
ericsnowcurrently added 4 commits May 9, 2023 10:11
@ericsnowcurrently
Drop an unnecessary assert.
6b5c952
@ericsnowcurrently
Do not bother inc/dec-refing Py_EMPTY_KEYS.
bdbce5f
@ericsnowcurrently
Do not update Py_Reftotal for Py_EMPTY_KEYS.
aab493a
@ericsnowcurrently
Clarify the comment about dictkeys_incref() and dictkeys_decref().
5aabfd7
@ericsnowcurrently
ericsnowcurrently requested a review from sunmy2019 May 9, 2023 17:45
@sunmy2019 sunmy2019 added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label May 9, 2023
@bedevere-bot

bedevere-bot commented May 9, 2023

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @sunmy2019 for commit 5aabfd7 🤖

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label May 9, 2023
methane
methane approved these changes May 10, 2023
View reviewed changes
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting core review labels May 10, 2023
sunmy2019

This comment was marked as duplicate.

Sign in to view

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
sunmy2019
sunmy2019 approved these changes May 10, 2023
View reviewed changes

@sunmy2019 sunmy2019 left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! now it's much cleaner

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@ericsnowcurrently
ericsnowcurrently merged commit b8f7ab5 into python:main May 10, 2023
@bedevere-bot bedevere-bot removed the awaiting merge label May 10, 2023
@ericsnowcurrently
ericsnowcurrently deleted the immortalize-empty-keys branch May 10, 2023 13:28
@ericsnowcurrently

ericsnowcurrently commented May 10, 2023

Copy link
Copy Markdown
Member Author

Thanks for the reviews!
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
carljm added a commit to carljm/cpython that referenced this pull request May 10, 2023
@carljm
Merge branch 'main' into fixloadsuperspec
5976df9 
* main:
  pythonGH-102181: Improve specialization stats for SEND (pythonGH-102182)
  pythongh-103000: Optimise `dataclasses.asdict` for the common case (python#104364)
  pythongh-103538: Remove unused TK_AQUA code (pythonGH-103539)
  pythonGH-87695: Fix OSError from `pathlib.Path.glob()` (pythonGH-104292)
  pythongh-104263: Rely on Py_NAN and introduce Py_INFINITY (pythonGH-104202)
  pythongh-104010: Separate and improve docs for `typing.get_origin` and `typing.get_args` (python#104013)
  pythongh-101819: Adapt _io._BufferedIOBase_Type methods to Argument Clinic (python#104355)
  pythongh-103960: Dark mode: invert image brightness (python#103983)
  pythongh-104252: Immortalize Py_EMPTY_KEYS (pythongh-104253)
  pythongh-101819: Clean up _io windows console io after pythongh-104197 (python#104354)
  pythongh-101819: Harden _io init (python#104352)
  pythongh-103247: clear the module cache in a test in test_importlib/extensions/test_loader.py (pythonGH-104226)
  pythongh-103848: Adds checks to ensure that bracketed hosts found by urlsplit are of IPv6 or IPvFuture format (python#103849)
  pythongh-74895: adjust tests to work on Solaris (python#104326)
  pythongh-101819: Refactor _io in preparation for module isolation (python#104334)
  pythongh-90953: Don't use deprecated AST nodes in clinic.py (python#104322)
  pythongh-102327: Extend docs for "url" and "headers" parameters to HTTPConnection.request()
  pythongh-104328: Fix typo in ``typing.Generic`` multiple inheritance error message (python#104335)
carljm added a commit to carljm/cpython that referenced this pull request May 11, 2023
@carljm
Merge branch 'main' into supermon
19b3d9e 
* main: (27 commits)
  pythongh-87849: fix SEND specialization family definition (pythonGH-104268)
  pythongh-101819: Adapt _io.IOBase.seek and _io.IOBase.truncate to Argument Clinic (python#104384)
  pythongh-101819: Adapt _io._Buffered* methods to Argument Clinic (python#104367)
  pythongh-101819: Refactor `_io` futher in preparation for module isolation (python#104369)
  pythongh-101819: Adapt _io.TextIOBase methods to Argument Clinic (python#104383)
  pythongh-101117: Improve accuracy of sqlite3.Cursor.rowcount docs (python#104287)
  pythonGH-92184: Convert os.altsep to '/' in filenames when creating ZipInfo objects (python#92185)
  pythongh-104357: fix inlined comprehensions that close over iteration var (python#104368)
  pythonGH-90208: Suppress OSError exceptions from `pathlib.Path.glob()` (pythonGH-104141)
  pythonGH-102181: Improve specialization stats for SEND (pythonGH-102182)
  pythongh-103000: Optimise `dataclasses.asdict` for the common case (python#104364)
  pythongh-103538: Remove unused TK_AQUA code (pythonGH-103539)
  pythonGH-87695: Fix OSError from `pathlib.Path.glob()` (pythonGH-104292)
  pythongh-104263: Rely on Py_NAN and introduce Py_INFINITY (pythonGH-104202)
  pythongh-104010: Separate and improve docs for `typing.get_origin` and `typing.get_args` (python#104013)
  pythongh-101819: Adapt _io._BufferedIOBase_Type methods to Argument Clinic (python#104355)
  pythongh-103960: Dark mode: invert image brightness (python#103983)
  pythongh-104252: Immortalize Py_EMPTY_KEYS (pythongh-104253)
  pythongh-101819: Clean up _io windows console io after pythongh-104197 (python#104354)
  pythongh-101819: Harden _io init (python#104352)
  ...
@eduardo-elizondo

eduardo-elizondo commented Jun 19, 2023

Copy link
Copy Markdown
Contributor

Nice fix! I just got around to looking at this PR, it was weird thinking about the PyDictKeysObject since they were not PyObjects, so I just let them be. That being said, this LGTM specially if we are trying to share as much state as possible.
ericsnowcurrently reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@methane methane methane approved these changes
@sunmy2019 sunmy2019 sunmy2019 approved these changes
@markshannon markshannon Awaiting requested review from markshannon markshannon is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@ericsnowcurrently @methane @bedevere-bot @eduardo-elizondo @sunmy2019
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.