GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-120906: Support arbitrary hashable keys in FrameLocalsProxy by encukou · Pull Request #122309 · python/cpython

Conversation

@encukou

@encukou encukou commented Jul 26, 2024 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Make FrameLocalsProxy support arbitrary keys for lookup and update.

The given key needs to be hashable. It's a mapping, after all.
For the linear scan through the "real" locals:

  • If the given key is an interned exact string, we use pointer comparison
  • Otherwise, we fall back to generic PyObject_RichCompareBool
    If there's no match, we know it matches none of the "real" locals,
    and the read/write functions fall back to the "extras" dict.

Add tests, including tests for the mapping protocol (with selected exceptions).

  • Issue: FrameLocalsProxy is stricter than dict about what constitutes a match #120906

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
encukou added 2 commits July 26, 2024 13:56
@encukou
pythongh-120906: Support arbitrary hashable keys in FrameLocalsProxy
56b7a2d
@encukou
Run mapping protocol tests on FrameLocalsProxy
791e4fc
@encukou
encukou requested a review from gaogaotiantian July 26, 2024 13:35
@encukou
encukou requested a review from markshannon as a code owner July 26, 2024 13:35
@bedevere-app bedevere-app Bot mentioned this pull request Jul 26, 2024
FrameLocalsProxy is stricter than dict about what constitutes a match #120906
Closed
@bedevere-app bedevere-app Bot added the awaiting core review label Jul 26, 2024
@gaogaotiantian

gaogaotiantian commented Jul 26, 2024

Copy link
Copy Markdown
Member

So for the first glance, there are two questions:

  1. What if the key equals to a unicode string, but has a different hash? I know it's not "supposed" to be, but that could happen and this implementation has a different behavior than a real dict.
  2. The separate loop was intentional - do a quick pointer equality check first, then do the slow check. It will save us some time. I think we should keep that.

I did no go through the whole change because I don't believe we've reached to the point where we agree how to do this.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

encukou added 2 commits July 27, 2024 08:33
@encukou
Revert to two loops
03dc148
@encukou
Check that the hash matches
ddc4384
@encukou

encukou commented Jul 27, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author
  1. I don't think we need to match dict in this anomalous case.
    The dict behaviour itself depends on internal implementation details one shouldn't rely on.

    However, we can use the hash to skip PyObject_RichCompareBool calls, which seems beneficial on its own.
    (But it doesn't match dict behaviour exactly. Dict doesn't use the full hash; how much of the hash it uses depends on the internal table size.)

  2. OK. (I'd be surprised if the speedup didn't matter in real-world use cases, so I went for simpler code before.)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

picnixz
picnixz reviewed Jul 27, 2024
View reviewed changes
Comment thread Lib/test/test_frame.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_frame.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

ncoghlan
ncoghlan approved these changes Jul 27, 2024
View reviewed changes

@ncoghlan ncoghlan left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

General approach LGTM, some suggestions inline regarding test case details.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/test/test_frame.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_frame.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_frame.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_frame.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Jul 27, 2024
@encukou @ncoghlan
Apply test suggestions from code review
4bba239
Co-authored-by: Alyssa Coghlan <[email protected]>
picnixz
picnixz approved these changes Jul 27, 2024
View reviewed changes
gaogaotiantian
gaogaotiantian reviewed Jul 28, 2024
View reviewed changes
Comment thread Objects/frameobject.c
for (int i = 0; i < co->co_nlocalsplus; i++) {
PyObject *name = PyTuple_GET_ITEM(co->co_localsplusnames, i);
if (name == key) {
found_key = true;

@gaogaotiantian gaogaotiantian Jul 28, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you removed found_key check here, maybe just return -1 if the condition does not meet. Otherwise it will go through the loop again and check something we already knew.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

encukou reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

@gaogaotiantian gaogaotiantian Jul 28, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rest of the PR LGTM.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@encukou encukou Jul 29, 2024

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, thanks for the catch!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@encukou encukou Jul 29, 2024

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you check the change?
I'm still not too familiar with the internals of this mapping; I just found out (hopefully correctly) what the undocumented CO_FAST_HIDDEN does.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@ncoghlan ncoghlan Jul 30, 2024

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks correct to me (with the search flag restored, the CO_FAST_HIDDEN handling is unchanged from @gaogaotiantian's original PR).

I did suggest an explanatory comment on the early return, but that's definitely not a merge blocker. So even if @gaogaotiantian doesn't get time to take a look, I think this is good to go for rc1.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@ncoghlan ncoghlan Jul 30, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capturing my understanding of the CO_FAST_HIDDEN handling, since it's genuinely not clear:

  • in Python 3.12, the iteration variables for inlined comprehensions are visible for reads while the comprehension(s) is(/are) running, but not writable (even with LocalsToFast shenanigans), and not visible outside the comprehension(s)
  • in Python 3.13, all of those properties remain true even though sys._getframe().f_locals produces fast locals proxy instances during iteration (writes to hidden variables go into the frame's "extra locals" storage rather than modifying the comprehension iteration variable)

Future versions could revisit that design if debugger developers ask for it to change, but live editing loop variables seems sufficiently fraught with peril that I'll be genuinely surprised if anyone ever notices this small discrepancy between module level comprehensions and function level ones.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
encukou added 2 commits July 29, 2024 15:13
@encukou
Early return for NULL/hidden values
d37d8f2
@encukou
Handle hidden values in the short-circuit
f5df751
ncoghlan
ncoghlan reviewed Jul 30, 2024
View reviewed changes
Comment thread Objects/frameobject.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@encukou @ncoghlan
Add comment in Objects/frameobject.c
34c3d24
Co-authored-by: Alyssa Coghlan <[email protected]>
@encukou

encukou commented Jul 30, 2024

Copy link
Copy Markdown
Member Author

@gaogaotiantian and/or @markshannon, your review would be appreciated, so we can get this in 3.13.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@markshannon

markshannon commented Jul 30, 2024

Copy link
Copy Markdown
Member

Looks good.

There is one test I'd like to added, though.
This little program now succeeds (it fails on main):

class Str(str):
    pass

def foo():
    sys._getframe().f_locals[Str("x")] = 2
    assert x == 2
    x = 1

foo()

but it doesn't seem like there is a test for this case (writing a fast local using a non-str key).
Could you add a test for this?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@encukou

encukou commented Jul 30, 2024

Copy link
Copy Markdown
Member Author

Will do in ~6 hours, feel free to beat me to it.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

gaogaotiantian
gaogaotiantian approved these changes Jul 30, 2024
View reviewed changes
encukou added 2 commits July 30, 2024 23:45
@encukou
Test setting an unintialized local through the proxy
7d3e8b5
@encukou
Merge branch 'main' into frame_locals_eq
6927ae0
@encukou

encukou commented Jul 30, 2024

Copy link
Copy Markdown
Member Author

Test added. (There was one for writing a fast local using a non-str key, but not for writing a so far uninitialized local like your example, so I added that.)

Thank you for the reviews!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@encukou
encukou enabled auto-merge (squash) July 30, 2024 22:06
@encukou
encukou merged commit 5912487 into python:main Jul 30, 2024
@bedevere-app bedevere-app Bot removed the awaiting merge label Jul 30, 2024
@encukou encukou added the needs backport to 3.13 only security fixes label Jul 30, 2024
@miss-islington-app

miss-islington-app Bot commented Jul 30, 2024

Copy link
Copy Markdown

Thanks @encukou for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@miss-islington-app

miss-islington-app Bot commented Jul 30, 2024

Copy link
Copy Markdown

Sorry, @encukou, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 5912487938ac4b517209082ab9e6d2d3d0fb4f4d 3.13
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@miss-islington-app miss-islington-app Bot assigned encukou Jul 30, 2024
@encukou
encukou deleted the frame_locals_eq branch July 30, 2024 22:22
encukou added a commit to encukou/cpython that referenced this pull request Jul 30, 2024
@encukou @ncoghlan
[3.13] pythongh-120906: Support arbitrary hashable keys in FrameLocal…
58bf6e4
…sProxy (pythonGH-122309)

Co-authored-by: Alyssa Coghlan <[email protected]>
(cherry picked from commit 5912487)
encukou added a commit to encukou/cpython that referenced this pull request Jul 30, 2024
@encukou @ncoghlan
[3.13] pythongh-120906: Support arbitrary hashable keys in FrameLocal…
a0ceae0
…sProxy (pythonGH-122309)

Co-authored-by: Alyssa Coghlan <[email protected]>
(cherry picked from commit 5912487)
@bedevere-app

bedevere-app Bot commented Jul 30, 2024

Copy link
Copy Markdown

GH-122488 is a backport of this pull request to the 3.13 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the needs backport to 3.13 only security fixes label Jul 30, 2024
Yhg1s pushed a commit that referenced this pull request Jul 31, 2024
@encukou @ncoghlan
[3.13] gh-120906: Support arbitrary hashable keys in FrameLocalsProxy (…
8c8c43e
…GH-122309) (#122488)

[3.13] gh-120906: Support arbitrary hashable keys in FrameLocalsProxy  (GH-122309)

Co-authored-by: Alyssa Coghlan <[email protected]>
(cherry picked from commit 5912487)
blhsing pushed a commit to blhsing/cpython that referenced this pull request Aug 22, 2024
@encukou @ncoghlan @blhsing
pythongh-120906: Support arbitrary hashable keys in FrameLocalsProxy (…
a22fbd1
…pythonGH-122309)


Co-authored-by: Alyssa Coghlan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ncoghlan ncoghlan ncoghlan approved these changes
@picnixz picnixz picnixz approved these changes
@gaogaotiantian gaogaotiantian gaogaotiantian approved these changes
@markshannon markshannon Awaiting requested review from markshannon markshannon is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@encukou encukou

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@encukou @gaogaotiantian @markshannon @ncoghlan @picnixz
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.