GeezerVison is showing github.com — `FrameLocalsProxy` is stricter than `dict` about what constitutes a match · Issue #120906 · python/cpython
FrameLocalsProxy is stricter than dict about what constitutes a match #120906
DescriptionBug report
Bug description:
import inspect
class MyString(str):
pass
def f():
a = 1
locals = inspect.currentframe().f_locals
print(MyString("a") in locals)
f()In Python 3.12 and below this prints True. In Python 3.13 this print False. I think it comes down to the check for exact unicode:
|
if (PyUnicode_CheckExact(key)) { |
The change in behaviour isn't a huge problem so if it's intended then I won't spend waste any time complaining about it, but I do think it's worth confirming that it is intended/desired.
CPython versions tested on:
3.13
Operating systems tested on:
Linux
Linked PRs
Activitygaogaotiantian commented on Jun 23, 2024 Interesting issue. I don't have a definitive answer here but this is something we need to deal with, because we are getting some weird issues here:
import sys
class MyString(str):
pass
def f():
x = 1
local = sys._getframe().f_locals
local[MyString('x')] = 2
print(local.keys())
# ['x', 'local', 'x']
print(local)
# {'x': 2, 'local': {...}}
f()Internally we check if the input key is an exact unicode, but we do utilize dict for certain features (repr for example). This causes an inconsistent behavior.
My preferred solution is to enforce any key to be an exact unicode string. The reason for that is, unlike a generic dict, FrameLocalsProxy should be a direct proxy for local variables and keys other than unicode string does not quite make sense. However, I know I'm the more aggressive type when it comes to changes so I'll list some concerns as well.
frame.f_locals for an unoptimized frame (module/class) is a real dict and can take any type of keys. There might also be a usage to use a subclass as keys? If the original usage is shown to be helpful, that might tip the scale for me.
I'm a bit worried about the can of worms when we allow subclasses of unicode strings - what if it overwrite the __hash__ and __eq__ methods? Will it cause more troubles than the benefits? da-woods commented on Jun 23, 2024 ContributorAuthor More actions Just to explain the context I ran into it in:
The Cython compiler wraps most of it's strings into an EncodedString class (for reasons that are possibly historic and which I don't quite understand). To clarify - that's a class used internally by the compiler, rather than a class used in Cython extension modules.
We have an inline method of compilation, and here missing variables are taken from the surrounding frame. The frame comes from Python so the variable names are just a normal str. However the names we're using to look them up are our EncodedString class. These used to work fine when used to look up keys (since they just matched the equivalent str) but now don't.
It's easy enough to work around so I'm relaxed about the solution whatever you decide to do. I think what we were doing was mostly accidental and there wasn't a hidden special use-case behind it.
But for this use-case I'm just reading existing values from the FrameLocalsProxy and not worried about storing subclassed strings. Obviously you need to consider both sides of it though. gaogaotiantian commented on Jun 23, 2024 We need to discuss this further with @markshannon, but I think the desired behavior would be either
- Report an error when the key is not an exact unicode string
- or allow all string-like (subclasses of unicode) keys and access the actual fast locals even with those strings.
gaogaotiantian commented on Jun 23, 2024 I just remembered that we already had that discussion when I was implementing PEP 667. The PEP suggests that we should allow arbitrary types.
Then the question would be what if the user gives a unicode-like key, do we try to access the fast variables with that key? ncoghlan commented on Jul 16, 2024 To avoid breaking mapping invariants, if a given key compares equal to the name of a local variable, it needs to be handled as a lookup of that variable name. Having a PyUnicode_CheckExact guard on a string comparison fast path still makes sense, but the fallback needs to be a linear search using the abstract object comparison API rather than assuming the key is absent. gaogaotiantian commented on Jul 16, 2024 But that's not the mapping protocol right? What if a key has a different hash, but the equal value? What if the key is
class Evil:
def __eq__(self, other):
return TrueIt's unhashable but only comparing equality will match it to an arbitrary local (the first one).
Are we going to implement the full mapping interface? Checking everything as dict/mapping does? The dark corner we left here might eat us in the future. encukou commented on Jul 17, 2024
What if a key has a different hash, but the equal value
That's a violation of the hash protocol: objects that compare equal must have the same hash (or at least one must be unhashable).
The Evil example is incomplete, since __hash__ is not inherited if __eq__ is overridden.
But if Evil had __hash__ explicitly set to str.__hash__, it would be OK to rely on it and skip the call to __eq__.
IMO, it would also be OK to convert the argument to PyUnicode, and do the lookup with the resulting temporary object. Reacted by Alyssa Coghlan gaogaotiantian commented on Jul 25, 2024 My worry about supporting a unicode subclass or an arbitrary object as a key is - it does not fit the mapping protocol. If an object overrides it's __eq__ but not __hash__, it can't be used as a key. If an object has a different hash to another one but also equal to that object, they will be different keys in the dict.
None of the proposals above have this feature. So basically, we are inventing a very new mapping protocol specifically for FrameLocalsproxy and it's not easy to explain - we don't have a simple enough rule.
That's why I liked "unicode only" solution - that's a simple rule that works and easy to understand. It will cause some trouble, but we are already causing troubles.
We need to consider - what if a key is a unicode subclass, but has different __eq__ and/or __hash__ method? It would be very difficult for us to behave exactly as a dict. What if it's not even a subclass of unicode?
If we can't do that, I kind of like what @encukou suggested - we convert the non-exact-unicode keys to unicode first, for both read and write. That's a golden rule that everyone can follow, and that will solve many benign cases like this very issue. encukou commented on Jul 26, 2024 I finally looked at the code, rather than going just by the conversation here. But I couldn't just look...
Please see PR #122309, as a suggestion for how to handle this.
If an object overrides its __eq__ but not __hash__, it can't be used as a key. It's not hashable.
If an object has a different hash to another one but also equal to that object, that's a violation of the hash protocol, and dict can behave surprisingly (give weird results/exceptions, but not crash). We don't need to match that behavior.
The hash protocol:
- limits the types (to hashable ones) and
- allows dicts to do an optimization: they can look at a small subset of entries. We don't need to use that optimization.
Keys that aren't unicode subclasses shouldn't be a problem. gaogaotiantian commented on Jul 26, 2024 Oh sorry I did not realize you posted in the issue. I don't think I'm getting notification from this issue.
Could you point to me the documentation about the hash protocol? I could not find it. encukou commented on Jul 27, 2024 See hashable in the docs:
Hashable objects which compare equal must have the same hash value.
ncoghlan commented on Jul 27, 2024 The data model docs in https://docs.python.org/3/reference/datamodel.html#object.__hash__ also state "The only required property is that objects which compare equal have the same hash value". However, they go into more detail about how to define __hash__ and __eq__ to be consistent with each other, as well as how to indicate when subclasses of hashable parent classes are not themselves hashable.
Classes that don't abide by those rules just straight up don't work properly, so the fact they won't misbehave as badly with locals proxy instances as they do with regular dicts isn't a problem we need to be concerned about. Yhg1s commented on Jul 31, 2024 This issue is all fixed now, right? ncoghlan commented on Aug 1, 2024
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|