GeezerVison is showing github.com — 3.12.0b4 Backwards incompatible change with reassignment of `cls.__new__` and `super()` · Issue #106917 · python/cpython
3.12.0b4 Backwards incompatible change with reassignment of cls.__new__ and super() #106917Labels 3.12only security fixesonly security fixes3.13only security fixesonly security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error Description
Bug report
Noticed this while testing 3.12.0b4. While in my particular case I can work around it, it never the less is a change in behavior to 3.11.
class FixedIntType(int):
_signed: bool
def __new__(cls, *args, **kwargs):
print(f"{cls}, {args=}")
return super().__new__(cls, *args, **kwargs)
def __init_subclass__(cls, signed: int | None = None) -> None:
super().__init_subclass__()
if signed is not None:
cls._signed = signed
if "__new__" not in cls.__dict__:
cls.__new__ = cls.__new__ # <-- This line triggers the error
class uint_t(FixedIntType, signed=False):
pass
class CustomInt(uint_t):
def __new__(cls, value: int = 0):
return super().__new__(cls, value)
def with_id(self, value: int):
return type(self)(value)
CustomInt().with_id(1024)<class '__main__.CustomInt'>, args=(0,)
<class '__main__.CustomInt'>, args=(<class '__main__.CustomInt'>, 1024)
Traceback (most recent call last):
File "/.../cpython/test.py", line 26, in <module>
CustomInt().with_id(1024)
File "/.../cpython/test.py", line 24, in with_id
return type(self)(value)
^^^^^^^^^^^^^^^^^
File "/.../cpython/test.py", line 21, in __new__
return super().__new__(cls, value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/.../cpython/test.py", line 6, in __new__
return super().__new__(cls, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: int() base must be >= 2 and <= 36, or 0
Without cls.__new__ = cls.__new__
<class '__main__.CustomInt'>, args=(0,)
<class '__main__.CustomInt'>, args=(1024,)
I bisected the issue to #103497.
/CC: @carljm
Your environment
- CPython versions tested on:
3.120b4
- Operating system and architecture: macOS ARM64
Linked PRs
Activitycarljm commented on Jul 21, 2023 Thanks for this report!
__new__ is automagically treated as a staticmethod (but only when it's part of the type at construction, not if it's assigned afterward.) So the repro case ultimately simplifies and de-sugars to this:
class A:
@staticmethod
def some(cls, *args, **kwargs):
print(f"{cls}, {args=}")
assert not args
assert not kwargs
class B(A):
def some(cls, *args, **kwargs):
return super().some(cls, *args, **kwargs)
class C(B):
@staticmethod
def some(cls):
return super().some(cls)
C.some(C)
C.some(C)
Note that B.some is not a @staticmethod, while A.some and C.some both are. This is equivalent to what happens in the OP. The assignment cls.__new__ = cls.__new__ looks up the staticmethod in the right hand side (from the base class via MRO lookup, since it doesn't exist yet on cls) and "unwraps" the staticmethod in descriptor resolution, returning a plain function object, and then assigns that plain function object (not a staticmethod) as the __new__ method on the new subclass. (And dynamic assignment of __new__ does not magically wrap it in staticmethod.) So the middle class in the hierarchy (uint_t in the OP) ends up with a non-staticmethod __new__, while the top and bottom classes have a normal staticmethod __new__.
This simplified and de-sugared repro works in 3.11, but fails in a similar way to the OP in main and 3.12:
➜ ../dbg/python superbug3.py
<class '__main__.C'>, args=()
<class '__main__.C'>, args=(<class '__main__.C'>,)
Traceback (most recent call last):
File "/home/carljm/cpython-builds/repros/superbug3.py", line 18, in <module>
C.some(C)
File "/home/carljm/cpython-builds/repros/superbug3.py", line 15, in some
return super().some(cls)
^^^^^^^^^^^^^^^^^
File "/home/carljm/cpython-builds/repros/superbug3.py", line 10, in some
return super().some(cls, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/carljm/cpython-builds/repros/superbug3.py", line 5, in some
assert not args
AssertionError
The double call to C.some(C) in the repro is necessary so that we specialize LOAD_SUPER_ATTR to LOAD_SUPER_ATTR_METHOD, triggering the bug. The base LOAD_SUPER_ATTR goes through the same code path as an unoptimized LOAD_ATTR; CALL.
What we have in these examples is a "class-mode" super() call (e.g. super(MyClass, MyClass).foo(...), where the second super arg is also a class, not an instance). (Zero-arg super() implicitly uses the first argument of the method as the second arg to super(), and in this case that first arg of the method is a class, not an instance.)
In a class-mode super() call (just like for a normal lookup directly on a class), we do not pass an instance to the descriptor-getter, only a type (e.g. func_descr_get(descr, NULL, type) instead of func_descr_get(descr, obj, type). In the case of a regular function object, this causes the function descriptor to just return the function itself (no bound method), which is the same behavior the staticmethod descriptor always has! So as long as we are only doing class lookups, staticmethod has no effect, and therefore we can get away with this mixed staticmethod/non-staticmethod hierarchy just fine.
The bug here is that the LOAD_SUPER_ATTR_METHOD specialization tries to use the LOAD_METHOD optimization (avoid actually creating a bound method object via descriptor, and instead just detect that case and ensure the extra argument is provided), but fails to account for the fact that a class lookup of a regular function object (not a classmethod) is just a function call, not a method call.
Fix coming shortly.
MetadataMetadata3.12only security fixesonly security fixes3.13only security fixesonly security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DevelopmentNo branches or pull requestsIssue actions
|