GeezerVison is showing github.com — 3.12.0b1 includes backwards incompatible change to operation of `super()` · Issue #105035 · python/cpython
3.12.0b1 includes backwards incompatible change to operation of super() #105035Labels 3.12only security fixesonly security fixes3.13only security fixesonly security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DescriptionBug report
The optimisation to LOAD_SUPER_ATTR introduced by #104270 appears to have introduced a backwards incompatibility.
The following code will reproduce the problem:
class MyInstance:
def __new__(cls, ptr, name, bases, attrs):
self = super().__new__(cls, name, bases, attrs)
print(f"{MyInstance=} {self=} {type(self)=}, {super(MyInstance, type(self))=}")
super(MyInstance, type(self)).__setattr__(self, "ptr", ptr)
return self
def __setattr__(self, name, value):
raise Exception()
class MyClass(MyInstance, type):
def __new__(cls, name):
self = super().__new__(cls, id(name), name, (MyInstance,), {})
return self
class1 = MyClass("Class1")
print(f"{class1.ptr=}")
class2 = MyClass("Class2")
print(f"{class2.ptr=}")
Under 3.12.0a7 (and previous stable Python versions going back to at least 3.7), this will succeed, outputting:
MyInstance=<class '__main__.MyInstance'> self=<class '__main__.Class1'> type(self)=<class '__main__.MyClass'>, super()=<super: <class 'MyInstance'>, <MyClass object>>
class1.ptr=4364457392
MyInstance=<class '__main__.MyInstance'> self=<class '__main__.Class2'> type(self)=<class '__main__.MyClass'>, super()=<super: <class 'MyInstance'>, <MyClass object>>
class2.ptr=4365761904
Under 3.12.0b1, it raises an error:
MyInstance=<class '__main__.MyInstance'> self=<class '__main__.Class1'> type(self)=<class '__main__.MyClass'>, super()=<super: <class 'MyInstance'>, <MyClass object>>
class1.ptr=4370144336
MyInstance=<class '__main__.MyInstance'> self=<class '__main__.Class2'> type(self)=<class '__main__.MyClass'>, super()=<super: <class 'MyInstance'>, <MyClass object>>
Traceback (most recent call last):
File "/Users/rkm/beeware/rubicon/objc/repro.py", line 24, in <module>
class2 = MyClass("Class2")
^^^^^^^^^^^^^^^^^
File "/Users/rkm/beeware/rubicon/objc/repro.py", line 16, in __new__
self = super().__new__(cls, id(name), name, (MyInstance,), {})
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/rkm/beeware/rubicon/objc/repro.py", line 7, in __new__
super().__setattr__(self, "ptr", ptr)
TypeError: expected 2 arguments, got 3
That is - the first MyClass instance can be instantiated; however, the second instance fails in mid-construction when invoking __setattr__. The state of the objects prior to the __setattr__ invocation appear to be identical, but the __setattr__ method behaves differently on the second invocation.
Git bisect has narrowed the cause down to #104270 (CC @carljm, @markshannon).
The reproduction case also fails if the call to super:
super(MyInstance, type(self)).__setattr__(self, "ptr", ptr)
is replaced with the simpler:
super().__setattr__(self, "ptr", ptr)
Background
This test case has been extracted from rubicon-objc, causing beeware/rubicon-objc#313. Rubicon is a wrapper around the Objective C runtime used by macOS and iOS; MyInstance is an analog of ObjCInstance, and MyClass is a an analog of ObjCClass. ObjCInstance has an implementation of __setattr__ to redirect attribute access to the underlying ObjC calls. However, during construction, ObjCInstance needs to store a ptr of the underlying ObjC instance. This isn't a valid ObjC attribute, so super() is used to access the underlying __setattr__ implementation to set the attribute.
Your environment
- CPython versions tested on: 3.7.9, 3.10.11, 3.12.0a7, 3.12.0b1
- Operating system and architecture: macOS ARM64
Linked PRs
Activitysunmy2019 commented on May 28, 2023 Reduced test case
class MyInstance(type):
pass
def test(name):
self = MyInstance(name, (MyInstance,), {})
super(MyInstance, type(self)).__setattr__(self, name, 1)
test("call1")
test("call2") # <-- only at the second callsunmy2019 commented on May 28, 2023 Also, note the following passes. So it's definitely related to the interpreter core.
class MyInstance(type):
pass
def test(name):
self = MyInstance(name, (MyInstance,), {})
m = super(MyInstance, type(self)).__setattr__
m(self, name, id(name))
test("call1")
test("call2")sunmy2019 commented on May 28, 2023 The C call frame when the TypeError is raised
Details
#0 PyErr_Format (exception=0x555555bc11c0 <_PyExc_TypeError>, format=format@entry=0x555555a2b730 "%.200s expected %s%zd argument%s, got %zd") at Python/errors.c:1193
#1 0x000055555589f2ae in _PyArg_CheckPositional (name=name@entry=0x5555559cb7a6 "", nargs=nargs@entry=3, min=min@entry=2, max=max@entry=2) at Python/getargs.c:2813
#2 0x000055555589f344 in unpack_stack (args=0x7ffff7a23848, nargs=3, name=name@entry=0x5555559cb7a6 "", min=min@entry=2, max=max@entry=2, vargs=vargs@entry=0x7fffffffd8b0) at Python/getargs.c:2836
#3 0x000055555589f478 in PyArg_UnpackTuple (args=<optimized out>, name=name@entry=0x5555559cb7a6 "", min=min@entry=2, max=max@entry=2) at Python/getargs.c:2864
#4 0x0000555555796796 in wrap_setattr (self=0x555555db65d0, args=<optimized out>, wrapped=0x55555579cbd6 <type_setattro>) at Objects/typeobject.c:7972
#5 0x0000555555713412 in wrapperdescr_raw_call (kwds=0x0, args=0x7ffff7a23830, self=0x555555db65d0, descr=0x7ffff7b4cb90) at Objects/descrobject.c:537
#6 wrapperdescr_call (descr=descr@entry=0x7ffff7b4cb90, args=0x7ffff7a23830, args@entry=0x7ffff7a78470, kwds=kwds@entry=0x0) at Objects/descrobject.c:574
#7 0x0000555555702422 in _PyObject_MakeTpCall (tstate=tstate@entry=0x555555d21cd0 <_PyRuntime+478416>, callable=callable@entry=0x7ffff7b4cb90, args=args@entry=0x7ffff7ae50f0, nargs=<optimized out>, keywords=keywords@entry=0x0)
at Objects/call.c:240
#8 0x0000555555702856 in _PyObject_VectorcallTstate (tstate=0x555555d21cd0 <_PyRuntime+478416>, callable=0x7ffff7b4cb90, args=0x7ffff7ae50f0, nargsf=<optimized out>, kwnames=0x0) at ./Include/internal/pycore_call.h:90
#9 0x0000555555702882 in PyObject_Vectorcall (callable=callable@entry=0x7ffff7b4cb90, args=args@entry=0x7ffff7ae50f0, nargsf=<optimized out>, kwnames=kwnames@entry=0x0) at Objects/call.c:325
#10 0x0000555555865909 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x555555d21cd0 <_PyRuntime+478416>, frame=0x7ffff7ae5090, throwflag=throwflag@entry=0) at Python/bytecodes.c:2643
#11 0x000055555586d3aa in _PyEval_EvalFrame (throwflag=0, frame=<optimized out>, tstate=0x555555d21cd0 <_PyRuntime+478416>) at ./Include/internal/pycore_ceval.h:87
#12 _PyEval_Vector (tstate=tstate@entry=0x555555d21cd0 <_PyRuntime+478416>, func=func@entry=0x7ffff7a184d0, locals=locals@entry=0x7ffff7a21250, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:1610
#13 0x000055555586d468 in PyEval_EvalCode (co=co@entry=0x7ffff7ba9470, globals=globals@entry=0x7ffff7a21250, locals=locals@entry=0x7ffff7a21250) at Python/ceval.c:567
#14 0x00005555558decdf in run_eval_code_obj (tstate=tstate@entry=0x555555d21cd0 <_PyRuntime+478416>, co=co@entry=0x7ffff7ba9470, globals=globals@entry=0x7ffff7a21250, locals=locals@entry=0x7ffff7a21250) at Python/pythonrun.c:1695
#15 0x00005555558df43f in run_mod (mod=mod@entry=0x555555dbff20, filename=filename@entry=0x7ffff7a22560, globals=globals@entry=0x7ffff7a21250, locals=locals@entry=0x7ffff7a21250, flags=flags@entry=0x7fffffffdea8, arena=arena@entry=0x7ffff7a63ee0)
at Python/pythonrun.c:1716
#16 0x00005555558df54f in pyrun_file (fp=fp@entry=0x555555d3cd20, filename=filename@entry=0x7ffff7a22560, start=start@entry=257, globals=globals@entry=0x7ffff7a21250, locals=locals@entry=0x7ffff7a21250, closeit=closeit@entry=1,
flags=0x7fffffffdea8) at Python/pythonrun.c:1616
#17 0x00005555558e2c7d in _PyRun_SimpleFileObject (fp=fp@entry=0x555555d3cd20, filename=filename@entry=0x7ffff7a22560, closeit=closeit@entry=1, flags=flags@entry=0x7fffffffdea8) at Python/pythonrun.c:433
#18 0x00005555558e2ed8 in _PyRun_AnyFileObject (fp=fp@entry=0x555555d3cd20, filename=filename@entry=0x7ffff7a22560, closeit=closeit@entry=1, flags=flags@entry=0x7fffffffdea8) at Python/pythonrun.c:78
#19 0x000055555590c13a in pymain_run_file_obj (program_name=program_name@entry=0x7ffff7a63d60, filename=filename@entry=0x7ffff7a22560, skip_source_first_line=0) at Modules/main.c:360
#20 0x000055555590c4d9 in pymain_run_file (config=config@entry=0x555555d04008 <_PyRuntime+356360>) at Modules/main.c:379
#21 0x000055555590d656 in pymain_run_python (exitcode=exitcode@entry=0x7fffffffe024) at Modules/main.c:610
#22 0x000055555590d6c6 in Py_RunMain () at Modules/main.c:689
#23 0x000055555590d73d in pymain_main (args=args@entry=0x7fffffffe080) at Modules/main.c:719
#24 0x000055555590d800 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:743
#25 0x00005555556537e6 in main (argc=<optimized out>, argv=<optimized out>) at ./Programs/python.c:15
From the C side, it seems that
the first is called with ("call1", id(...))
the second is called with (self, "call2", id(...)) sunmy2019 commented on May 28, 2023 The comment of _PyEval_EvalFrameDefault says
// On entry, the stack is either
// [NULL, callable, arg1, arg2, ...]
// or
// [method, self, arg1, arg2, ...]
And confirmed that in the latter case, the stack is
// [method, self, arg1, arg2, ...]
But it should be NULL?
We can dis this function, and we get
5 0 RESUME 0
6 2 LOAD_GLOBAL 1 (NULL + MyInstance)
12 LOAD_FAST 0 (name)
14 LOAD_GLOBAL 0 (MyInstance)
24 BUILD_TUPLE 1
26 BUILD_MAP 0
28 CALL 3
36 STORE_FAST 1 (self)
12 38 LOAD_GLOBAL 2 (super)
48 LOAD_GLOBAL 0 (MyInstance)
58 LOAD_GLOBAL 5 (NULL + type)
68 LOAD_FAST 1 (self)
70 CALL 1
78 LOAD_SUPER_ATTR 15 (NULL|self + __setattr__)
82 LOAD_FAST 1 (self)
84 LOAD_FAST 0 (name)
86 LOAD_CONST 1 (1)
88 CALL 3
96 POP_TOP
98 RETURN_CONST 0 (None)
78 LOAD_SUPER_ATTR 15 (NULL|self + __setattr__)
must have something to do with this. carljm commented on May 30, 2023 Thanks @freakboy3742, and @sunmy2019 for the simplified repro! I've identified the issue and working on a PR to fix.
Just for the record (and only because it confused me a bit until I double-checked), my testing shows that this wasn't broken by #104270, it was broken by the original introduction of LOAD_SUPER_ATTR in #103497. carljm commented on May 30, 2023 This should be fixed now, in both 3.12 and main. Thanks for the report! freakboy3742 commented on May 30, 2023 ContributorAuthor More actions Confirming this is now working for me. Thanks for the fast turnaround, @carljm! Reacted by Raymond Hettinger Reacted by Carl Meyer, sunmy2019 and Raymond Hettinger
MetadataMetadata3.12only security fixesonly security fixes3.13only security fixesonly security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DevelopmentNo branches or pull requestsIssue actions
|