GeezerVison is showing github.com — Annotation scopes containing nested scopes · Issue #109118 · python/cpython
Annotation scopes containing nested scopes #109118DescriptionBug report
Bug description:
The following code causes a SystemError during compilation.
class name_2[*name_5, name_3: int]:
(name_3 := name_4)
class name_4[name_5: name_5]((name_4 for name_5 in name_0 if name_3), name_2 if name_3 else name_0):
passoutput (Python 3.12.0rc2+):
SystemError: compiler_lookup_arg(name='name_3') with reftype=3 failed in <generic parameters of name_4>; freevars of code <genexpr>: ('name_3',)removing the named-expression makes it worse and causes a crash
class name_2[*name_5, name_3: int]:
class name_4[name_5: name_5]((name_4 for name_5 in name_0 if name_3), name_2 if name_3 else name_0):
passoutput (Python 3.12.0rc2+):
Traceback (most recent call last):
File "/home/frank/projects/pysource-codegen/bug2.py", line 2, in <module>
class name_2[*name_5, name_3: int]:
File "/home/frank/projects/pysource-codegen/bug2.py", line 2, in <generic parameters of name_2>
class name_2[*name_5, name_3: int]:
File "/home/frank/projects/pysource-codegen/bug2.py", line 4, in name_2
class name_4[name_5: name_5]((name_4 for name_5 in name_0 if name_3), name_2 if name_3 else name_0):
File "/home/frank/projects/pysource-codegen/bug2.py", line 4, in <generic parameters of name_4>
class name_4[name_5: name_5]((name_4 for name_5 in name_0 if name_3), name_2 if name_3 else name_0):
^^^^^^
NameError: name 'name_0' is not defined
Modules/gcmodule.c:113: gc_decref: Assertion "gc_get_refs(g) > 0" failed: refcount is too small
Enable tracemalloc to get the memory block allocation traceback
object address : 0x7f4c77059a30
object refcount : 1
object type : 0x560ea4b6cfe0
object type name: dict
object repr : {'__module__': '__main__', '__qualname__': 'name_2', '__type_params__': (name_5, name_3)}
Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: finalizing (tstate=0x0000560ea4cc9ba0)
Current thread 0x00007f4c7756c280 (most recent call first):
Garbage-collecting
<no Python frame>side note:
The reason for the strange looking language constructs and variable names is that I generated this code. I found it while working on my pysource-codegen library.
CPython versions tested on:
3.12
Operating systems tested on:
Linux
Linked PRs
ActivityJelleZijlstra commented on Sep 8, 2023 Thanks, this is very helpful!
Minimized version:
class name_2[name_3]:
name_3 = name_4
class name_4[name_5]((name_3 for name_5 in name_0), name_3):
passThe runtime crash when removing the name_3 = name_4 line is a bug in the interpreter that I will send a PR for. The compiler crash (SystemError) is a different bug that I haven't yet tracked down. JelleZijlstra commented on Sep 8, 2023 The first thing to figure out with the compiler crash is how the code should behave, as class scopes are weird. We have a genexp within the type parameters of a generic class nested inside another generic class.
Here's the test I wrote, demonstrating how this works if the inner class is not generic:
+ def test_gen_exp_in_nested_class(self):
+ class C[T]:
+ T = "class"
+ class Inner(make_base(T for _ in (1,)), make_base(T)):
+ pass
+ T, = C.__type_params__
+ base1, base2 = C.Inner.__bases__
+ self.assertEqual(list(base1.__arg__), [T])
+ self.assertEqual(base2.__arg__, "class")
+
+ def test_gen_exp_in_nested_generic_class(self):
+ class C[T]:
+ T = "class"
+ class Inner[U](make_base(T for _ in (1,)), make_base(T)):
+ pass
+ T, = C.__type_params__
+ base1, base2, _ = C.Inner.__bases__
+ self.assertEqual(list(base1.__arg__), [T])
+ self.assertEqual(base2.__arg__, "class")
+
+
+def make_base(arg):
+ class Base:
+ __arg__ = arg
+ return Base
+
In short, the genexp should not see the definition in the outer class body, only the one in the generic parameters of the outer class. However, the other bases should see the definition in the class body. This is how it works now if the inner class is not generic, and this is correct.
If the inner class is generic, the behavior should be analogous, but evidently there's something wrong in the bookkeeping. added 3 commits that reference this issue on Sep 9, 2023gh-109118: Fix runtime crash when NameError happens in PEP 695 functi… changed the title [-]SystemError/crash while compiling python code[/-] [+]SystemError with generator expression within PEP 695 scope within class[/+] on Sep 9, 2023JelleZijlstra commented on Sep 9, 2023 Thinking about it more, I recommend that we should disallow nested scopes (comprehensions, generator expressions, and lambdas) within PEP 695 scopes (bases of generic classes, annotations of generic functions, values of type aliases) within classes. The normal scoping rules would produce very weird outcomes and would I think be difficult to implement. Also, disallowing them is a safe option this late in the Python 3.12 release cycle: if we can implement it correctly, we can always allow these constructs later for 3.13.
We would disallow the following constructs:
class C:
type T = lambda: T # SyntaxError
type T = [T for T in T] # SyntaxError
type T[U]: (T for T in U)] = int # SyntaxError
def f[T](x: lambda: T): ... # SyntaxError
class I[T](T for T in T): ... # SyntaxErrorAll of these constructs would still be allowed by the interpreter outside of class scopes.
Why do I say the outcomes are weird? Consider this code:
def make_base(arg):
class Base:
__arg__ = arg
return Base
class C[T]:
T = "class"
U = "class"
class Inner[U](make_base((T, U)), make_base((T, U) for _ in (1,))):
passThere are four occurrences of T and U in the bases of Inner. What should they mean (referring to https://peps.python.org/pep-0695/#scoping-behavior)?
- The first T is inside the annotation scope for Inner, but annotation scopes (unlike normal function scopes) see names defined in the enclosing class scope, so it should refer to the class-scoped variable
"class".
- The first U is also inside the annotation scope, so it should refer to the type parameter U for the inner class.
- The second T is inside a generator expression that is in turn inside an annotation scope. It should not see the class scope, because genexps inside classes cannot normally see variables in that class scope (as discussed in List comprehensions now have access to the enclosing class scope #104374). Therefore, here
T should refer to the type parameter for the outer generic class C.
- The second
U is also inside a genexp inside an annotation scope. I think the intuitive answer is that it should refer to the inner type parameter U. But that creates a weird result where the genexp can see the enclosing annotation scope, but not all names that are visible to that annotation scope. This is, as far as I can tell, unlike any other scope in Python, and it would require some changes to the abstractions in symtable.c to implement correctly.
Why is it OK to disallow comprehensions/lambdas/genexps in PEP 695 scopes within classes?
- This is new syntax, so we won't be breaking anyone's code by disallowing it. We can always allow this later.
- There are few conceivable use cases for nested scopes within annotation scopes. Annotation scopes can include the following:
- The right-hand side of type aliases. This should be a type, which cannot include a lambda or comprehension if you follow the standard type system.
- The bounds and constraints of TypeVars. These are also supposed to be valid as types.
- The parameter and return annotations of generic functions. Same.
- The bases and keyword arguments of generic classes. While these could contain any code and therefore also comprehensions or lambdas, that's going to be very rare.
I will submit a PR implementing this restriction. Reacted by Guido van Rossum JelleZijlstra commented on Apr 18, 2024 Reopening this to consider changing some of the behavior for Python 3.13, where actual annotations will also be implemented using annotation scopes (PEP-649; cc @larryhastings).
I now think the right behavior is that if a scope is nested in an annotation scope that is itself in a class scope, it can see names defined in the annotation scope, but not names defined in the enclosing class scope. After all, that's how generic methods currently work:
T = "global T"
U = "global U"
class C:
T = "class T"
U = "class U"
def meth[T](self):
print(T, U)
C().meth()This currently picks up the type variable for T and the global for U.
This is also the natural behavior of the current implementation, at least for lambdas. I'll send a PR first to change the behavior for lambdas, then work on comprehensions. changed the title [-]SystemError with generator expression within PEP 695 scope within class[/-] [+]Annotation scopes containing nested scopes[/+] on Apr 18, 2024added 3 commits that reference this issue on Apr 18, 2024gh-109118: Make comprehensions work within annotation scopes, but wit…
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|