GeezerVison is showing github.com — line number for a return in a with block after 3.10 is that of the with, not the return statement · Issue #93975 · python/cpython
line number for a return in a with block after 3.10 is that of the with, not the return statement #93975Labels 3.10 (EOL)end of lifeend of life3.11only security fixesonly security fixes3.12only security fixesonly security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error Descriptiondef foo() -> int:
with open('/dev/null') as devnull:
return 'a'In Python 3.10 onwards, the reported line number for the return statement is incorrect.
Python main 3.12-main-ish not far from 3.11:
>>> def foo() -> int:
... with open('a') as f:
... return 'b'
...
>>> dis.dis(foo)
1 0 RESUME 0
2 2 LOAD_GLOBAL 1 (NULL + open)
14 LOAD_CONST 1 ('a')
16 CALL 1
26 BEFORE_WITH
28 STORE_FAST 0 (f)
3 30 NOP
2 32 LOAD_CONST 0 (None)
34 LOAD_CONST 0 (None)
36 LOAD_CONST 0 (None)
38 CALL 2
48 POP_TOP
50 LOAD_CONST 2 ('b')
52 RETURN_VALUE
>> 54 PUSH_EXC_INFO
56 WITH_EXCEPT_START
58 POP_JUMP_FORWARD_IF_TRUE 4 (to 68)
60 RERAISE 2
>> 62 COPY 3
64 POP_EXCEPT
66 RERAISE 1
>> 68 POP_TOP
70 POP_EXCEPT
72 POP_TOP
74 POP_TOP
76 LOAD_CONST 0 (None)
78 RETURN_VALUE
ExceptionTable:
28 to 30 -> 54 [1] lasti
54 to 60 -> 62 [3] lasti
68 to 68 -> 62 [3] lasti
>>> sys.version_info
sys.version_info(major=3, minor=12, micro=0, releaselevel='alpha', serial=0)
The only thing attributed to line 3 is a NOP. The RETURN_VALUE of 'b' is listed as line 2. That is wrong.
Python 3.9:
>>> def foo() -> int:
... with open('a') as f:
... return 'b'
...
>>> dis.dis(foo)
2 0 LOAD_GLOBAL 0 (open)
2 LOAD_CONST 1 ('a')
4 CALL_FUNCTION 1
6 SETUP_WITH 18 (to 26)
8 STORE_FAST 0 (f)
3 10 POP_BLOCK
12 LOAD_CONST 0 (None)
14 DUP_TOP
16 DUP_TOP
18 CALL_FUNCTION 3
20 POP_TOP
22 LOAD_CONST 2 ('b')
24 RETURN_VALUE
>> 26 WITH_EXCEPT_START
28 POP_JUMP_IF_TRUE 32
30 RERAISE
>> 32 POP_TOP
34 POP_TOP
36 POP_TOP
38 POP_EXCEPT
40 POP_TOP
42 LOAD_CONST 0 (None)
44 RETURN_VALUE
>>> sys.version_info
sys.version_info(major=3, minor=9, micro=12, releaselevel='final', serial=0)
Noticed by @martindemello working on a 3.10 error attribution bug in pytype. Activitydignissimus commented on Jun 18, 2022 The issue was introduced in 937cebc
See bpo-44298: #88464
The same behaviour occurs with try-finally blocks on both 3.9 and 3.10+
def function():
try:
return
finally:
passWith the above code, disassembling on Python 3.10.5 produces the following
1 0 LOAD_CONST 0 (<code object function at 0x7f96b4e0d000, file "a.py", line 1>)
2 LOAD_CONST 1 ('function')
4 MAKE_FUNCTION 0
6 STORE_NAME 0 (function)
8 LOAD_CONST 2 (None)
10 RETURN_VALUE
Disassembly of <code object function at 0x7f96b4e0d000, file "tryfinally.py", line 1>:
2 0 SETUP_FINALLY 3 (to 8)
3 2 POP_BLOCK
5 4 LOAD_CONST 0 (None)
6 RETURN_VALUE
>> 8 RERAISE 0And on 3.9.0
1 0 LOAD_CONST 0 (<code object function at 0x7f72158a8be0, file "/tmp/a.py", line 1>)
2 LOAD_CONST 1 ('function')
4 MAKE_FUNCTION 0
6 STORE_NAME 0 (function)
8 LOAD_CONST 2 (None)
10 RETURN_VALUE
Disassembly of <code object function at 0x7f72158a8be0, file "tryfinally.py", line 1>:
2 0 SETUP_FINALLY 6 (to 8)
3 2 POP_BLOCK
4 LOAD_CONST 0 (None)
6 RETURN_VALUE
5 >> 8 RERAISE
10 LOAD_CONST 0 (None)
12 RETURN_VALUEmarkshannon commented on Jun 20, 2022 I believe the current behavior is correct.
def foo() -> int:
with open('/dev/null') as devnull:
return 'a'The sequence of events is:
- Function start at line 1
- Line 2
- Evaluate
open and enter the with block
- Line 3
- Evaluate 'a' and the return statement
- Line 2
- Exit the with block
The jump back to line 2 may seem surprising, but it is necessary.
Because the call to __exit__ when exiting the with statement is is observable and may raise an exception, it needs to have a line number. markshannon commented on Jun 20, 2022 If you change "a" to len("a"), note that len("a") is on line 3.
def foo() -> int:
with open('/dev/null') as devnull:
return len("a") 1 0 RESUME 0
2 2 LOAD_GLOBAL 1 (NULL + open)
14 LOAD_CONST 1 ('/dev/null')
16 CALL 1
26 BEFORE_WITH
28 STORE_FAST 0 (devnull)
3 30 LOAD_GLOBAL 3 (NULL + len)
42 LOAD_CONST 2 ('a')
44 CALL 1
2 54 SWAP 2
56 LOAD_CONST 0 (None)
58 LOAD_CONST 0 (None)
60 LOAD_CONST 0 (None)
62 CALL 2
72 POP_TOP
74 RETURN_VALUE
>> 76 PUSH_EXC_INFO
78 WITH_EXCEPT_START
80 POP_JUMP_FORWARD_IF_TRUE 1 (to 84)
82 RERAISE 2
>> 84 POP_TOP
86 POP_EXCEPT
88 POP_TOP
90 POP_TOP
92 LOAD_CONST 0 (None)
94 RETURN_VALUE
>> 96 COPY 3
98 POP_EXCEPT
100 RERAISE 1
ExceptionTable:
28 to 52 -> 76 [1] lasti
76 to 84 -> 96 [3] lasti
markshannon commented on Jun 20, 2022 On more thing.
In Python 3.9, if __exit__ raises, it reports the wrong line number.
def foo() -> int:
with raises_on_exit() as mgr:
return 'a'
passWill report the exception on line 4, which is clearly wrong.
It is 3.9 that is broken, not 3.10 or 3.11, IMO. jaraco commented on Jun 20, 2022 Please disregard the commits above referencing this issue. They should have referenced #93957. Reacted by Gregory P. Smith gpshead commented on Jun 20, 2022 Thanks Mark! From a bytecode perspective I can understand how this might not be a bug, especially as our bytecode becomes more optimal.
If this is a non-issue feel free to close it.
I believe the question from a pytype perspective, which does its program analysis via bytecode, is if it is possible to determine where the return statement was in the code for error reporting reasons. '' not being an int, the return statement attempting the wrong type is where an error message would ideally point. In absense of concrete information from the bytecode the analysis could possibly detect that a with or try block is being exited and perhaps tell the user the more vague block the errant return was exiting from. I'll leave that up to pytype folks to figure out. markshannon commented on Jun 20, 2022 If the expression in the return statement is anything but a constant, it should have the exact location information, not just the line number. If it is a constant, then simple AST analysis will do.
My advice to the pytype developers would be to build their own CFG. The transformations required for precise type analysis and for code generation are different enough that a custom CFG is probably the way to go.
Here's one I made earier 🙂
https://codeql.github.com/docs/codeql-language-guides/analyzing-control-flow-in-python/ markshannon commented on Jun 20, 2022
MetadataMetadata3.10 (EOL)end of lifeend of life3.11only security fixesonly security fixes3.12only security fixesonly security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DevelopmentNo branches or pull requestsIssue actions
|