Description

JelleZijlstra

opened on May 24, 2023

Issue body actions

% cat gsl.py
import inspect

def test_list_add(self):
    def capybara() -> None:
        assert_is_value(
            [x] + [y],
            z,
        )
        assert_is_value(
            # in the binop implementation
            a,
        )

print(inspect.getsourcelines(test_list_add))
% python3.11 gsl.py 
(['def test_list_add(self):\n', '    def capybara() -> None:\n', '        assert_is_value(\n', '            [x] + [y],\n', '            z,\n', '        )\n', '        assert_is_value(\n', '            # in the binop implementation\n', '            a,\n', '        )\n'], 3)
% ~/py/cpython/python.exe gsl.py 
(['def test_list_add(self):\n', '    def capybara() -> None:\n', '        assert_is_value(\n', '            [x] + [y],\n', '            z,\n', '        )\n', '        assert_is_value(\n', '            # in the binop implementation\n'], 3)
% ~/py/cpython/python.exe -V
Python 3.12.0b1+

On 3.12, the last two lines (a, )) are not returned as part of the source lines. When I tried to minify by removing the first assert_is_value() call, it instead returned the print() line as part of the function.

This looks related to the tokenizer, cc @pablogsal @lysnikolaou.

Linked PRs

• gh-104866: Tokenize should emit NEWLINE after exiting block with comment #104870
• [3.12] gh-104866: Tokenize should emit NEWLINE after exiting block with comment (GH-104870) #104872

Activity

1. JelleZijlstra

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   3.12only security fixesonly security fixes

   3.13only security fixesonly security fixes

   on May 24, 2023
2. AlexWaygood

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   and removed

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   on May 24, 2023
3. lysnikolaou commented on May 24, 2023

   

   lysnikolaou

   on May 24, 2023

   Member

   More actions

   This is a bug in the implementation of the new tokenize module. The tokenizer checks whether something is in a parenthesized block in order to emit a NL token instead of a NEWLINE token. It also sets and unsets a flag about a comment-only line, in order to do the same. When both are true (a comment inside a parenthesized block), it nevers resets the comment flag. Here's a minimal example:

   cpython on  main [$?] via C v14.0.3-clang via 🐍 pyenv 3.11.3
   ❯ cat hello.py              
   [
       # hello
       1
   ]
   
   cpython on  main [$?] via C v14.0.3-clang via 🐍 pyenv 3.11.3
   ❯ ./python.exe -m tokenize hello.py
   0,0-0,0:            ENCODING       'utf-8'        
   1,0-1,1:            OP             '['            
   1,1-1,2:            NL             '\n'           
   2,4-2,11:           COMMENT        '# hello'      
   2,11-2,12:          NL             '\n'           
   3,4-3,5:            NUMBER         '1'            
   3,5-3,6:            NL             '\n'           
   4,0-4,1:            OP             ']'            
   4,1-4,2:            NL             '\n'       # This should be a NEWLINE instead    
   5,0-5,0:            ENDMARKER      ''             

   I'm opening a PR shortly.
4. lysnikolaou

   added a commit that references this issue on May 24, 2023

   pythongh-104866: Tokenize should emit NEWLINE after exiting block wit…

   be2eff3
5. pablogsal

   added a commit that references this issue on May 24, 2023

   gh-104866: Tokenize should emit NEWLINE after exiting block with comm…

   c90a862
6. miss-islington

   added a commit that references this issue on May 24, 2023

   pythongh-104866: Tokenize should emit NEWLINE after exiting block wit…

   3c3ea89
7. pablogsal

   closed this as completedon May 24, 2023
8. terryjreedy

   added a commit that references this issue on May 24, 2023

   [3.12] gh-104866: Tokenize should emit NEWLINE after exiting block wi…

   ddc29c8

Issue actions