Description

nedbat

opened on May 26, 2023

Issue body actions

The line attribute in tokens returned by tokenize.generate_tokens incorrectly indicate multiple lines. The tokens should have an invariant that using the .start and .end attributes to index into the .line attribute will produce the .string attribute.

tokbug.py:

import io
import sys
import tokenize

SOURCE = r"""
a + \
b
"""

print(sys.version)
readline = io.StringIO(SOURCE).readline
for tok in tokenize.generate_tokens(readline):
    correct = (tok.string) == (tok.line[tok.start[1]: tok.end[1]])
    print(tok, "" if correct else "<*****!!!")

Run with 3.12.0a7:

% /usr/local/pyenv/pyenv/versions/3.12.0a7/bin/python tokbug.py
3.12.0a7 (main, Apr  5 2023, 05:51:58) [Clang 14.0.3 (clang-1403.0.22.14.1)]
TokenInfo(type=62 (NL), string='\n', start=(1, 0), end=(1, 1), line='\n')
TokenInfo(type=1 (NAME), string='a', start=(2, 0), end=(2, 1), line='a + \\\n')
TokenInfo(type=54 (OP), string='+', start=(2, 2), end=(2, 3), line='a + \\\n')
TokenInfo(type=1 (NAME), string='b', start=(3, 0), end=(3, 1), line='b\n')
TokenInfo(type=4 (NEWLINE), string='\n', start=(3, 1), end=(3, 2), line='b\n')
TokenInfo(type=0 (ENDMARKER), string='', start=(4, 0), end=(4, 0), line='')

Run with 3.12.0b1:

% /usr/local/pyenv/pyenv/versions/3.12.0b1/bin/python tokbug.py
3.12.0b1 (main, May 23 2023, 16:19:59) [Clang 14.0.3 (clang-1403.0.22.14.1)]
TokenInfo(type=65 (NL), string='\n', start=(1, 0), end=(1, 1), line='\n')
TokenInfo(type=1 (NAME), string='a', start=(2, 0), end=(2, 1), line='a + \\\n')
TokenInfo(type=55 (OP), string='+', start=(2, 2), end=(2, 3), line='a + \\\n')
TokenInfo(type=1 (NAME), string='b', start=(3, 0), end=(3, 1), line='a + \\\nb\n') <*****!!!
TokenInfo(type=4 (NEWLINE), string='\n', start=(3, 1), end=(3, 2), line='a + \\\nb\n') <*****!!!
TokenInfo(type=0 (ENDMARKER), string='', start=(4, 0), end=(4, 0), line='')

Related to #104825? cc @pablogsal

Linked PRs

• gh-104972: Ensure that line attributes in tokens in the tokenize module are correct #104975
• [3.12] gh-104972: Ensure that line attributes in tokens in the tokenize module are correct (GH-104975) #104982

Activity

1. nedbat

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on May 26, 2023
2. nedbat commented on May 26, 2023

   

   nedbat

   on May 26, 2023

   MemberAuthor

   More actions

   The tip of 3.12 shows one more style of change, due to omitting the newlines from .line:

   3.12.0b1+ (heads/3.12:6324458bef, May 26 2023, 06:25:21) [Clang 14.0.3 (clang-1403.0.22.14.1)]
   TokenInfo(type=65 (NL), string='\n', start=(1, 0), end=(1, 1), line='') <*****!!!
   TokenInfo(type=1 (NAME), string='a', start=(2, 0), end=(2, 1), line='a + \\')
   TokenInfo(type=55 (OP), string='+', start=(2, 2), end=(2, 3), line='a + \\')
   TokenInfo(type=1 (NAME), string='b', start=(3, 0), end=(3, 1), line='a + \\\nb') <*****!!!
   TokenInfo(type=4 (NEWLINE), string='\n', start=(3, 1), end=(3, 2), line='a + \\\nb') <*****!!!
   TokenInfo(type=0 (ENDMARKER), string='', start=(4, 0), end=(4, 0), line='')
   
3. pablogsal commented on May 26, 2023

   

   pablogsal

   on May 26, 2023

   Member

   More actions

   CC: @mgmacias95
4. pablogsal

   added 5 commits that reference this issue on May 26, 2023

   pythongh-104972: Ensure that line attributes in tokens in the tokeniz…

   c381380

   pythongh-104972: Ensure that line attributes in tokens in the tokeniz…

   0b23e0f

   pythongh-104972: Ensure that line attributes in tokens in the tokeniz…

   d477678

   pythongh-104972: Ensure that line attributes in tokens in the tokeniz…

   3d2f429

   gh-104972: Ensure that line attributes in tokens in the tokenize modu…

   3fdb55c
5. miss-islington

   added a commit that references this issue on May 26, 2023

   pythongh-104972: Ensure that line attributes in tokens in the tokeniz…

   25cf9f2
6. pablogsal commented on May 26, 2023

   

   pablogsal

   on May 26, 2023

   Member

   More actions

   @nedbat can you check with main now?
7. pablogsal

   closed this as completedon May 26, 2023
8. terryjreedy

   added a commit that references this issue on May 26, 2023

   [3.12] gh-104972: Ensure that line attributes in tokens in the tokeni…

   8ca2957
9. terryjreedy commented on May 26, 2023

   

   terryjreedy

   on May 26, 2023

   Member

   More actions

   If you set automerge on the backport, the usual test-hypothesis failure disabled it. I merged it.
10. nedbat commented on May 26, 2023

    

    nedbat

    on May 26, 2023

    MemberAuthor

    More actions

    @pablogsal Gorgeous! Thanks for the quick turnaround.

    Reacted by Pablo Galindo Salgado
11. pablogsal commented on May 26, 2023

    

    pablogsal

    on May 26, 2023

    Member

    More actions

    If you set automerge on the backport, the usual test-hypothesis failure disabled it. I merged it.

    Thanks a lot @terryjreedy !

Issue actions