GeezerVison is showing github.com — Token lines in the tokenize module always end in new line · Issue #104825 · python/cpython
Token lines in the tokenize module always end in new line #104825DescriptionAs a consequence of the changes in the Python tokenizer post PEP701, now the line attribute always ends in a new line character. We should remove this as it adds no information and wasn't there before.
Linked PRs
Activitylysnikolaou commented on May 24, 2023 Yeah, I ran across that while working on #104798. Are you already working on a fix or shall I take it up? pablogsal commented on May 24, 2023
Yeah, I ran across that while working on #104798. Are you already working on a fix or shall I take it up?
I am working on a fix, should be up in half an hour or so Reacted by Lysandros Nikolaou added 2 commits that reference this issue on May 24, 2023gh-104825: Remove implicit newline in the line attribute in tokens em… lysnikolaou commented on May 24, 2023 #104846 now removes a newline, even when a newline exists in the original source.
❯ ./python.exe
Python 3.13.0a0 (heads/setuptools-peg-generator-dirty:c7503656c2, May 24 2023, 12:24:20) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tokenize
>>> import io
>>> file = io.StringIO('1 + 2 + 3\n')
>>> for token in tokenize.generate_tokens(file.readline):
... print(token)
...
TokenInfo(type=2 (NUMBER), string='1', start=(1, 0), end=(1, 1), line='1 + 2 + 3')
TokenInfo(type=55 (OP), string='+', start=(1, 2), end=(1, 3), line='1 + 2 + 3')
TokenInfo(type=2 (NUMBER), string='2', start=(1, 4), end=(1, 5), line='1 + 2 + 3')
TokenInfo(type=55 (OP), string='+', start=(1, 6), end=(1, 7), line='1 + 2 + 3')
TokenInfo(type=2 (NUMBER), string='3', start=(1, 8), end=(1, 9), line='1 + 2 + 3')
TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 9), end=(1, 10), line='1 + 2 + 3')
TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')lysnikolaou commented on May 24, 2023 @pablogsal Do you think the above should be fixed? Will you work on it? pablogsal commented on May 24, 2023
@pablogsal Do you think the above should be fixed? Will you work on it?
Hummmmm, not sure how easy is going to be for us to distinguish between implicit and explicit newlines at the end. I would say, leave it like is now unless you disagree, in which case let's see if we can fix it easily. lysnikolaou commented on May 24, 2023 My only concern is that tests that rely on that will fail. For example, test_pegen now fails:
def test_repeat_0_simple(self) -> None:
grammar = """
start: thing thing* NEWLINE
thing: NUMBER
"""
parser_class = make_parser(grammar)
node = parse_string("1 2 3\n", parser_class)
self.assertEqual(
node,
[
TokenInfo(NUMBER, string="1", start=(1, 0), end=(1, 1), line="1 2 3\n"),
[
TokenInfo(
NUMBER, string="2", start=(1, 2), end=(1, 3), line="1 2 3\n"
),
TokenInfo(
NUMBER, string="3", start=(1, 4), end=(1, 5), line="1 2 3\n"
),
],
TokenInfo(
NEWLINE, string="\n", start=(1, 5), end=(1, 6), line="1 2 3\n"
),
],
)
node = parse_string("1\n", parser_class)
self.assertEqual(
node,
[
TokenInfo(NUMBER, string="1", start=(1, 0), end=(1, 1), line="1\n"),
[],
TokenInfo(NEWLINE, string="\n", start=(1, 1), end=(1, 2), line="1\n"),
],
)The failure is:
======================================================================
FAIL: test_repeat_0_simple (test.test_peg_generator.test_pegen.TestPegen.test_repeat_0_simple)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/lysnikolaou/repos/python/cpython/Lib/test/test_peg_generator/test_pegen.py", line 248, in test_repeat_0_simple
self.assertEqual(
AssertionError: Lists differ: [Toke[67 chars]1 2 3'), [TokenInfo(type=2 (NUMBER), string='2[201 chars] 3')] != [Toke[67 chars]1 2 3\n'), [TokenInfo(type=2 (NUMBER), string=[209 chars]\n')]
First differing element 0:
TokenInfo(type=2 (NUMBER), string='1', start=(1, 0), end=(1, 1), line='1 2 3')
TokenInfo(type=2 (NUMBER), string='1', start=(1, 0), end=(1, 1), line='1 2 3\n')
Diff is 1018 characters long. Set self.maxDiff to None to see it.
The fix is very simple, so I'm okay with going with this as-is. I will fix test_pegen in my setuptools PR. nedbat commented on May 26, 2023 You say:
As a consequence of the changes in the Python tokenizer post PEP701, now the line attribute always ends in a new line character. We should remove this as it adds no information and wasn't there before.
But it looks to me like the lines used to always end in newline, except for ENDMARKER. This uses tokbug.py from #104972:
3.10.9 (main, Dec 7 2022, 07:15:24) [Clang 12.0.0 (clang-1200.0.32.29)]
TokenInfo(type=62 (NL), string='\n', start=(1, 0), end=(1, 1), line='\n')
TokenInfo(type=1 (NAME), string='a', start=(2, 0), end=(2, 1), line='a + \\\n')
TokenInfo(type=54 (OP), string='+', start=(2, 2), end=(2, 3), line='a + \\\n')
TokenInfo(type=1 (NAME), string='b', start=(3, 0), end=(3, 1), line='b\n')
TokenInfo(type=4 (NEWLINE), string='\n', start=(3, 1), end=(3, 2), line='b\n')
TokenInfo(type=0 (ENDMARKER), string='', start=(4, 0), end=(4, 0), line='')
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|