Description

pekkaklarck

opened on May 23, 2023

Issue body actions

I tested Python 3.12 beta 1 with Robot Framework and noticed that tokenize.generate_tokens() handles expressions containing $ differently than earlier. Earlier $ yielded ERRORTOKEN but nowadays we get OP:

Python 3.11.3 (main, Apr  5 2023, 14:15:06) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from tokenize import generate_tokens
>>> from io import StringIO
>>> next(generate_tokens(StringIO('$x').readline))
TokenInfo(type=60 (ERRORTOKEN), string='$', start=(1, 0), end=(1, 1), line='$x')

Python 3.12.0b1 (main, May 22 2023, 23:31:26) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from tokenize import generate_tokens
>>> from io import StringIO
>>> next(generate_tokens(StringIO('$x').readline))
TokenInfo(type=55 (OP), string='$', start=(1, 0), end=(1, 1), line='$x\n')

We support Python evaluation with special variables like $var > 1 in Robot Framework data and this change breaks our tokenizing code. I didn't notice anything related in the release notes and decided to report this. If the change is intentional, we can easily update our code to handle also these semantics.

Notice also that there's a small change with TokenInfo.line above. With Python 3.12 there's an additional \n even though the original string didn't contain any newlines.

Activity

1. pekkaklarck

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on May 23, 2023
2. sunmy2019 commented on May 23, 2023

   

   sunmy2019

   on May 23, 2023

   Member

   More actions

   I didn't notice anything related in the release notes

   It just makes it to the deadline. I think the docs can be updated later.
   #104323 (comment)
3. pekkaklarck commented on May 23, 2023

   

   pekkaklarck

   on May 23, 2023

   Author

   More actions

   If I understand the referenced comment correctly, generate_tokens nowadays uses a totally different tokenizer underneath and there can be subtle differences like this one. If that's the case, this likely won't be considered a bug and the old behavior won't be restored. That's fine, I can update our code, but mentioning this under backwards incompatible changes in the release notes would be a good idea.

   Reacted by René
4. arhadthedev

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   on May 23, 2023
5. AlexWaygood

   assigned

   pablogsal

   and

   lysnikolaou

   on May 23, 2023
6. pablogsal commented on May 23, 2023

   

   pablogsal

   on May 23, 2023

   Member

   More actions

   Thanks for raising this with us! I can confirm that this is a expected side effect of the change. Docs are being worked on here:

   #104824

   I am closing this issue as we will cover the docs part on the linked PR.

   Reacted by Pekka Klärck
7. pablogsal

   closed this as not plannedon May 23, 2023
8. pablogsal commented on May 23, 2023

   

   pablogsal

   on May 23, 2023

   Member

   More actions

   Opened #104825 for the new line character

   Reacted by Pekka Klärck
9. fsc-eriker

   added a commit that references this issue on Feb 1, 2024

   Update for pythonGH-104802: register a defect instead of raising an e…

   a614ff1
10. fsc-eriker

    added a commit that references this issue on Feb 14, 2024

    Update for pythonGH-104802: don't register dot defect if there is a s…

    6db0a64
11. zware

    added a commit that references this issue on Feb 14, 2024

    Update for pythonGH-104802: don't register dot defect if there is a s…

    03efb0d

Issue actions