Description

pablogsal

opened on May 29, 2023

Issue body actions

Seems that some tools were relying on the implementation detail that the readline-like callable that is provided to functions in the tokenize module is called as tokens are emitted and not consumed in one go. Although this was never part of the contract and technically we don't need to change, it would make the implementation more efficient as we don't need to hold the entire input in memory at the same time and we won't break these tools.

Linked PRs

• gh-105069: Add a readline-like callable to the tokenizer to consume input iteratively #105070
• [3.12] gh-105069: Add a readline-like callable to the tokenizer to consume input iteratively (GH-105070) #105119

Activity

1. pablogsal commented on May 29, 2023

   

   pablogsal

   on May 29, 2023

   MemberAuthor

   More actions

   CC: @lysnikolaou @mgmacias95
2. pablogsal

   added 15 commits that reference this issue on May 29, 2023

   pythongh-105069: Add a readline-like callable to the tokenizer to con…

   ff4d45c

   pythongh-105069: Add a readline-like callable to the tokenizer to con…

   bc6d2da

   pythongh-105069: Add a readline-like callable to the tokenizer to con…

   98a180c

   pythongh-105069: Add a readline-like callable to the tokenizer to con…

   0a661c4

   pythongh-105069: Add a readline-like callable to the tokenizer to con…

   713379f

   pythongh-105069: Add a readline-like callable to the tokenizer to con…

   35328a4

   pythongh-105069: Add a readline-like callable to the tokenizer to con…

   1b8ff7e

   pythongh-105069: Add a readline-like callable to the tokenizer to con…

   7caac01

   fixup! pythongh-105069: Add a readline-like callable to the tokenizer…

   8903d0d

   fixup! fixup! pythongh-105069: Add a readline-like callable to the to…

   d370087

   fixup! fixup! fixup! pythongh-105069: Add a readline-like callable to…

   2d6f0a6

   fixup! fixup! fixup! fixup! pythongh-105069: Add a readline-like call…

   0935371

   fixup! fixup! fixup! fixup! fixup! pythongh-105069: Add a readline-li…

   9990b7e

   fixup! fixup! fixup! fixup! fixup! fixup! pythongh-105069: Add a read…

   0598127

   gh-105069: Add a readline-like callable to the tokenizer to consume i…

   9216e69
3. pablogsal

   closed this as completedon May 30, 2023
4. miss-islington

   added a commit that references this issue on May 30, 2023

   pythongh-105069: Add a readline-like callable to the tokenizer to con…

   6de88f2
5. pablogsal

   added a commit that references this issue on May 31, 2023

   [3.12] gh-105069: Add a readline-like callable to the tokenizer to co…

   c687946

Issue actions