Description

lysnikolaou

opened on Oct 6, 2022

Issue body actions

The tokenizer currently only holds information about the current line number (along with the starting line number for strings that span multiple lines). This makes computing column offset harder, since it has to be done with pointer arithmetic using pointers to the beginning and end of the token (even more complicated when line continuations or multiline tokens happen).

Feature or enhancement

All of this can be significantly simplified, if we keep a column offset counter in the tokenizer state.

Activity

1. lysnikolaou

   added

   type-featureA feature request or enhancementA feature request or enhancement

   on Oct 6, 2022
2. lysnikolaou

   self-assigned this

   on Oct 6, 2022
3. lysnikolaou

   added a commit that references this issue on Oct 6, 2022

   pythongh-97997: Add col_offset field to tokenizer and use that for AS…

   faefec5
4. lysnikolaou

   linked a pull request that will close this issuegh-97997: Add col_offset field to tokenizer and use that for AST nodes #98000on Oct 7, 2022
5. pablogsal

   closed this as completedin #98000on Oct 7, 2022
6. pablogsal

   added a commit that references this issue on Oct 7, 2022

   gh-97997: Add col_offset field to tokenizer and use that for AST nodes (

   3de08ce
7. carljm

   added a commit that references this issue on Oct 8, 2022

   Merge branch 'main' into typewatch

   27a4980
8. mpage

   added a commit that references this issue on Oct 11, 2022

   pythongh-97997: Add col_offset field to tokenizer and use that for AS…

   7a39f02

Issue actions