GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-112943: Correctly compute end offsets for multiline tokens in the tokenize module by pablogsal · Pull Request #112949 · python/cpython

Conversation

@pablogsal

@pablogsal pablogsal commented Dec 11, 2023 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member
  • Issue: Change in tokenize.generate_tokens behaviour with non-ASCII #112943

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@pablogsal
pythongh-112943: Correctly compute end offsets for multiline tokens i…
de5cc3c 
…n the tokenize module
@pablogsal
pablogsal requested a review from lysnikolaou as a code owner December 11, 2023 00:50
@pablogsal pablogsal added the needs backport to 3.12 only security fixes label Dec 11, 2023
@bedevere-app bedevere-app Bot added the awaiting core review label Dec 11, 2023
@bedevere-app bedevere-app Bot mentioned this pull request Dec 11, 2023
Change in tokenize.generate_tokens behaviour with non-ASCII #112943
Closed
pablogsal
pablogsal commented Dec 11, 2023
View reviewed changes
Comment thread Python/Python-tokenize.c
}
if (token.end != NULL && token.end >= it->tok->line_start) {
end_col_offset = _PyPegen_byte_offset_to_character_offset(line, token.end - it->tok->line_start);
end_col_offset = _PyPegen_byte_offset_to_character_offset_raw(it->tok->line_start, token.end - it->tok->line_start);

@pablogsal pablogsal Dec 11, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it->tok->line_start holds the last line tokenised (the last line of the multi-line string) while line holds the entire multi-line string from the start to the end.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
serhiy-storchaka
serhiy-storchaka approved these changes Dec 11, 2023
View reviewed changes
Comment thread Lib/test/test_tokenize.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Dec 11, 2023
@pablogsal @serhiy-storchaka
Update test_tokenize.py
947414b 
Co-authored-by: Serhiy Storchaka <[email protected]>
@pablogsal
pablogsal enabled auto-merge (squash) December 11, 2023 11:24
@pablogsal
pablogsal merged commit a135a6d into python:main Dec 11, 2023
@pablogsal
pablogsal deleted the gh-112943 branch December 11, 2023 11:44
@bedevere-app bedevere-app Bot removed the awaiting merge label Dec 11, 2023
@miss-islington-app

miss-islington-app Bot commented Dec 11, 2023

Copy link
Copy Markdown

Thanks @pablogsal for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@miss-islington-app

miss-islington-app Bot commented Dec 11, 2023

Copy link
Copy Markdown

Sorry, @pablogsal, I could not cleanly backport this to 3.12 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker a135a6d2c6d503b186695f01efa7eed65611b04e 3.12
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@miss-islington-app miss-islington-app Bot assigned pablogsal Dec 11, 2023
@bedevere-app

bedevere-app Bot commented Dec 11, 2023

Copy link
Copy Markdown

GH-112957 is a backport of this pull request to the 3.12 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot removed the needs backport to 3.12 only security fixes label Dec 11, 2023
pablogsal added a commit to pablogsal/cpython that referenced this pull request Dec 11, 2023
@pablogsal
[3.12] pythongh-112943: Correctly compute end offsets for multiline t…
2fe4bf7 
…okens in the tokenize module (pythonGH-112949)

(cherry picked from commit a135a6d)

Co-authored-by: Pablo Galindo Salgado <[email protected]>
pablogsal added a commit that referenced this pull request Dec 11, 2023
@pablogsal
[3.12] gh-112943: Correctly compute end offsets for multiline tokens …
e4d2fb2 
…in the tokenize module (GH-112949) (#112957)

(cherry picked from commit a135a6d)
aisk pushed a commit to aisk/cpython that referenced this pull request Feb 11, 2024
@pablogsal @aisk
pythongh-112943: Correctly compute end offsets for multiline tokens i…
79e2905 
…n the tokenize module (python#112949)
Glyphack pushed a commit to Glyphack/cpython that referenced this pull request Sep 2, 2024
@pablogsal @Glyphack
pythongh-112943: Correctly compute end offsets for multiline tokens i…
5bc545c 
…n the tokenize module (python#112949)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@serhiy-storchaka serhiy-storchaka serhiy-storchaka approved these changes
@lysnikolaou lysnikolaou Awaiting requested review from lysnikolaou lysnikolaou is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@pablogsal pablogsal

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@pablogsal @serhiy-storchaka
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.