GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-156894: Fix the position of syntax errors which cover a range by serhiy-storchaka · Pull Request #156901 · python/cpython

Conversation

@serhiy-storchaka

@serhiy-storchaka serhiy-storchaka commented Sep 3, 2026

Copy link
Copy Markdown
Member

SyntaxError.offset and end_offset are columns in characters, but all callers of _PyTokenizer_syntaxerror_known_range() pass columns in bytes, so the reported position drifts by one column per preceding non-ASCII character:

>>> compile("abcd = 00010", "<t>", "exec")     # before and after
  File "<t>", line 1
    abcd = 00010
           ^^^
>>> compile("αβγδ = 00010", "<t>", "exec")     # before
  File "<t>", line 1
    αβγδ = 00010
               ^
>>> compile("αβγδ = 00010", "<t>", "exec")     # after
  File "<t>", line 1
    αβγδ = 00010
           ^^^

The conversion is done in _syntaxerror_range(), so all three call sites are fixed at once, and the -1 case is unaffected: it already counts characters.

test_tokenize.test_tolerant_incompatible_prefix_position_after_non_ascii asserted the column in bytes, so its expected value changes from 6 to 5.

  • Issue: Wrong error position for tokenizer errors reporting a range if the line contains non-ASCII characters #156894

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@serhiy-storchaka @claude
pythongh-156894: Fix the position of syntax errors which cover a range
a1065a9 
The callers of _PyTokenizer_syntaxerror_known_range() pass columns in bytes,
but SyntaxError.offset and end_offset are columns in characters.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
@serhiy-storchaka
serhiy-storchaka requested review from iritkatriel, lysnikolaou and pablogsal as code owners September 3, 2026 16:34
@bedevere-app bedevere-app Bot mentioned this pull request Sep 3, 2026
Wrong error position for tokenizer errors reporting a range if the line contains non-ASCII characters #156894
Closed
@bedevere-app bedevere-app Bot added the awaiting core review label Sep 3, 2026
grayjk
grayjk reviewed Sep 3, 2026
View reviewed changes
Comment thread Lib/test/test_exceptions.py
# for the errors which cover a range
for source, offset, end_offset in [
('abcd = 00010', 8, 11),
('\u03b1\u03b2\u03b3\u03b4 = 00010', 8, 11),

@grayjk grayjk Sep 3, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

include a test with a unicode combining character (e.g. \u0301) which has zero display width

Suggested change
('\u03b1\u03b2\u03b3\u03b4 = 00010', 8, 11),
('\u03b1\u03b2\u03b3\u03b4 = 00010', 8, 11),
('e\u0301'*4 = 00010', 8, 11),

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@serhiy-storchaka serhiy-storchaka Sep 4, 2026

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It counts code points, not visible width.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@serhiy-storchaka @claude
Add test cases with combining characters
14f49de 
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
@serhiy-storchaka serhiy-storchaka added needs backport to 3.13 only security fixes needs backport to 3.14 bugs and security fixes needs backport to 3.15 bugs and security fixes labels Sep 4, 2026
@serhiy-storchaka
serhiy-storchaka merged commit 59a6913 into python:main Sep 4, 2026
60 checks passed

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot removed the awaiting core review label Sep 4, 2026
@miss-islington-app

miss-islington-app Bot commented Sep 4, 2026

Copy link
Copy Markdown

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14, 3.15.
🐍🍒⛏🤖
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@serhiy-storchaka
serhiy-storchaka deleted the fix-156894-range-offsets branch September 4, 2026 11:14
@miss-islington-app

miss-islington-app Bot commented Sep 4, 2026

Copy link
Copy Markdown

Sorry, @serhiy-storchaka, I could not cleanly backport this to 3.15 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 59a691361cb7b1d4fde3b92f98a61490381c62c9 3.15
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@miss-islington-app miss-islington-app Bot assigned serhiy-storchaka Sep 4, 2026
@miss-islington-app

miss-islington-app Bot commented Sep 4, 2026

Copy link
Copy Markdown

Sorry, @serhiy-storchaka, I could not cleanly backport this to 3.14 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 59a691361cb7b1d4fde3b92f98a61490381c62c9 3.14
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@miss-islington-app

miss-islington-app Bot commented Sep 4, 2026

Copy link
Copy Markdown

Sorry, @serhiy-storchaka, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 59a691361cb7b1d4fde3b92f98a61490381c62c9 3.13
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app

bedevere-app Bot commented Sep 4, 2026

Copy link
Copy Markdown

GH-156926 is a backport of this pull request to the 3.15 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot removed the needs backport to 3.15 bugs and security fixes label Sep 4, 2026
@bedevere-app

bedevere-app Bot commented Sep 4, 2026

Copy link
Copy Markdown

GH-156927 is a backport of this pull request to the 3.14 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label Sep 4, 2026
@bedevere-app

bedevere-app Bot commented Sep 4, 2026

Copy link
Copy Markdown

GH-156928 is a backport of this pull request to the 3.13 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot removed the needs backport to 3.13 only security fixes label Sep 4, 2026
serhiy-storchaka added a commit that referenced this pull request Sep 4, 2026
@serhiy-storchaka @claude
[3.13] gh-156894: Fix the position of syntax errors which cover a ran…
e22d4a0 
…ge (GH-156901) (GH-156928)

The callers of _PyTokenizer_syntaxerror_known_range() pass columns in bytes,
but SyntaxError.offset and end_offset are columns in characters.

(cherry picked from commit 59a6913)

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
serhiy-storchaka added a commit that referenced this pull request Sep 4, 2026
@serhiy-storchaka @claude
[3.14] gh-156894: Fix the position of syntax errors which cover a ran…
113ad29 
…ge (GH-156901) (GH-156927)

The callers of _PyTokenizer_syntaxerror_known_range() pass columns in bytes,
but SyntaxError.offset and end_offset are columns in characters.

(cherry picked from commit 59a6913)

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
clin1234 pushed a commit to clin1234/cpython that referenced this pull request Sep 12, 2026
@serhiy-storchaka @claude @clin1234
pythongh-156894: Fix the position of syntax errors which cover a range (
29a9f7e 
pythonGH-156901)

The callers of _PyTokenizer_syntaxerror_known_range() pass columns in bytes,
but SyntaxError.offset and end_offset are columns in characters.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
@ashm-dev ashm-dev mentioned this pull request Sep 12, 2026
Wrong SyntaxError.offset for "Non-UTF-8 code starting with ..." when a non-ASCII character precedes the invalid byte #157378
Closed
pablogsal pushed a commit that referenced this pull request Sep 24, 2026
@serhiy-storchaka
[3.15] gh-156894: Fix the position of syntax errors which cover a ran…
b4a2366 
…ge (GH-156901) (#156926)

The callers of `_PyTokenizer_syntaxerror_known_range()` pass columns in bytes, but `SyntaxError.offset` and `end_offset` are columns in characters.
(cherry picked from commit 59a6913)

Co-authored-by: Serhiy Storchaka <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pablogsal pablogsal Awaiting requested review from pablogsal pablogsal is a code owner
@lysnikolaou lysnikolaou Awaiting requested review from lysnikolaou lysnikolaou is a code owner
@iritkatriel iritkatriel Awaiting requested review from iritkatriel iritkatriel is a code owner
1 more reviewer
@grayjk grayjk grayjk left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@serhiy-storchaka serhiy-storchaka

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@serhiy-storchaka @grayjk
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.