Bug report

Bug description:

SyntaxError.offset and SyntaxError.end_offset for the error
"Non-UTF-8 code starting with ..." are too small when a valid multi-byte
character precedes the invalid byte on the same line.

Reproduction

compile(b'\xc3\xa9X\x80', '<bug>', 'exec')

Same via a file:

$ printf '\xc3\xa9X\x80' > bug.py
$ ./python bug.py

The input is "éX" followed by the invalid byte 0x80, with no encoding
cookie. The invalid byte is the 3rd character of line 1.

Expected

  File "<bug>", line 1
    éX
      ^
SyntaxError: Non-UTF-8 code starting with '\x80' on line 1, but no encoding declared; ...

offset == 3, end_offset == 3 (1-based character column).

Actual

  File "<bug>", line 1
    éX
     ^
SyntaxError: Non-UTF-8 code starting with '\x80' on line 1, but no encoding declared; ...

offset == 2, end_offset == 2.

Observed values:

More inputs on main, all one column short per preceding multi-byte character:

compile(b'\xc3\xa9abc\x80', ...)  # offset 4, expected 5
compile(b'\t\xc3\xa9X\x80', ...)  # offset 3, expected 4
compile(b'a\xc3\xa9b\x80c', ...)  # offset 3, expected 4

Cause

_PyTokenizer_ensure_utf8() (Parser/tokenizer/helpers.c) computes a 1-based
character column and passes it to _PyTokenizer_syntaxerror_known_range():

_PyTokenizer_syntaxerror_known_range(tok,
        col_offset + 1, col_offset + 1, ...)

Since 59a6913 (gh-156894, PR #156901), _syntaxerror_range() converts its
col_offset/end_col_offset arguments from bytes to characters with
byte_col_to_char_col(), so the character column from ensure_utf8() is
converted a second time.

The same function is unchanged on current main (b1e7554).

Suggested fix

Pass byte columns from ensure_utf8(), e.g. (int)(badchar - line_start) + 1
for both arguments, and let _syntaxerror_range() do the byte-to-character
conversion.

Tests

Lib/test/test_source_encoding.py only checks the message text for this error;
no test checks offset/end_offset, so this case is not covered.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-157378: Fix SyntaxError.offset for "Non-UTF-8 code" error #157412
• [3.14] gh-157378: Fix SyntaxError.offset for "Non-UTF-8 code" error (GH-157412) #157489
• [3.15] gh-157378: Fix SyntaxError.offset for "Non-UTF-8 code" error (GH-157412) #158123