Description

polprog

opened on Sep 6, 2022

Issue body actions

Bug report

In some cases, when dealing with multi-line string with non-utf8 encoded files, python will throw a SystemError: Negative size passed to PyUnicode_New and not execute any code.

Minimal test case:

print("""
ą""")

This is only a problem if the non-utf8 character lies on a new line (at any point in the line)

A similar test case behaves correctly

print("""ą""")

And reports an encoding warning, which is the expected behavior

SyntaxError: Non-UTF-8 code starting with '\xb1' in file C:\Users\xxxxx\test.py on line 2, but no encoding declared; see https://python.org/dev/peps/pep-0263/ for details

Since this is an encoding related errors, both files are attached (as .txt, GitHub does not allow .py attachments).
test.txt - single line (correct behavior)
test_ml.txt - multi line (bug)

My environment

• CPython versions tested on: Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)] on win32
• Operating system and architecture: Windows 10 Pro 21H2 (19044.1826)

Activity

1. polprog

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Sep 6, 2022
2. mdboom commented on Sep 6, 2022

   

   mdboom

   on Sep 6, 2022

   Contributor

   More actions

   #96270 may fix this. Let me confirm.
3. mdboom

   added

   topic-unicode

   on Sep 6, 2022
4. mdboom commented on Sep 6, 2022

   

   mdboom

   on Sep 6, 2022

   Contributor

   More actions

   #96270 may fix this. Let me confirm.

   It does not fix this.
5. mdboom

   self-assigned this

   on Sep 6, 2022
6. mdboom commented on Sep 6, 2022

   

   mdboom

   on Sep 6, 2022

   Contributor

   More actions

   Since copy-and-paste doesn't usually preserve broken encodings, this is a convenient way to reproduce the bug:

   open("x.py", "wb").write(b'print("""\n\xb1""")')
   

   $ python x.py
   
7. eryksun commented on Sep 6, 2022

   

   eryksun

   on Sep 6, 2022

   Contributor

   More actions

   In tok_get() in "Parser/tokenizer.c", the following code blindly handles EOF returned from tok_nextc() as if it's the end of the file.

   cpython/Parser/tokenizer.c

   Lines 1936 to 1948 in 6744490

   	/* Get rest of string */
   	while (end_quote_size != quote_size) {
   	c = tok_nextc(tok);
   	if (c == EOF || (quote_size == 1 && c == '\n')) {
   	assert(tok->multi_line_start != NULL);
   	// shift the tok_state's location into
   	// the start of string, and report the error
   	// from the initial quote character
   	tok->cur = (char *)tok->start;
   	tok->cur++;
   	tok->line_start = tok->multi_line_start;
   	int start = tok->lineno;
   	tok->lineno = tok->first_lineno;

   In this case, however, tok->done is E_DECODE instead of E_EOF. This gets set by error_ret(), which also clears tok->start and tok->cur to NULL. The above code increments tok->cur to 1. Subsequently, _syntaxerror_range() tries to decode the text for the syntax error using the negative size 1 - tok->line_start.
8. eryksun

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   on Sep 6, 2022
9. mdboom

   added a commit that references this issue on Sep 6, 2022

   pythongh-96611: Fix error message for invalid UTF-8 in mid-multiline …

   709d7e1
10. pablogsal

    added a commit that references this issue on Sep 6, 2022

    gh-96611: Fix error message for invalid UTF-8 in mid-multiline string (…

    05692c6
11. miss-islington

    added 4 commits that reference this issue on Sep 6, 2022

    pythongh-96611: Fix error message for invalid UTF-8 in mid-multiline …

    e0b323b

    pythongh-96611: Fix error message for invalid UTF-8 in mid-multiline …

    e971282

    gh-96611: Fix error message for invalid UTF-8 in mid-multiline string (…

    b6af933

    gh-96611: Fix error message for invalid UTF-8 in mid-multiline string (…

    bb0dab5
12. mdboom commented on Sep 7, 2022

    

    mdboom

    on Sep 7, 2022

    Contributor

    More actions

    Thanks for the report, @polprog, and the diagnostics, @eryksun.
13. mdboom

    closed this as completedon Sep 7, 2022

Issue actions