GeezerVison is showing github.com — Null characters in strings cause a C SystemError · Issue #97556 · python/cpython
Null characters in strings cause a C SystemError #97556DescriptionCrash report
Putting a null byte into a Python string causes a SystemError in Python 3.10, due to a call to strlen in the string parsing library. In Python 3.9, the following example runs without errors:
# -*- coding: latin-1 -*-
"""
<NULL>
"""
In Python 3.10, it raises SystemError: ../Parser/string_parser.c:219: bad argument to internal function.
Internally, the new string_parser library introduced in v3.10.0a1 uses a call to strlen to determine the string size, which is getting thrown off by the null byte. This call is actually unnecessary, as the length has already been calculated by the calling parser and can be retrieved with PyBytes_AsStringAndSize.
Error messages
For single line strings, the error is SystemError: Negative size passed to PyUnicode_New
For multiline strings, the error is SystemError: ../Parser/string_parser.c:219: bad argument to internal function
Linked PRs
Activityvstinner commented on Sep 26, 2022 This issue seems like a duplicate of the issue #96670. apccurtiss commented on Sep 26, 2022 Ah, you're totally right. I found that one earlier, but misunderstood the core bug and re-discovered it. Thanks! apccurtiss commented on Sep 26, 2022 Actually, I just re-read that issue, and I was a bit over-eager with closing this. I feel like this is a very specific fix which doesn't actually solve their use case: This particular issue only shows up in 3.10, and causes an internal parse error. That linked issue is present in every 3.x version I've tested, including after this fix. I believe a patch elsewhere in the parser is needed for that. gvanrossum commented on Sep 26, 2022 apccurtiss commented on Sep 26, 2022 bskinn commented on Apr 24, 2023 I cannot repro OP's SystemError on this. I do observe Python 3.12's fix for null characters in single-quoted strings (SyntaxError: source code cannot contain null bytes per #97594, when run with 3.12.0a7), so going forward I'm focusing on triple-quoted strings.
For Python 3.8.16, 3.9.16, and 3.10.0a1 (WSL Debian local builds), crasher.py with the following contents parses successfully (without error, I did not check correctness) when executed from the commandline ($ python3.<#> crasher.py):
<BLANKLINE>
'''
<NUL>
'''
For Python 3.10.0b1 (WSL Debian local), 3.10.0 (WSL Debian local), 3.10.11 (WSL Debian local and Win11), 3.11.3 (WSL Debian local and Win11), and (importantly) 3.12.0a7 (WSL Debian local), parsing fails with SyntaxError: unterminated triple-quoted string literal.
Interestingly, the attributed line number of the exception location changes in 3.12.0a7. It's reported as line 3 in 3.12.0a7, and as line 2 for all other versions tested.
Regardless, the triple-quoted string case is not yet fixed in the fashion indicated by #96670 (comment), where strings containing null characters are explicitly rejected with an exception and message like SyntaxError: source code cannot contain null bytes. bskinn commented on Apr 25, 2023 gvanrossum commented on Apr 25, 2023 added 2 commits that reference this issue on May 3, 2023pythongh-97556: Raise null bytes syntax error upon null in multiline … gh-97556: Raise null bytes syntax error upon null in multiline string (… vstinner commented on May 4, 2023 Script to reproduce the issues:
with open("script.py", "w") as fp:
print("# -*- coding: latin-1 -*-", file=fp)
print('"""', file=fp)
print('\0', file=fp)
print('"""', file=fp)Current behavior:
- main branch (3.12): SyntaxError: unterminated triple-quoted string literal (detected at line 3)
- 3.11: SystemError: Parser/string_parser.c:230: bad argument to internal function
Oh, it seems like the main branch was just changed (40 min ago) by PR #104136 (just merged).
@lysnikolaou: Do you consider backporting your change to Python 3.11? bskinn commented on May 4, 2023 With the merge of #97594 and #104136, I believe what OP has raised is now resolved in main/3.12, and per #96670 (comment) the changes are not to be backported.
Thus, I think this issue can be closed. vstinner commented on May 4, 2023 SystemError: Parser/string_parser.c:230: bad argument to internal function sounds like a bug in Python.
It's not as if Python 3.11 accepts NUL characters. The question is more about providing a better error message if it finds a NUL character. bskinn commented on May 4, 2023 It was @gvanrossum's call not to backport the original fix, and I wouldn't presume to speak authoritatively for him. But, I assume the motivation is the xkcd 1172 aspect of changing the error message string. pablogsal commented on May 4, 2023 I think we should backport this to 3.11 gvanrossum commented on May 4, 2023 bskinn commented on May 4, 2023 Should #97594 be backported also, then? lysnikolaou commented on May 4, 2023 Backporting both makes the most sense to me, since just improving the error message would require much more work with the tokenizer buffer working the way it is. If no one has any serious concerns over this, I can work on backporting the necessary PRs. gvanrossum commented on May 4, 2023 Reacted by Lysandros Nikolaou pablogsal commented on May 4, 2023
Backporting both makes the most sense to me, since just improving the error message would require much more work with the tokenizer buffer working the way it is. If no one has any serious concerns over this, I can work on backporting the necessary PRs.
Yeah, I prefer to backport the restriction to keep the changes to a minimum in 3.11 as making a bugfix for the error message can be much more complex 👍 Reacted by Lysandros Nikolaou lysnikolaou commented on May 7, 2023 Raising a SyntaxError when parsing NULL bytes has been backported to 3.11. I'm closing this. vstinner commented on May 9, 2023 I confirm that 3.11 and main branches now display the same output for ./python script.py (see #97556 (comment) to generate script.py).
File "/home/vstinner/python/main/../3.11/script.py", line 3
SyntaxError: source code cannot contain null bytes
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|