GeezerVison is showing github.com — mishandling of c-strings in parser · Issue #96670 · python/cpython
mishandling of c-strings in parser #96670DescriptionBug report
the parser mishandles lines containing null bytes when parsing source -- this allows the code to be misleadingly different from what it looks like.
I've been told by security@ that it is ok to post this publicly.
in the below example, <NUL> is an actual null byte:
x = '<NUL>' nothing to see here
';import os;os.system('echo pwnd')and the execution and appearance in the terminal:
$ cat t.py
x = '' nothing to see here
';import os;os.system('echo pwnd')
$ python3 t.py
pwndit appears that after splitting the source into lines, the individual lines are treated as c strings and so the null terminator is misinterpreted, jamming the string contents together and it executes similar to this:
x = '';import os;os.system('echo pwnd')note that if you want to write out a file like this here's a simple bit of code you can paste into an interactive prompt:
open('t.py', 'w').write("x = '\0' nothing to see here\n';import os;os.system('echo pwnd')\n")here is perhaps a shorter example:
open('t.py', 'w').write("x = 1\0 + 1\n+2\nprint(x)\n")I originally found this due to a bug report where the ast parser rejects code containing null bytes:
>>> import ast
>>> ast.parse("x = '\0'")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.8/ast.py", line 47, in parse
return compile(source, filename, mode, flags,
ValueError: source code string cannot contain null bytes
>>> ast.parse(b"x = '\0'")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.8/ast.py", line 47, in parse
return compile(source, filename, mode, flags,
ValueError: source code string cannot contain null bytesideally I would want the interpreter to reject files containing null bytes as a SyntaxError (and update the ast.parse error to a SyntaxError as well) -- though it appears there are some of these files in the wild -- such as https://github.com/univention/univention-corporate-server/blob/5.0-2/services/univention-ldb-modules/buildtools/bin/waf-svn
Your environment
- CPython versions tested on: 3.7 ... 3.11rc1 (though pretty sure this reproduces on all versions)
- Operating system and architecture: ubuntu 22.04, linux, x86_64
Linked PRs
Reacted by qexat, Adam Johnson, Stepan, crazyman and yj94 Reacted by Eddie Darling Reacted by Shaw, Ferran Jovell, Jasper Spaans, Dionisio E Alonso, Henry Schreiner and Zeyu (Zayne) Zhang Reacted by Marsha, Bernát Gábor, Amin Alaee, Dionisio E Alonso, Yaroslav Pankovych, Shakya Majumdar and Ruairidh MacLeod Activityencukou commented on Sep 8, 2022 There was a discussion about the larger issue of code being ”misleadingly different from what it looks like", and the general consensus was that this should be solved in editors, linters and code review tools, rather than in Python. IMO it was this thread but might need more digging.
That doesn't mean handling of \0 can't be changed. Or of control characters in general: see the informational PEP 672 for some more details.
Perhaps all unusual control characters should be banned? Even in strings? With a warning for 2 releases, according to the backwards compatibility policy – waf would need to switch to e.g. base64. Reacted by Gregory P. Smith asottile commented on Sep 8, 2022 ContributorAuthor More actions that was the rationale for making this public, but it's different than those as it's a mishandling by the parser rather than a quirk of how unicode displays Reacted by Henry Schreiner, suic86, NogNoa, saurabh chopra and Tushar Sadhwani ypankovych commented on Sep 9, 2022 Ugh, this is freakin amazing.
C:\Users\user>more t.py
"""
How do i print 'Hello World' in Python?
Here is how:
print('Hello World')
"""
And run it:
C:\Users\user>py t.py
Hello World
open('t.py', 'w').write("\0'''\0How do i print \'Hello World\' in Python?\n\0Here is how:\nprint(\'Hello World\')\0'''")
gpshead commented on Sep 13, 2022
ideally I would want the interpreter to reject files containing null bytes as a SyntaxError (and update the ast.parse error to a SyntaxError as well)
FWIW I'd be in favor of that behavior change as a Feature. I wouldn't backport it as a bug though. Reacted by Guido van Rossum, Tim Hatch, Zachary Ware and Tushar Sadhwani asottile commented on Sep 13, 2022 ContributorAuthor More actions there's kinda 3 things I think:
- convert
ValueError => SyntaxError in ast.parse (probably 3.12+)
- treat source files containing null bytes as
SyntaxError (probably also 3.12+)
- fix the thing considering it as a c string by using
(char*, size_t) or whatever function (probably backportable?)
does it make sense to pursue these three things and separately? gvanrossum commented on Sep 13, 2022
there's kinda 3 things I think:
1. convert `ValueError` => `SyntaxError` in `ast.parse` (probably 3.12+)
Agreed, it's changing behavior, so can't backport. I like SyntaxError for this situation. The root exception isn't in ast.parse(), it's in compile(), probably even deeper.
2. treat source files containing null bytes as `SyntaxError` (probably also 3.12+)
That's the biggest hole, should probably do this first.
3. fix the thing considering it as a c string by using `(char*, size_t)` or whatever function (probably backportable?)
Depends on where that fix is (can you tell I haven't looked at the source code yet? :-). If any of the affected functions are public it's going to be more difficult.
does it make sense to pursue these three things and separately?
Likely. Reacted by Nick and saurabh chopra Reacted by Nick gpshead commented on Sep 14, 2022
2. treat source files containing null bytes as `SyntaxError` (probably also 3.12+)
That's the biggest hole, should probably do this first.
Be careful about this one. Python pre-pended to raw binary data for the executed Python to locate within the file and use (embedded zip or other data) is a common idiom that must continue to work. gvanrossum commented on Sep 14, 2022
Be careful about this one. Python pre-pended to raw binary data for the executed Python to locate within the file and use (embedded zip or other data) is a common idiom that must continue to work.
Are you sure that works? Unlike Unix shells, Python parses the entire source file before executing any code. How would you get Python to ignore a blob of arbitrary binary data embedded in the source code, even if \0 is accepted? If the blob contains \n characters you can't hide it behind a # comment. I suppose you could prefix it with a """ quote, if you can arrange for the file to also end in """, and you're lucky that the blob doesn't contain embedded """ sequences.
But if I had to do something like that I'd probably just embed the Python code in a bash script as a "here" document and end the bash script with exit. gpshead commented on Sep 14, 2022 Oh you're right, I guess what I've seen do that is a bash+python+data hybrid monster. gvanrossum commented on Sep 14, 2022 Okay then we should be safe banning \0 in files starting with 3.12. Reacted by qexat and Tushar Sadhwani asottile commented on Sep 14, 2022 ContributorAuthor More actions I suspect one could probably hack something together like this unfortunately
# coding: latin1
with open(__file__, 'rb') as f:
contents = f.read().split(b'### BINARY\n')[1]
GARB = '''\
### BINARY
(actual binary here)
### BINARY
'''gvanrossum commented on Sep 14, 2022 You'd still have to arrange for the actual binary not to contain the sequence ''' -- because that would end the string started at GARB =. I don't see how setting the coding to Latin-1 changes matters. (Honestly it seems you're making the same mistake as Greg.) asottile commented on Sep 14, 2022 ContributorAuthor More actions latin1 is to prevent a decoding error while parsing the source -- I linked an in-the-wild example of such a file in the original post gvanrossum commented on Sep 14, 2022 Okay, that's impressive. It looks like the blob is lightly encoded -- \n and \r are encoded using #. and #&. (I'm curious what they'd do if the blob contains one of those sequences, there doesn't seem to be a way to quote them.)
The proposed change in 3.12 will break them, but they have version checks and they can just cope with it, I don't think we need to preserve this machine-dependent quirk forever. The code looks like it had to deal with various other versioning issues already (e.g. it tries to handle Python 2 and 3!). Reacted by Gregory P. Smith and Tushar Sadhwani gvanrossum commented on Sep 27, 2022 Closing this. Starting with Python 3.12, NUL bytes won't be allowed in source code read from files. We're not backporting this since this could be considered a feature by some. vstinner commented on Sep 28, 2022
Starting with Python 3.12, NUL bytes won't be allowed in source code read from files
It was changed by the commit aab01e3: PR #97594. CarliJoy commented on Oct 17, 2022 Reacted by Victor Stinner
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|