GeezerVison is showing github.com — Safer data serialization with marshal module · Issue #113626 · python/cpython
Safer data serialization with marshal module #113626DescriptionFeature or enhancement
The main purpose of the marshal module -- serialization of precompiled module code objects. This requires support of code objects, strings for names, and other primitive Python types and simple collection types referred by the code object.
It allows to use it as more data generic serialization tool -- more limited than pickle, but less limited than JSON. The marshal module supports different versions of the format and backward compatible with all earlier versions. But only if the data does not contain code objects. The format of the code objects changed with every Python version, and this is not reflected in marshal format version. Loading marshal data created in different Python version has undefined behavior if the data contains a code object.
I propose to add a keyword-only parameter allow_code with default value True in marshal functions. Specifying allow_code=False forbid saving and loading code objects. It allows to be safer when you load external data and to guarantee that the output can be safely loaded in other Python.
Linked PRs
Activityvaishnavi192 commented on Jan 6, 2024 changed the title [-]Safe data serialization with marshal module[/-] [+]Safer data serialization with marshal module[/+] on Jan 11, 2024gpshead commented on Jan 11, 2024 I appreciate that in your PR you are not describing marshal as "safe" or fit for any particular purpose. ie: the big Warning in https://docs.python.org/3/library/marshal.html always applies.
While I don't mind this feature, I wonder what the point is? Who is expected to use this and why?
We cannot make promises about the marshal data format safety and capabilities. It was never designed for such use.
ie: Anywhere someone should not use pickle, they should also not use marshal, even with allow_code=False. Given that, in what scenarios do you see this feature actually being used?
The marshal code hasn't been written with security in mind, nor do we fuzz test it. Nor do we do meaningful interoperability testing across versions. Because that hasn't been its purpose. serhiy-storchaka commented on Jan 12, 2024 It is still not safe because you can still create a malicious marshal data that has exponentially large complexity of loading, and the unmarshaller is perhaps vulnerable to stack overflow. But you need to construct it specially, not using the marshal module.
marshal is one of builtin data formats, it is often faster and more compact than pickle and JSON. It is safe if used with trusted input. It is used as a data format. If it was unsupported, we would not bother with versioning and backward compatibility. But if a code object is accidentally leaked in the marshalled data, unmarshalling it in other Python version can crash or corrupt the Python interpreter. This option will protect you from this kind of errors.
Reacted by Gregory P. Smith serhiy-storchaka commented on Jan 12, 2024 An example of malicious marshal data:
t = ()
for i in range(100):
t = (t, t)
k = (1, 2)
import marshal
data = marshal.dumps([k, t, {k: t}])
# replace the reference to k as the dict key with the reference to t
data = data.replace(b'r\0', b'r\3')
x = marshal.loads(data) # hangsgpshead commented on Jan 13, 2024 Thanks, that makes sense. I'm fine with adding this. I think our existing marshal documentation warns people well enough about its caveats.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|