GeezerVison is showing github.com — pickle docs: Pickler's file.write() can receive bytes-like objects, not only bytes · Issue #161803 · python/cpython
pickle docs: Pickler's file.write() can receive bytes-like objects, not only bytes #161803DescriptionDocumentation
The pickle docs (and the Pickler/dump docstrings) say:
The file argument must have a write() method that accepts a single bytes argument.
That is not what the picklers actually pass with protocol 4 and 5. write() can get a bytearray, a memoryview or a pickle.PickleBuffer. A custom file object written to the documented contract, for example one that returns len(data) from write(), works for small pickles. With protocol 5 it fails on large buffer-backed objects:
TypeError: object of type 'pickle.PickleBuffer' has no len()
For a non-byte-format buffer, such as array('q') wrapped in PickleBuffer, len(memoryview(data)) silently gives the element count, not the byte count.
Reproducer
import array, io, pickle
class RecordingFile:
def __init__(self):
self.types, self._out = [], io.BytesIO()
def write(self, data):
self.types.append(type(data).__name__)
return self._out.write(data)
SIZE = 1 << 20
values = {
"bytes": b"x" * SIZE,
"bytearray": bytearray(SIZE),
"PickleBuffer(bytearray)": pickle.PickleBuffer(bytearray(SIZE)),
"PickleBuffer(array q)": pickle.PickleBuffer(array.array("q", range(SIZE // 8))),
}
for impl in (pickle.Pickler, pickle._Pickler):
for name, value in values.items():
f = RecordingFile()
impl(f, protocol=5).dump(value)
print(impl.__module__, name, sorted(set(f.types) - {"bytes"}) or "-")
class LenCountingFile:
def write(self, data):
return len(data)
pickle.Pickler(LenCountingFile(), protocol=5).dump(pickle.PickleBuffer(bytearray(SIZE)))
# TypeError: object of type 'pickle.PickleBuffer' has no len()
Observed (protocol 5, 1 MiB values; identical on 3.10, 3.11, 3.13, 3.14 and 3.15)
| Value |
C _pickle.Pickler passes |
Pure-Python pickle._Pickler passes |
bytes |
bytes |
memoryview |
bytearray |
bytearray |
bytearray, memoryview |
PickleBuffer(...) |
PickleBuffer |
memoryview |
Other cases:
- With the pure-Python pickler, even small protocol 4/5 pickles go through
write(memoryview), because _Framer.commit_frame() writes f.getbuffer(). Protocols 2 and 3 pass only bytes.
- In the C pickler, payloads at or above the 64 KiB frame size target bypass the output buffer.
_Pickler_write_bytes() and save_picklebuffer() call write() with the original object.
Prior art
bpo-44439 / gh-88605 hit the same mismatch: LZMAFile.write() / BZ2File.write() used len(data) and failed with protocol 5. Those write() methods were fixed to use memoryview(data).nbytes (GH-26764), but the pickle documentation was not updated, so third-party file-like objects keep hitting it.
Proposal
This is a docs-only change; the behaviour is intentional for zero-copy and efficiency.
- Change "accepts a single bytes argument" to "accepts a single bytes-like object argument" in
Doc/library/pickle.rst, the Lib/pickle.py docstring and the two Modules/_pickle.c clinic docstrings.
- Add a short note: depending on the protocol and the object being pickled,
write() may receive bytes, bytearray, memoryview or PickleBuffer instances, which may not support len(). Implementations that need the size of the data should use memoryview(data).nbytes.
I'm happy to submit the PR.
Linked PRs
ActivityMalavika007 commented on Nov 11, 2026 helloo! I'd like to work on this issue. I'd be happy to take it up if it's still available. Thanks :D eachimei commented on Nov 11, 2026
helloo! I'd like to work on this issue. I'd be happy to take it up if it's still available. Thanks :D
Hi! It's addressed in #161804 Malavika007 commented on Nov 11, 2026
helloo! I'd like to work on this issue. I'd be happy to take it up if it's still available. Thanks :D
Hi! It's addressed in #161804
Ohh, got it! Thanks for letting me know
MetadataMetadataStatusShow more project fields StatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|