GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — pickle docs: Pickler's file.write() can receive bytes-like objects, not only bytes · Issue #161803 · python/cpython

pickle docs: Pickler's file.write() can receive bytes-like objects, not only bytes #161803

New issue
Copy link
New issue
Copy link
Open
Open
pickle docs: Pickler's file.write() can receive bytes-like objects, not only bytes#161803
Copy link
Labels
docsDocumentation in the Doc dirDocumentation in the Doc dir

Description

@eachimei
eachimei
opened on Nov 11, 2026
Issue body actions

Documentation

The pickle docs (and the Pickler/dump docstrings) say:

The file argument must have a write() method that accepts a single bytes argument.

That is not what the picklers actually pass with protocol 4 and 5. write() can get a bytearray, a memoryview or a pickle.PickleBuffer. A custom file object written to the documented contract, for example one that returns len(data) from write(), works for small pickles. With protocol 5 it fails on large buffer-backed objects:

TypeError: object of type 'pickle.PickleBuffer' has no len()

For a non-byte-format buffer, such as array('q') wrapped in PickleBuffer, len(memoryview(data)) silently gives the element count, not the byte count.

Reproducer

import array, io, pickle

class RecordingFile:
    def __init__(self):
        self.types, self._out = [], io.BytesIO()
    def write(self, data):
        self.types.append(type(data).__name__)
        return self._out.write(data)

SIZE = 1 << 20
values = {
    "bytes": b"x" * SIZE,
    "bytearray": bytearray(SIZE),
    "PickleBuffer(bytearray)": pickle.PickleBuffer(bytearray(SIZE)),
    "PickleBuffer(array q)": pickle.PickleBuffer(array.array("q", range(SIZE // 8))),
}
for impl in (pickle.Pickler, pickle._Pickler):
    for name, value in values.items():
        f = RecordingFile()
        impl(f, protocol=5).dump(value)
        print(impl.__module__, name, sorted(set(f.types) - {"bytes"}) or "-")

class LenCountingFile:
    def write(self, data):
        return len(data)

pickle.Pickler(LenCountingFile(), protocol=5).dump(pickle.PickleBuffer(bytearray(SIZE)))
# TypeError: object of type 'pickle.PickleBuffer' has no len()

Observed (protocol 5, 1 MiB values; identical on 3.10, 3.11, 3.13, 3.14 and 3.15)

Value C _pickle.Pickler passes Pure-Python pickle._Pickler passes
bytes bytes memoryview
bytearray bytearray bytearray, memoryview
PickleBuffer(...) PickleBuffer memoryview

Other cases:

  • With the pure-Python pickler, even small protocol 4/5 pickles go through write(memoryview), because _Framer.commit_frame() writes f.getbuffer(). Protocols 2 and 3 pass only bytes.
  • In the C pickler, payloads at or above the 64 KiB frame size target bypass the output buffer. _Pickler_write_bytes() and save_picklebuffer() call write() with the original object.

Prior art

bpo-44439 / gh-88605 hit the same mismatch: LZMAFile.write() / BZ2File.write() used len(data) and failed with protocol 5. Those write() methods were fixed to use memoryview(data).nbytes (GH-26764), but the pickle documentation was not updated, so third-party file-like objects keep hitting it.

Proposal

This is a docs-only change; the behaviour is intentional for zero-copy and efficiency.

  • Change "accepts a single bytes argument" to "accepts a single bytes-like object argument" in Doc/library/pickle.rst, the Lib/pickle.py docstring and the two Modules/_pickle.c clinic docstrings.
  • Add a short note: depending on the protocol and the object being pickled, write() may receive bytes, bytearray, memoryview or PickleBuffer instances, which may not support len(). Implementations that need the size of the data should use memoryview(data).nbytes.

I'm happy to submit the PR.

Linked PRs

  • gh-161803: Document that pickle's file.write() receives bytes-like objects #161804

Activity

  1. picnixz
    added
    docsDocumentation in the Doc dirDocumentation in the Doc dir
    on Nov 11, 2026
  2. github-project-automation
    added this to docs issueson Nov 11, 2026
  3. github-project-automation
    moved this to Todo in docs issueson Nov 11, 2026
  4. picnixz
    added this to Pickle and copy issues 🥒on Nov 11, 2026
  5. Malavika007 commented on Nov 11, 2026

    @Malavika007
    Malavika007
    on Nov 11, 2026
    More actions

    helloo! I'd like to work on this issue. I'd be happy to take it up if it's still available. Thanks :D

  6. eachimei commented on Nov 11, 2026

    @eachimei
    eachimei
    on Nov 11, 2026
    Author
    More actions

    helloo! I'd like to work on this issue. I'd be happy to take it up if it's still available. Thanks :D

    Hi! It's addressed in #161804

  7. Malavika007 commented on Nov 11, 2026

    @Malavika007
    Malavika007
    on Nov 11, 2026
    More actions

    helloo! I'd like to work on this issue. I'd be happy to take it up if it's still available. Thanks :D

    Hi! It's addressed in #161804

    Ohh, got it! Thanks for letting me know

    Reacted by eachimei
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    docsDocumentation in the Doc dirDocumentation in the Doc dir

    Projects

    • Pickle and copy issues 🥒
      Status
      No status
      Show more project fields
    • docs issues
      Status
      Todo
      Show more project fields

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

    You are visitor 01402503 — GeezerVison does not store the pages you look up.
    Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.