Bug report

On Windows, PipeConnection._send_bytes() in multiprocessing.connection writes each message with a single WriteFile(). Windows refuses a pipe write of more than 4,294,967,247 bytes (0xFFFFFFCF) with ERROR_INVALID_PARAMETER, so send() and send_bytes() of a larger message raise OSError: [WinError 87] The parameter is incorrect.

Worse, when the other end is already waiting in recv() or recv_bytes(), Windows completes that pending read, which asks for 128 bytes, with 128 bytes and ERROR_MORE_DATA, although nothing of the message is left in the pipe. _get_more_data() then finds nothing left and fails at assert left > 0. This script shows it; it needs about 4 GiB of free memory:

import multiprocessing as mp
import threading
import time

if __name__ == "__main__":
    reader, writer = mp.Pipe(duplex=False)
    receiver = threading.Thread(target=reader.recv_bytes, daemon=True)
    receiver.start()
    time.sleep(0.5)  # let the receiver start waiting
    try:
        writer.send_bytes(bytes(0xFFFFFFD0))
    except Exception as e:
        print("send_bytes():", repr(e))
    receiver.join(1)
    print("receiver still waiting:", receiver.is_alive())

Output with 3.14.7 (3.15.2 and main behave the same; paths shortened):

Exception in thread Thread-1 (recv_bytes):
Traceback (most recent call last):
  File "...\Lib\threading.py", line 1082, in _bootstrap_inner
    self._context.run(self.run)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "...\Lib\threading.py", line 1024, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\Lib\multiprocessing\connection.py", line 226, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "...\Lib\multiprocessing\connection.py", line 348, in _recv_bytes
    return_value = self._get_more_data(ov, maxsize)
  File "...\Lib\multiprocessing\connection.py", line 373, in _get_more_data
    assert left > 0
           ^^^^^^^^
AssertionError
send_bytes(): OSError(22, 'The parameter is incorrect', None, 87, None)
receiver still waiting: False

A Pool worker waiting for its next task dies the same way when a pickled task is larger than that, for example a task with a 4.5 GB bytes argument on a spawn pool: the worker's traceback ends at the same assert left > 0 under task = get() in pool.py's worker(), and result.get() raises the OSError.

I measured the limit with overlapped WriteFile() calls on named pipes created the way Pipe() creates them, on Windows 11 version 26H2 (build 26300); I have not tried other versions. The largest write Windows accepted was 0xFFFFFFCF bytes in every combination I tried: writes from the client end of message-mode inbound pipes with input buffers of 0, 8 KiB, 64 KiB, 1 MiB and 16 MiB, of a message-mode duplex pipe and of byte-mode inbound pipes, and writes from the server end of message-mode and byte-mode duplex pipes. I could not find this limit documented. Messages from 2 GiB up to that size have worked since gh-78744 fixed reads larger than INT_MAX; one of exactly 0xFFFFFFCF bytes arrives whole on main.

The documentation of Connection.send() says: "Very large pickles (approximately 32 MiB+, though it depends on the OS) may raise a ValueError exception." send_bytes() says the same of very large buffers.

I have a fix ready: PipeConnection._send_bytes() checks the size and raises ValueError before calling WriteFile(), so nothing of the message reaches the pipe and both ends stay usable (the receiver keeps waiting, and later messages arrive). Code that catches OSError from send() would get ValueError instead, which is the exception the documentation names. Other Windows versions might set the limit a little lower; if you prefer a margin, a round limit such as 0xFFFF0000 would do as well. The fix comes with tests that fail before the change: one with the limit lowered, which runs in every Windows test run, and a bigmemtest with the real sizes.

The alternative is to support larger messages. gh-61760 did that for Connection (sockets, and pipes on POSIX) in 3.8 with a 64-bit length header; its review noted that PipeConnection would need message chunking instead (#10305). PipeConnection messages carry no header today, so that needs a new framing, which an older Python at the other end of a Listener/Client connection could not read. Hardening only the receiver would not be enough: _get_more_data() could take left == 0 after ERROR_MORE_DATA for a refused message and keep waiting, but when the sender goes on to send another message before the receiver looks for the rest, the receiver cannot tell. In a test where the sender caught the error and sent the next message at once, today's receiver returned the 128 bytes and that next message as one message.

Would the ValueError check be acceptable, or would you rather have larger messages supported?

I used AI tools (Claude Code with Claude Opus 5.5) to investigate this, write and test the fix and draft this text, and other AI agents reviewed it; I have reviewed it myself.

CPython versions tested on:

3.14, 3.15, CPython main branch

Operating systems tested on:

Windows

Linked PRs

• gh-161668: Refuse multiprocessing messages too large for a Windows pipe #161685