GeezerVison

A few design considerations:

• I used the names Reader and Writer to match existing ABC names in typing/collections.abc. Alternatives are Readable and Writable, but I think they are used more rarely for these kind of (pseudo-)protocols. SupportsX would work for Writer, but not for Reader, since the latter supports multiple methods. (Any we maybe want to use SupportsRead etc. at some point for tighter protocols.)
• I would prefer these protocols to be in io, where they fit better thematically, but that would require importing typing with unforseeable (performance) implications for such a foundational module.
• I deliberated using tighter protocols, but for ease of use reasons – and since they are mostly an alternative to using IO et al. – I went with larger protocols for now.
• Seekable (with methods seek and tell) would be an interesting addition as well, but I think this should wait for easy protocol composition.

https://cpython-previews--127648.org.readthedocs.build/en/127648/library/typing.html#abcs-and-protocols-for-working-with-i-o

I would prefer these protocols to be in io, where they fit better thematically, but that would require importing typing with unforseeable (performance) implications for such a foundational module.

io feels like a more natural home to me too. Could you not do something similar to the pseudo-protocol ABCs in collections.abc, contextlib and os.path? The approach there is:

• Make the classes ABCs rather than protocols
• But document them as protocols
• And write __subclasshook__ methods so that they behave identically to runtime-checkable Protocols in their isinstance()/issubclass() behaviour:

  cpython/Lib/contextlib.py

  Lines 34 to 38 in 023b7d2

  	@classmethod
  	def __subclasshook__(cls, C):
  	if cls is AbstractContextManager:
  	return _collections_abc._check_methods(C, "__enter__", "__exit__")
  	return NotImplemented

• And special-case the ABCs in typing.py so that the typing module treats them identically to protocols:

  cpython/Lib/typing.py

  Lines 1940 to 1948 in 023b7d2

  	_PROTO_ALLOWLIST = {
  	'collections.abc': [
  	'Callable', 'Awaitable', 'Iterable', 'Iterator', 'AsyncIterable',
  	'AsyncIterator', 'Hashable', 'Sized', 'Container', 'Collection',
  	'Reversible', 'Buffer',
  	],
  	'contextlib': ['AbstractContextManager', 'AbstractAsyncContextManager'],
  	'os': ['PathLike'],
  	}

• And pretend in typeshed that they're protocols, so that type checkers treat them as protocols (I feel like python/typeshed@f554f54 missed the point there slightly)

That feels terribly hackish to me, even if there is precedence. And in my experience, hackish stuff often falls on one's feet one way or the other. But I can move it to io if that's the consensus.

This has been the way the collections.abc and contextlib ABCs have always worked. As far as I'm aware, it's worked pretty well up to now. And it doesn't actually feel like the most hackish part of collections.abc ;)

Well, considering that typing.is_protocol uses the _is_protocol marker to determine whether something is a protocol, I guess that's the official way to mark protocols at runtime, not the fact that they are derived from Protocol. Which contradicts the typing spec at the moment. This all seems a bit precarious to me.

I'm open to something less hacky if you can find a generalized solution that doesn't make _collections_abc depend on typing (which would significantly slow down startup for all Python applications, since _collections_abc is imported as part of Python's initialisation).

considering that typing.is_protocol uses the _is_protocol marker to determine whether something is a protocol, I guess that's the official way to mark protocols at runtime, not the fact that they are derived from Protocol. Which contradicts the typing spec at the moment.

I don't think the typing spec needs to be cognisant of the hacks we use to make things work at runtime in the stdlib! This knowledge is irrelevant for third-party users of protocols, to users of type checkers and to implementers of type checkers. It's definitely not encouraged for any external users to make use of the _is_protocol attribute.

CI seems flaky, failures are unrelated.

I'll leave it a little bit in case any of the other reviewers here have any remaining concerns before merging

Could this be merged before the next alpha next week?

I only saw the update now, great that it's merged!

I have a few comments. I hope they're not too bothersome - I just think these protocols are important so it's worth the effort.

Can the docs include type annotations? I know that Sphinx supports it. Given that this is meant for static typing, it makes sense to me to include.

I think it is important to document the expected semantics of these methods, otherwise you don't really know what you're going to get when you accept a Reader or Writer. What I can think of:

• What should happen on EOF?
• Is EOF sticky?
• Short reads/writes
• What does it mean to omit size?
• What happens on size 0?
• Is size allowed to be -1 or more generally negative?
• Can read return value be None? Empty?
• Can write return value be 0?
• Expected exceptions -- IOError?

I don't like the parameter name size, because from the caller perspective it is not exactly a size of anything. I would have preferred just n. But, I see this name is already used e.g. IOBase and consistency probably wins the argument here.

The docs say "If size is specified, it should be an integer, and at most size items (bytes/characters) will be read" and "Write data to the output stream and return the number of items (bytes/characters) written". I would drop the "(bytes/characters)" since T is not necessarily that.

The docs say "The following protocols can be used for annotating function and method arguments for simple stream reading or writing operations". I have two nitpicks:

• What is annotated is the parameter, not the argument (= a value for a parameter in a specific invocation). So would change "arguments" -> "parameters".
• I don't think it is just for simple operations, so I'd remove "simple".

I think T should have some bound. I think at least Sized, it's hard to interpret the return value without a notion of size.

A merged PR is probably not the best place to discuss this, but a few points:

Can the docs include type annotations?

Personally, I'd be fine with, and actually prefer if the docs used type annotations more, but I didn't want to divert from the existing style in the documentation.

I think it is important to document the expected semantics of these methods

While it would make sense to me to document some of those semantics (e.g. size must be >= 1), most are not part of this protocol. Exceptions and EOF are not concepts that are understood by those protocols.

I would drop the "(bytes/characters)" since T is not necessarily that.

It might have been clearer to say "(e.g. bytes or characters)", but I think calling out those specific types here makes the documentation clearer, as in 99% of cases it will be either of those. Otherwise the concept of "item" is a bit nebulous.

I think T should have some bound.

I don't think it's necessary to make the protocol more complex. Whoever is using the protocol will have to choose an type that they can work with.

Also in general, protocols are not a replacement for API documentation. It doesn't make too much sense to include expectations around a protocol that can't be checked statically. To document these expectations is the responsibility of the author of a function that uses these protocols, and to follow them is the responsibility of the user. These expectations can vary significantly from function to function.

(Please feel free to ignore this if you don't want to discuss this further -- I couldn't resist posting another comment...)

If I understand correctly, you prefer a narrow approach to these protocols because 1) they can't be checked statically 2) semantics can vary between uses. My replies:

1. In an ideal world, it would be possible to convey all semantics in a static way. But if it's not possible, documentation is the alternative. Stated differently, the Platonic ideal of the Reader interface contains all of its (agreed) semantics, but its projection onto the Python type system is incomplete, so the documentation should try its best to fill the gap. That's how I view it at least.

2. Why should the semantics vary between uses? A single set of semantics has a big advantage - it reduces (N impls * M uses) coordination points to N + M. As an implementer, I can (implicitly or explicitly) follow the protocol and don't need to invent or document my semantics. As a user, I write to a single set of semantics, not needing to handle multiple sets, defensively protect against unknown ones, or document my expectations. As a "middleware" (e.g. a BufferedReader which adds buffering to any reader), I can take any generic reader. What's the advantage of allowing implementations to be inconsistent on something like how EOF is signaled?

@srittau I've been working on refactoring the I/O tests and I don't think ProtocolsTest in test_io is being run at the moment.

test_io.py (now test_io/test_general.py) currently uses load_tests to customize what tests are found / run rather than the more typical auto discovery based on subclassing and the ProtocolsTest added in this PR but not added to the test list. Adding the test to the list for me it doesn't pass, not sure what your preferred way to get it passing would be; would appreciate if you could have a look.

You should be able to run just the ProtocolsTest by:

1. Update load_tests test list to include it:

   cpython/Lib/test/test_io/test_general.py

   Lines 5041 to 5051 in a2ba0a7

   	tests = (CIOTest, PyIOTest, APIMismatchTest,
   	CBufferedReaderTest, PyBufferedReaderTest,
   	CBufferedWriterTest, PyBufferedWriterTest,
   	CBufferedRWPairTest, PyBufferedRWPairTest,
   	CBufferedRandomTest, PyBufferedRandomTest,
   	StatefulIncrementalDecoderTest,
   	CIncrementalNewlineDecoderTest, PyIncrementalNewlineDecoderTest,
   	CTextIOWrapperTest, PyTextIOWrapperTest,
   	CMiscIOTest, PyMiscIOTest,
   	CSignalsTest, PySignalsTest, TestIOCTypes,
   	)

2. ./python -m test test_io.test_general -v -m 'ProtocolsTest'

With that I currently get:

./python -m test test_io.test_general -v -m 'ProtocolsTest'
== CPython 3.15.0a0 (heads/exp/test_bufferedio_split_v0-dirty:f4f150e5bbd, Sep 1 2025, 17:15:19) [Clang 20.1.8 ]
== Linux-6.16.4-arch1-1-x86_64-with-glibc2.42 little-endian
== Python build: debug
== cwd: <build_dir>/build/build/test_python_worker_790373æ
== CPU count: 32
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests

Using random seed: 1859464968
0:00:00 load avg: 0.74 Run 1 test sequentially in a single process
0:00:00 load avg: 0.74 [1/1] test_io.test_general
test_reader_subclass (test.test_io.test_general.ProtocolsTest.test_reader_subclass) ... ERROR
test_writer_subclass (test.test_io.test_general.ProtocolsTest.test_writer_subclass) ... ERROR

======================================================================
ERROR: test_reader_subclass (test.test_io.test_general.ProtocolsTest.test_reader_subclass)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/<source_dir>/cpython/Lib/test/test_io/test_general.py", line 5032, in test_reader_subclass
    self.assertIsSubclass(MyReader, io.Reader[bytes])
                          ^^^^^^^^
NameError: name 'MyReader' is not defined. Did you mean: 'self.MyReader'?

======================================================================
ERROR: test_writer_subclass (test.test_io.test_general.ProtocolsTest.test_writer_subclass)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/<source_dir>/cpython/Lib/test/test_io/test_general.py", line 5036, in test_writer_subclass
    self.assertIsSubclass(MyWriter, io.Writer[bytes])
                          ^^^^^^^^
NameError: name 'MyWriter' is not defined. Did you mean: 'self.MyWriter'?

----------------------------------------------------------------------
Ran 2 tests in 0.003s

FAILED (errors=2)
test test_io.test_general failed
0:00:00 load avg: 0.74 [1/1/1] test_io.test_general failed (2 errors)

== Tests result: FAILURE ==

1 test failed:
    test_io.test_general

Total duration: 55 ms
Total tests: run=2 (filtered)
Total test files: run=1/1 (filtered) failed=1
Result: FAILURE

@cmaloney Thanks, #138369 should fix that! Do you think it would make sense to add a warning to the top of the file (similar to the warning about testing both the C and Python versions)?

@srittau PR looks good. I'm hoping to remove the custom load_tests in test_io to resolve. Found this while experimenting with that :).

#138366 gets load_tests to just have a list of tests, I have a couple PRs I'm planning after that to split up the list (into files which use autodetect), then a PR to move to autodetect for what remains in test_general; main change there is making the "helper" classes being found/run by autodetect by removing their direct inheritance from unittest.TestCase.