GeezerVison is showing github.com — Buffer protocol types · Issue #593 · python/typing
Buffer protocol types #593DescriptionWe had several typeshed issues and pull requests lately that try to work around the fact that there is no way to express that a method receives any object following the buffer protocol. The typing documentation mentions BytesTypeByteString, which is an alias for Union[bytes, memoryview, bytearray] (and that bytes can be used as an alias in argument types), but this is missing other types such as array.array or user-defined objects. As this is a C API protocol, just defining such a protocol in typeshed in not possible. Reacted by dgelessus, Stephan Hoyer, Titusz, Patrick Reader, Sergei Izmailov, David Lechner, Nathaniel Manista, Marco, GalaxySnail and Bar Harel ActivityJelleZijlstra commented on Nov 20, 2018 Since the buffer protocol is a standard Python feature and we need to be able to use it in type annotations, it makes sense to me to add typing.Buffer or similar. Perhaps we can make concrete buffer classes inherit from Buffer, so we'd do something like
# typing.pyi
class Buffer: ... # empty at the Python level
# builtins.pyi
class bytes(Buffer, Sequence[int]): ... We'd have to add Buffer also to typing_extensions.
(Side note: I think you mean ByteString (https://docs.python.org/3/library/typing.html#typing.ByteString), not BytesType.) ilevkivskyi commented on Nov 20, 2018 I like what @JelleZijlstra proposes (obviously the type should be abstract). JelleZijlstra commented on Nov 20, 2018 Actually, it's a bit more complicated, since some buffers are writable and others aren't (see the types in python/typeshed#2610). This is controlled by whether the type responds to requests with PyBuf_WRITABLE (https://docs.python.org/3/c-api/buffer.html#c.PyBUF_WRITABLE) set. So here's a revised proposal:
# typing.pyi
class ReadableBuffer: ... # abstract, no Python attributes; corresponds to C types that expose buffers without PyBUF_WRITABLE set
class WriteableBuffer(ReadableBuffer): ... # same; corresponds to C types that expose buffers with PyBUF_WRITABLE set
# builtins.pyi
class bytes(ReadableBuffer, Sequence[int]): ...
class bytearray(WritableBuffer, Sequence[int]): ... There are a number of other flags controlling format, dimensions, etc., but I'm not sure those could be easily expressed in the type system. Perhaps we could implement format flag by making Buffer generic over a typevar that is restricted to certain types, but Python types don't map cleanly to C types, so that doesn't seem like it would work well. Reacted by Sebastian Rittau, Ivan Levkivskyi, Emma Smith, Spencer Brown, dgelessus, Jakub Stasiak and Aaron Denney srittau commented on Mar 30, 2019 CollaboratorAuthor More actions ilevkivskyi commented on Apr 2, 2019
python/typeshed#2895 is one example where this could be useful.
It is fine to experiment with such things defined locally (with an underscore, like _Reader), we can put something in typing later, when we will have more experience.
ilevkivskyi commented on Apr 2, 2019 Oh sorry, this is a wrong issue, disregard my last comment. srittau commented on Aug 7, 2019 CollaboratorAuthor More actions Is this something that could be considered? What steps are necessary to continue? ilevkivskyi commented on Aug 7, 2019 @srittau If it is not too hard maybe you can directly make a PoC PR to typeshed, so that we can discuss the details (IIUC you want this to be a stub-only feature).
cc @gvanrossum christopher-hesse commented on Nov 26, 2019 rwarren commented on Mar 28, 2020 Until a proper type for the buffer protocol is available, would it make sense to at least partially fix this (in places like zlib) with "better-than-just-bytes" coverage workarounds? For example:
Union[bytes, bytearray, memoryview]
It seems like that would cover the vast majority of use cases.
An example of something missing from that type definition that works, at least, for zlib.compress is an array.array of bytes. I can't seem to figure out how to force to be an array of bytes from a typing perspective, though.
Also - typing.Bytestring (mentioned in the original post) doesn't seem appropriate in all cases since it looks like it is Sequence[int] in there. A sequence of ints is not accepted by zlib.decompress, for example, although my dusty memory uncertainly thinks that a sequence of ints was supposed to be legit for a true buffer protocol (I'm really not sure, though). covert-encryption commented on Jan 2, 2022 As mentioned in #997 it would also be useful to be able to specify length for any buffer types, in particular where a fixed length string is expected. itaisteinherz commented on Feb 7, 2022 Bump :)
@ilevkivskyi @srittau What's the process to getting this accepted? Does this require a new PEP? I'd be open to working on this, but I'm not sure where to start. jakirkham commented on Feb 7, 2022 A related question is how this would be handled in Python given the move to builtins for type hints (like with PEP 585) itaisteinherz commented on Feb 7, 2022 JelleZijlstra commented on Feb 7, 2022 I think it doesn't necessarily require a PEP: we could just add the types to typing.pyi and typing_extensions.pyi (as I suggested in #593 (comment) a long time ago). The process could be similar to what we just did with reveal_type(): a typing-sig discussion, followed by direct implementation in CPython. Reacted by Sebastian Rittau jakirkham commented on Feb 23, 2022 JelleZijlstra commented on Apr 22, 2022 Reacted by GalaxySnail and Ralf Gommers srittau commented on Apr 22, 2022 CollaboratorAuthor More actions @JelleZijlstra LGTM so far, although it's unfortunate that it's difficult to distinguish between readable and read/writable buffers, but it makes sense. rgommers commented on Apr 25, 2022
although it's unfortunate that it's difficult to distinguish between readable and read/writable buffers, but it makes sense.
Readonly is only one of a number of attributes that are important for determining whether a buffer can be used. Some libraries can only deal with contiguous buffers, or native endianness, or aligned data. It looks to me like the PEP does the right thing here - best to support either all attributes or none, but not make readonly more important than other attributes. JelleZijlstra commented on Apr 25, 2022 Reacted by GalaxySnail, Leo Fang, Shantanu and Bas van Beek Reacted by Alex Waygood, dgelessus, Shantanu and ZhengYu, Xu JelleZijlstra commented on May 22, 2023 Reacted by jakirkham Reacted by jakirkham and Zac Hatfield-Dodds Reacted by jakirkham and Zac Hatfield-Dodds
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|