GeezerVison is showing github.com — Pickle ignores custom getstate methods on TextIOWrapper in Python 3.12 · Issue #122559 · python/cpython
Pickle ignores custom getstate methods on TextIOWrapper in Python 3.12 #122559DescriptionBug report
Bug description:
So I am not entirely sure whether this is unintended behaviour, but it is definitely a noticeable change between 3.11 and 3.12 that is rather unintuitive
import pickle
from io import BytesIO, TextIOWrapper
class EncodedFile(TextIOWrapper):
def __getstate__(self):
return "string"
def __setstate__(self, state):
pass
file = EncodedFile(BytesIO(b"string"))
pickle.dumps(file) This works in Python 3.11 and 3.10, but fails in 3.12 with
pickle.dumps(file)
TypeError: cannot pickle 'EncodedFile' instances CPython versions tested on:
3.10, 3.11, 3.12
Operating systems tested on:
macOS
Linked PRs
Activitytomasr8 commented on Aug 1, 2024 SimonSorgQC commented on Aug 1, 2024 Interesting! While it makes sense that the original IOBase can not be pickled, I feel like it should be possible for subclasses overriding getstate SimonSorgQC commented on Aug 1, 2024 So I can verify that overriding reduce and reduce_ex worked for me. I would leave the issue open for now as I do not think getstate and setstate not working is nice - especially since reduce and reduce_ex of IOBase are buried in the C code.
Please feel free to correct me if you think this has been resolved. serhiy-storchaka commented on Aug 2, 2024 Pickling file objects was explicitly forbidden in bpo-10180 by adding __getstate__ methods that raise TypeError. These methods were removed from the C implementation in bpo-33138, because these classes were recognized as non-pickleable by default. Then these classes were made heap types in #101948 (gh-101819), and new __reduce__ and __reduce_ex__ methods that raise TypeError were added, because condition that prevented them from pickling was gone.
The right way is to remove __reduce__ and __reduce_ex__ methods and restore __getstate__ methods. This will make the C implementation consistent with the Python implementation. serhiy-storchaka commented on Aug 2, 2024 It turned out that this issue is a bit more complex. There are more differences between implementations. Classes are not tested separately, existing tests do not cover all cases. I am working on this problem. serhiy-storchaka commented on Aug 2, 2024 I added tests as a guard against such type of regression, but in general there are no guaranties about pickleability (especially with protocols 0 and 1).
In meantime, the workaround is to reset __reduce__ and __reduce_ex__ in your subclass to default:
class EncodedFile(TextIOWrapper):
...
__reduce__ = object.__reduce__
__reduce_ex__ = object.__reduce_ex__added 3 commits that reference this issue on May 4, 2025gh-122559: Synchronize C and Python implementation of the io module a…
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|