GeezerVison is showing github.com — Deprecate pickle support for itertools · Issue #101588 · python/cpython
Deprecate pickle support for itertools #101588DescriptionPickle support was long ago added to some itertools. It was done mostly to support an atypical use case for a single company. It was implemented in a very inefficient manner, essentially replaying iteration from the beginning to the mid-stream state where it was frozen. The implementation was of low quality and had many bugs. Also, it was not a documented or advertised feature. Newer itertools don't support pickling and no one has noticed or cared. The popular third-party package more-itertools is implemented with generators which do not have pickle support — again, none of their users seems to have noticed or cared.
IMO, this is just cruft that has made maintenance more difficult and we should get rid of it. As an undocumented feature, we could just remove it directly. But to be on the safe side, we can go through a deprecation cycle.
Linked PRs
Reacted by Oleg Iarygin, Frank Sachsenheim, Ramin RX7, Gouvernathor, Ivan Nazarov, Sander Maijers, Agriya Khetarpal and EDM115 Reacted by Arusekk ActivityJosephSBoyle commented on Feb 21, 2023 ZackerySpytz commented on Feb 21, 2023 @JosephSBoyle I am already working on a patch for this issue. Please find a different issue to work on. rhettinger commented on Feb 21, 2023 ContributorAuthor More actions Thanks for working on this. When the PR is ready for review, please assign to me :-) rhettinger commented on Apr 4, 2023 ContributorAuthor More actions ZackerySpytz commented on Apr 5, 2023 I still plan to create a pull request for this issue (within the next two weeks, hopefully). JosephSBoyle commented on Apr 5, 2023 Hi @rhettinger, I'd be happy to work on this.
@ZackerySpytz, perhaps we could work together somehow? Up to you - I appreciate that you were working on this before I volunteered to help. Gouvernathor commented on Apr 18, 2023 I was about to use the pickling feature on my end, and I wouldn't have noticed this had I not checked in the doc, noticed it wasn't there and propose it to be in the issue above.
So, I wouldn't be surprised if others than me use the feature as well, and while I fully understand and support the rationale for deprecating, I think making it follow a deprecation cycle wouldn't hurt.
In my experience and opinion, iterators in python generally fail spectacularly when you try to pickle them, so having the itertools one not fail may have felt for many to be a low-key guarantee on that being a supported feature, even if it was not explicitly documented (and even though it's not a very good practice). rhettinger commented on Apr 24, 2023 ContributorAuthor More actions @JosephSBoyle You no longer need to wait. There is no particular reason for @ZackerySpytz to reserve this one. There are plenty of other open bugs. JosephSBoyle commented on Apr 24, 2023 Sure, @rhettinger. Where would you recommend starting on this? rhettinger commented on May 24, 2023 ContributorAuthor More actions Thomas, can this deprecation still go into 3.12? It deprecates an undocumented feature. Yhg1s commented on May 24, 2023 It may be undocumented but pickle use is so viral, it's very likely depended on without people even knowing :) Let's get the deprecation into beta 2 if we can (it's scheduled for next week), but given that it's not actually problematic and just something we don't want to support, I would suggest holding back on removing it until we have a clearer picture of what it would break. serhiy-storchaka commented on May 9, 2024 I missed when this was deprecated, but I support this. It was inefficient, it leaked internal state, and it often not worked as expected for complex iterators like chain or tee.
Should not this also be deprecated for builtins zip, map and filter, and for iterators of builtin collections? Reacted by Gouvernathor, inoperable and Agriya Khetarpal Grissess commented on Apr 11, 2025 A little late to the party, but exposing this surface makes for a relatively easy crash in Py3.13.1:
Python 3.13.1 (main, Dec 28 2024, 13:25:56) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import itertools
>>> tdo = itertools._tee_dataobject(object(), [1], None)
>>> tob = itertools._tee(())
>>> tob.__setstate__((tdo, 0))
<python-input-3>:1: DeprecationWarning: Pickle, copy, and deepcopy support will be removed from itertools in Python 3.14.
tob.__setstate__((tdo, 0))
>>> next(tob)
1
>>> next(tob)
Segmentation fault This seems to work on any non-iterable object in the first argument to the TDO constructor. I suppose it's because the TDO calls PyIter_Next on this object without ever having done a PyIter_Check.
Arguably that's a separate bug, but removing this support should probably see the interface to _tee_dataobject from user code go away. TDOs seem to be a fragile internal detail in any case. Reacted by Enzan umo and Joshua-Luke Bevan
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|