GeezerVison is showing github.com — Defining iterator in a separate class no longer works in 3.13 · Issue #128161 · python/cpython
Defining iterator in a separate class no longer works in 3.13 #128161Labels 3.13only security fixesonly security fixes3.14bugs and security fixesbugs and security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DescriptionBug report
Bug description:
Defining an interator for a class in a separate class no longer works properly in 3.13. With the following test_iter.py:
class list2(list):
def __iter__(self):
return list2iterator(self)
class list2iterator:
def __init__(self, X):
self._X = X
self._pointer = -1
def __next__(self):
self._pointer += 1
if self._pointer == len(self._X):
self._pointer = -1
raise StopIteration
return self._X[self._pointer]With Python 3.13.1 one gets:
>>> from test_iter import list2
>>> X=list2([1,2,3])
>>> [x for x in X]
Traceback (most recent call last):
File "<python-input-2>", line 1, in <module>
[x for x in X]
^
TypeError: 'list2iterator' object is not iterableWith Python 3.12.7 it works:
>>> from test_iter import list2
>>> X=list2([1,2,3])
>>> [x for x in X]
[1, 2, 3] Bisected to bcc7227e
CPython versions tested on:
3.12, 3.13
Operating systems tested on:
Linux
Linked PRs
ActivityEclips4 commented on Dec 21, 2024 ronaldoussoren commented on Dec 21, 2024 Reacted by Ethan Furman, Terry Jan Reedy and Stefan antonio-rojas commented on Dec 21, 2024 Eclips4 commented on Dec 22, 2024 @ronaldoussoren You're right, and this code is definitely doing something weird, but it works in 3.12:
example.py:
class list2(list):
def __iter__(self):
return list2iterator(self)
class list2iterator:
def __init__(self, X):
self._X = X
self._pointer = -1
def __next__(self):
self._pointer += 1
if self._pointer == len(self._X):
self._pointer = -1
raise StopIteration
return self._X[self._pointer]
target = list2([1, 2, 3])
for i in target:
print(i)
[print(i) for i in target]3.12.8:
eclips4@suffering ~/tmp> python3.12 example.py
1
2
3
1
2
3 3.13.1:
1
2
3
Traceback (most recent call last):
File "/Users/eclips4/tmp/example.py", line 23, in <module>
[print(i) for i in target]
^^^^^^
TypeError: 'list2iterator' object is not iterableThough it probably shouldn't work, it is definitely a regression.
Bytecode for [x for x in y] 3.12.8:
0 0 RESUME 0
1 2 LOAD_NAME 0 (y)
4 GET_ITER
6 LOAD_FAST_AND_CLEAR 0 (x)
8 SWAP 2
10 BUILD_LIST 0
12 SWAP 2
>> 14 FOR_ITER 4 (to 26)
18 STORE_FAST 0 (x)
20 LOAD_FAST 0 (x)
22 LIST_APPEND 2
24 JUMP_BACKWARD 6 (to 14)
>> 26 END_FOR
28 SWAP 2
30 STORE_FAST 0 (x)
32 RETURN_VALUE
>> 34 SWAP 2
36 POP_TOP
38 SWAP 2
40 STORE_FAST 0 (x)
42 RERAISE 0
ExceptionTable:
10 to 26 -> 34 [2]3.13.1:
0 RESUME 0
1 LOAD_NAME 0 (y)
GET_ITER
LOAD_FAST_AND_CLEAR 0 (x)
SWAP 2
L1: BUILD_LIST 0
SWAP 2
GET_ITER - problematic instruction
L2: FOR_ITER 4 (to L3)
STORE_FAST_LOAD_FAST 0 (x, x)
LIST_APPEND 2
JUMP_BACKWARD 6 (to L2)
L3: END_FOR
POP_TOP
L4: SWAP 2
STORE_FAST 0 (x)
RETURN_VALUE
-- L5: SWAP 2
POP_TOP
1 SWAP 2
STORE_FAST 0 (x)
RERAISE 0
ExceptionTable:
L1 to L4 -> L5 [2]ronaldoussoren commented on Dec 22, 2024
@ronaldoussoren You're right, and this code is definitely doing something weird, but it works in 3.12: example.py:
Right, the code from the OP is broken according to the data model, but we still did a change that's not backward compatible. I'm not deep enough into this to fully understand the ramifications. Reacted by Kirill Podoprigora efimov-mikhail commented on Dec 22, 2024 This behavior change is a direct consequence of change in bytecode generation in my PR. I don't think this is a bug. Reacted by Terry Jan Reedy and Kirill Podoprigora terryjreedy commented on Dec 22, 2024 The exact example worked in the CPython implementation of 3.12, but that may be an accident, as it should not have by the language definition. It might have already failed in 3.12 in other implementations. In any case, adding the following exposes the defect in list2iterable even in 3.12.
target2 = iter(target) # A list2iterator object
target3 = iter(target2) # Fails in 3.12.8
# TypeError: 'list2iterator' object is not iterable
If there is still a question about changing anything, Guido might remember the original design discussions. Reacted by Mikhail Efimov pochmann3 commented on Dec 23, 2024 Relevant bit from glossary: iterator:
CPython implementation detail: CPython does not consistently apply the requirement that an iterator define __iter__().
Reacted by Raymond Hettinger and Bénédikt Tran efimov-mikhail commented on May 26, 2025 I think we should decide what to do with this issue.
FYI, on current CPython 3.12 branch this code snippet works, but on 3.13, 3.14 and main branches it doesn't.
@picnixz @Eclips4 @markshannon
Basically, there are two ways which I can see as reliable:
- Just close the issue w/o any changes in codebase.
- Provide changes to 3.13 branch only just to fix some "very big change for minor release".
And some links on related issues and PRs:
#125038
#125178
#132351 efimov-mikhail commented on May 26, 2025 And maybe it's reasonable to do some changes in docs?
Some clear mention about generator expressions like (x for x in None), for example.
They aren't correct in Python 3.13.3 and before, an exception will be raised on this expression in code.
But on current 3.13 branch this code is correct.
Maybe it's better to remove all related changes in 3.13 branch?
And have exactly the same behavior as on 3.12?
@hugovk @terryjreedy @JelleZijlstra donhatch commented on May 27, 2025 The issue title could perhaps be improved.
Current title: "Defining iterator in a separate class no longer works in 3.13"
Suggested new title: "Iterator without __iter__ no longer works in 3.13.1"
Details:
- If I understand correctly, the iterator being defined in a separate class isn't really relevant; what's very relevant is that it doesn't have an
__iter__ member function
- it's significant that the version in which this broke is minor release 3.13.1, not 3.13 (I observe the example code succeeds in 3.13.0)
donhatch commented on May 27, 2025 The title and text of committed #132351 seems to imply that maybe it fixes this, i.e. reverts the behavior to pre-3.13.1 in which the example code works (because the generated bytcode has no redundant GET_ITER on something already known to be an iterator).
Is that correct?
That would make me very happy, since I would very much like to go back to happily ignoring the IMO ill-advised "requirement" that iterators be iterable (in fact it is my hope that some day that requirement might be officially removed, for a brighter less-confusing future in which Iterator and Iterable are two clearly distinct non-overlapping concepts). donhatch commented on May 27, 2025 @efimov-mikhail wrote:
And maybe it's reasonable to do some changes in docs?
Some clear mention about generator expressions like (x for x in None), for example.
They aren't correct in Python 3.13.3 and before, an exception will be raised on this expression in code.
But on current 3.13 branch this code is correct.
(Unfortunately I don't currently have anything higher than 3.13.0 to try this out on...)
In 3.13.0 I see that (x for x in None) throws TypeError: 'NoneType' object is not iterable which seems reasonable to me.
Are you saying that, in later versions, it returns something instead of throwing, and that it's correct? Why? efimov-mikhail commented on May 27, 2025
The title and text of committed #132351 seems to imply that maybe it fixes this, i.e. reverts the behavior to pre-3.13.1 in which the example code works (because the generated bytcode has no redundant GET_ITER on something already known to be an iterator).
Is that correct?
No, this is not correct.
On current main branch
-> % cat test_list2.py
class list2(list):
def __iter__(self):
return list2iterator(self)
class list2iterator:
def __init__(self, X):
self._X = X
self._pointer = -1
def __next__(self):
self._pointer += 1
if self._pointer == len(self._X):
self._pointer = -1
raise StopIteration
return self._X[self._pointer]
[print(i) for i in target]
-> % ./python test_list2.py
Traceback (most recent call last):
File "/home/sikko/projects/cpython/test_iter.py", line 21, in <module>
[print(i) for i in target]
^^^^^^
TypeError: 'list2iterator' object is not iterable
>>> dis.dis('[print(i) for i in target]')
0 RESUME 0
1 LOAD_NAME 0 (target)
GET_ITER
LOAD_FAST_AND_CLEAR 0 (i)
SWAP 2
L1: BUILD_LIST 0
SWAP 2
GET_ITER
L2: FOR_ITER 11 (to L3)
STORE_FAST 0 (i)
LOAD_NAME 1 (print)
PUSH_NULL
LOAD_FAST_BORROW 0 (i)
CALL 1
LIST_APPEND 2
JUMP_BACKWARD 13 (to L2)
L3: END_FOR
POP_ITER
L4: SWAP 2
STORE_FAST 0 (i)
RETURN_VALUE
-- L5: SWAP 2
POP_TOP
1 SWAP 2
STORE_FAST 0 (i)
RERAISE 0
ExceptionTable:
L1 to L4 -> L5 [2]This is because of changes in bytecode.
You can see two GET_ITER bytecodes for list comprehension here.
Suggested new title: "Iterator without __iter__ no longer works in 3.13.1"
It sounds reasonable for me. Although, this is not the only change.
(Unfortunately I don't currently have anything higher than 3.13.0 to try this out on...)
In 3.13.0 I see that (x for x in None) throws TypeError: 'NoneType' object is not iterable which seems reasonable to me. Are you saying that, in later versions, it returns something instead of throwing, and that it's correct? Why?
Yes, on current main, 3.14 and 3.13 branches we can see the following result:
>>> g = (x for x in None)
>>> g
<generator object <genexpr> at 0x7f923bb48130>
>>> list(g)
Traceback (most recent call last):
File "<python-input-4>", line 1, in <module>
list(g)
~~~~^^^
File "<python-input-2>", line 1, in <genexpr>
g = (x for x in None)
^^^^
TypeError: 'NoneType' object is not iterableYou can see some motivation here:
#125178 (comment)
#132351 (comment)
cc @markshannon @sobolevn Roman513 commented on Jun 8, 2025 This is fixed in 3.13.4:
Python 3.13.4 (main, Jun 4 2025, 17:14:13) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from test_iter import list2
>>> X=list2([1,2,3])
>>> [x for x in X]
[1, 2, 3]
But in 3.14.0b2 it sill does not work.
However it could be reintroduced again, related issue #135171 and fix #135225
If there's a decision how this should work, do we need some additional tests to make the behavior stable?
cc: @picnixz @markshannon serhiy-storchaka commented on Jun 10, 2025 Whatever we decide on the main branch, you're better off adding an __iter__ method returning self to your iterator. There's no guarantee that this will always work in CPython or other implementations. efimov-mikhail commented on Jun 22, 2025 Reacted by Hugo van Kemenade
MetadataMetadata3.13only security fixesonly security fixes3.14bugs and security fixesbugs and security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DevelopmentNo branches or pull requestsIssue actions
|