GeezerVison is showing github.com — Regression of 3.13.1 with iterator creation being duplicated · Issue #127682 · python/cpython
Regression of 3.13.1 with iterator creation being duplicated #127682Labels interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DescriptionBug report
Bug description:
For my Python compiler Nuitka, I use CPython as the oracle of what the correct behaviour is. I am running some tests that I used to clarify the behavior from decades ago, in this case I wanted to know when exactly the iterator creation is used. I striped this test a bunch, so the regression is still visible. I first noticed the issue on GitHub Actions, where 3.13.0 got replaced with 3.13.1 for Windows and Linux, but it applies to all OSes. See below for a diff, that the same iterator is created multiple times.
""" Generator expression tests
"""
from __future__ import print_function
import inspect
print("Generator expression that demonstrates the timing:")
def iteratorCreationTiming():
def getIterable(x):
print("Getting iterable", x)
return Iterable(x)
class Iterable:
def __init__(self, x):
self.x = x # pylint: disable=invalid-name
self.values = list(range(x))
self.count = 0
def __iter__(self):
print("Giving iterator now", self.x)
return self
def __next__(self):
print("Next of", self.x, "is", self.count)
if len(self.values) > self.count:
self.count += 1
return self.values[self.count - 1]
else:
print("Raising StopIteration for", self.x)
raise StopIteration
# Python2/3 compatibility.
next = __next__
def __del__(self):
print("Deleting", self.x)
gen = ((y, z) for y in getIterable(3) for z in getIterable(2))
print("next value is", next(gen))
res = tuple(gen)
print("remaining generator is", res)
try:
next(gen)
except StopIteration:
print("Usage past end gave StopIteration exception as expected.")
try:
print("Generator state then is", inspect.getgeneratorstate(gen))
except AttributeError:
pass
print("Its frame is now", gen.gi_frame)
print("Early aborting generator:")
gen2 = ((y, z) for y in getIterable(3) for z in getIterable(2))
del gen2
iteratorCreationTiming()The unified diff between 3.13.0 output (and basically all Python versions before) and 3.13.1 output.
--- out-3.13.0.txt 2024-12-06 12:37:19.447115100 +0100
+++ out-3.13.1.txt 2024-12-06 12:37:23.452239500 +0100
@@ -1,9 +1,11 @@
Generator expression that demonstrates the timing:
Getting iterable 3
Giving iterator now 3
+Giving iterator now 3
Next of 3 is 0
Getting iterable 2
Giving iterator now 2
+Giving iterator now 2
Next of 2 is 0
next value is (0, 0)
Next of 2 is 1
@@ -13,6 +15,7 @@
Next of 3 is 1
Getting iterable 2
Giving iterator now 2
+Giving iterator now 2
Next of 2 is 0
Next of 2 is 1
Next of 2 is 2
@@ -21,6 +24,7 @@
Next of 3 is 2
Getting iterable 2
Giving iterator now 2
+Giving iterator now 2
Next of 2 is 0
The duplicated prints out the iterator creation are new. This is not optimal and new. I don't know if the iterator being through a slot cause cause this or what it is. I checked if generator.c changed but I think it didn't at all.
My self compiled Python 3.13.1 for Linux and the official Windows download agree in behaviour.
CPython versions tested on:
3.13
Operating systems tested on:
Linux, Windows
Linked PRs
Activitybrianschubert commented on Dec 6, 2024 JelleZijlstra commented on Dec 6, 2024 This is an expected result from the change @brianschubert linked, which fixes a crash. Our belief was that __iter__ is supposed to return self on iterators, so it's safe to call it multiple times. kayhayen commented on Dec 6, 2024 It does return self though in my concrete code here. My goal was to see what generator expressions do when to be 100% compatible.
def __iter__(self):
print("Giving iterator now", self.x)
return self
But nobody ever promised to not have a side effect in there, and calling it twice is strange, which one of the two gets used for what there? Right now I cannot distinguish the two.
As for real-life code relevance, I am more than willing to admit this is garbage code. I just did a bunch of prints in iterator functions to see when they get called, and now they're getting called more often. Would it have to be a performance regression to make two calls? efimov-mikhail commented on Dec 6, 2024 Actually, there is a PR for main branch related to this issue:
#126408
It it will be accepted, then multiple calls of __iter__ be removed.
But on 3.13 current behavior is not a bug. markshannon commented on Dec 6, 2024 The problem is specific to generator expressions which have acted in a subtly different way from generators that (almost) no one noticed.
For a generator, any iteration occurs once the generator is executed. For generator expressions creating an iterator from the iterable happened when the generator expression is created. However there was no check that the iteration variable held an iterator when the generator expression executed, which could lead to a crash in exceptional circumstances.
This can be seen by disassembling this function:
def f(seq):
return (x for x in seq)Up to 3.13
2 LOAD_CONST 0 (<code object <genexpr> at 0x7f1ad8c25be0, file "<python-input-9>", line 2>)
MAKE_FUNCTION
LOAD_FAST 0 (seq)
GET_ITER
CALL 0
RETURN_VALUE
Disassembly of <code object <genexpr> at 0x7f1ad8c25be0, file "<python-input-9>", line 2>:
2 RETURN_GENERATOR
POP_TOP
L1: RESUME 0
LOAD_FAST 0 (.0)
L2: FOR_ITER 6 (to L3) # This is unsafe if .0 contains a non-iterator
...
Now:
2 LOAD_CONST 0 (<code object <genexpr> at 0x7f1ad8c25be0, file "<python-input-9>", line 2>)
MAKE_FUNCTION
LOAD_FAST 0 (seq)
GET_ITER
CALL 0
RETURN_VALUE
Disassembly of <code object <genexpr> at 0x7f1ad8c25be0, file "<python-input-9>", line 2>:
2 RETURN_GENERATOR
POP_TOP
L1: RESUME 0
LOAD_FAST 0 (.0)
GET_ITER
L2: FOR_ITER 6 (to L3)
...
What I would like:
2 LOAD_CONST 0 (<code object <genexpr> at 0x7f1ad8c25be0, file "<python-input-9>", line 2>)
MAKE_FUNCTION
LOAD_FAST 0 (seq)
CALL 0
RETURN_VALUE
Disassembly of <code object <genexpr> at 0x7f1ad8c25be0, file "<python-input-9>", line 2>:
2 RETURN_GENERATOR
POP_TOP
L1: RESUME 0
LOAD_FAST 0 (.0)
GET_ITER
L2: FOR_ITER 6 (to L3)
...
I think we should make generator expressions behave exactly like generators. It seems surprising that they would not. hroncok commented on Dec 6, 2024 Reacted by Mikhail Efimov and Chris Adams hroncok commented on Dec 11, 2024 efimov-mikhail commented on May 25, 2025 efimov-mikhail commented on Jun 22, 2025
MetadataMetadatainterpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DevelopmentNo branches or pull requestsIssue actions
|