GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-101892: Fix SystemError when a callable iterator call exhausts the iterator by workingpayload · Pull Request #101896 · python/cpython

Conversation

@workingpayload

@workingpayload workingpayload commented Feb 14, 2023 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Updated Cython/objects/iterobject.c Line 217

  • Issue: Callable iterator can SystemError if call exhausts the iterator #101892

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@workingpayload
gh-101892 : Updated Cython/objects/iterobject.c Line 217
de17a82 
Updated Cython/objects/iterobject.c Line 217
@bedevere-bot bedevere-bot added the awaiting review label Feb 14, 2023
@bedevere-bot bedevere-bot mentioned this pull request Feb 14, 2023
Callable iterator can SystemError if call exhausts the iterator #101892
Closed
@blurb-it
📜🤖 Added by blurb_it.
3fba831
@arhadthedev arhadthedev added the interpreter-core (Objects, Python, Grammar, and Parser dirs) label Feb 14, 2023
@arhadthedev
Clarify the news entry
bee2af7
@Eclips4

Eclips4 commented Feb 14, 2023

Copy link
Copy Markdown
Member

It's good to coverage this case in tests.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
workingpayload and others added 2 commits February 14, 2023 16:17
@workingpayload
Merge branch 'main' into workingpayload/issue101892
25f4c06
@arhadthedev
Add a test
0c631e3
@arhadthedev

arhadthedev commented Feb 14, 2023

Copy link
Copy Markdown
Member

Added a test copied from the issue with readability changes.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@arhadthedev
Amend wording
506fb6a
OTheDev
OTheDev reviewed Feb 15, 2023
View reviewed changes
Comment thread Objects/iterobject.c Outdated

result = _PyObject_CallNoArgs(it->it_callable);
if (result != NULL) {
if (result != NULL && it->it_callable != NULL) {

@OTheDev OTheDev Feb 15, 2023

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, but this won't fix it. Notice the first if statement in the function. If it->it_callable == NULL is true, NULL is returned. So it->it_callable != NULL is already guaranteed at this point.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@workingpayload workingpayload Feb 15, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if we check it->it_sentinel != NULL ?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@arhadthedev
Rep
78168bf
OTheDev
OTheDev reviewed Feb 15, 2023
View reviewed changes

@OTheDev OTheDev left a comment •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like NULL is being passed to PyObject_RichCompareBool() through it->it_sentinel when it expects an object?

If we add the check

if (result != NULL && it->it_sentinel != NULL) {

and get rid of that Py_DECREF(result) and place a new

Py_XDECREF(result);

just before return NULL;

Then the code in the issue will raise a StopIteration (no longer SystemError) and tests pass.

I'm not entirely sure if this is correct, TBH, but this is my guess.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
OTheDev
OTheDev reviewed Feb 15, 2023
View reviewed changes
Comment thread Lib/test/test_iter.py Outdated
return NO_MORE
spam.need_reentrance = True
spam.iterator = iter(spam, NO_MORE)
next(spam.iterator)

@OTheDev OTheDev Feb 15, 2023

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the expected behaviour here? StopIteration?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@arhadthedev arhadthedev Feb 15, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for noticing the oversight! Fixed.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@arhadthedev
Address the review
188e388
@OTheDev

OTheDev commented Feb 15, 2023

Copy link
Copy Markdown
Contributor

@workingpayload What do you think? Worth a try?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@workingpayload
gh-101892 : Updated Cython/objects/iterobject.c Line 217
0f9cea1
@workingpayload

workingpayload commented Feb 15, 2023

Copy link
Copy Markdown
Contributor Author

@workingpayload What do you think? Worth a try?

Yeah! Why not?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
workingpayload and others added 3 commits February 15, 2023 13:14
@workingpayload
Merge branch 'main' into workingpayload/issue101892
6abb09a
@arhadthedev
Update test_iter.py
c8b6bad
@arhadthedev
Remove an unnecessary abstraction of reentrance
6931c20
arhadthedev
arhadthedev reviewed Feb 15, 2023
View reviewed changes
Comment thread Lib/test/test_iter.py Outdated
def spam():
if spam.need_reentrance:
spam.need_reentrance = False
list(spam.iterator) # Implicitly call ourselves

@arhadthedev arhadthedev Feb 15, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This evening (after getting access to my dev machine) I'll check whether the inner list() is required to throw StopIteration to trigger the bug. If no, I will simplify the test:

Suggested change
list(spam.iterator) # Implicitly call ourselves
try:
list(spam.iterator) # Implicitly call ourselves.
except StopIteration:
pass

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
OTheDev reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

@arhadthedev arhadthedev Feb 15, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got it wrong. list() cannot throw an exception because it returns a ready-to-use list (possibly empty) processing StopIteration by itself.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
arhadthedev added 3 commits February 15, 2023 22:47
@arhadthedev
Clarify comments once more
36b043c
@arhadthedev
Update 2023-02-14-09-08-48.gh-issue-101892.FMos8l.rst
340f668
@arhadthedev
Fix issues reported by patchcheck
fe1502d
furkanonder
furkanonder approved these changes Feb 15, 2023
View reviewed changes
@bedevere-bot bedevere-bot added awaiting core review and removed awaiting review labels Feb 15, 2023
@furkanonder

furkanonder commented Feb 15, 2023

Copy link
Copy Markdown
Contributor

LGTM. By the way PR title can be more meaningful. Ex: Fix SystemError when a callable iterator call exhausts the iterator.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@AlexWaygood
AlexWaygood requested review from kumaraditya303 and removed request for OTheDev March 4, 2023 10:11
@AlexWaygood AlexWaygood added awaiting change review and removed awaiting changes labels Mar 4, 2023
kumaraditya303
kumaraditya303 reviewed Mar 4, 2023
View reviewed changes
Comment thread Lib/test/test_iter.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Lib/test/test_iter.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@arhadthedev
Address the review
1ee2293
@arhadthedev

arhadthedev commented Mar 4, 2023

Copy link
Copy Markdown
Member

@kumaraditya303 Thank you, addressed.
OTheDev reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
kumaraditya303
kumaraditya303 approved these changes Mar 4, 2023
View reviewed changes
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting change review labels Mar 4, 2023
@kumaraditya303 kumaraditya303 added needs backport to 3.10 needs backport to 3.11 only security fixes labels Mar 4, 2023
@kumaraditya303
kumaraditya303 merged commit 705487c into python:main Mar 4, 2023
@miss-islington

miss-islington commented Mar 4, 2023

Copy link
Copy Markdown
Contributor

Thanks @workingpayload for the PR, and @kumaraditya303 for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10, 3.11.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the awaiting merge label Mar 4, 2023
@miss-islington

miss-islington commented Mar 4, 2023

Copy link
Copy Markdown
Contributor

Sorry, @workingpayload and @kumaraditya303, I could not cleanly backport this to 3.10 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 705487c6557c3d8866622b4d32528bf7fc2e4204 3.10
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot

bedevere-bot commented Mar 4, 2023

Copy link
Copy Markdown

GH-102418 is a backport of this pull request to the 3.11 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@miss-islington miss-islington assigned kumaraditya303 Mar 4, 2023
@bedevere-bot bedevere-bot removed the needs backport to 3.11 only security fixes label Mar 4, 2023
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Mar 4, 2023
@workingpayload @arhadthedev @miss-islington
pythongh-101892: Fix SystemError when a callable iterator call exha…
1d07066 
…usts the iterator (pythonGH-101896)

(cherry picked from commit 705487c)

Co-authored-by: Raj <[email protected]>
Co-authored-by: Oleg Iarygin <[email protected]>
miss-islington added a commit that referenced this pull request Mar 4, 2023
@miss-islington @workingpayload @arhadthedev
gh-101892: Fix SystemError when a callable iterator call exhausts t…
06a3bb8 
…he iterator (GH-101896)

(cherry picked from commit 705487c)

Co-authored-by: Raj <[email protected]>
Co-authored-by: Oleg Iarygin <[email protected]>
arhadthedev added a commit to arhadthedev/cpython that referenced this pull request Mar 4, 2023
@workingpayload @arhadthedev
pythongh-101892: Fix SystemError when a callable iterator call exha…
5573d36 
…usts the iterator (python#101896)

Co-authored-by: Oleg Iarygin <[email protected]>
(cherry picked from commit 705487c)
@bedevere-bot

bedevere-bot commented Mar 4, 2023

Copy link
Copy Markdown

GH-102422 is a backport of this pull request to the 3.10 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the needs backport to 3.10 label Mar 4, 2023
kumaraditya303 pushed a commit that referenced this pull request Mar 4, 2023
@arhadthedev @workingpayload
[3.10] gh-101892: Fix SystemError when a callable iterator call exh…
fe36778 
…austs the iterator (GH-101896) (#102422)

gh-101892: Fix `SystemError` when a callable iterator call exhausts the iterator (#101896)

Co-authored-by: Oleg Iarygin <[email protected]>
(cherry picked from commit 705487c)

Co-authored-by: Raj <[email protected]>
@OTheDev

OTheDev commented Mar 4, 2023

Copy link
Copy Markdown
Contributor

@arhadthedev Thanks for the continued work on this!
arhadthedev reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@workingpayload
workingpayload deleted the workingpayload/issue101892 branch March 5, 2023 07:42
hugovk pushed a commit to hugovk/cpython that referenced this pull request Mar 6, 2023
@workingpayload @arhadthedev @hugovk
pythongh-101892: Fix SystemError when a callable iterator call exha…
bf01a62 
…usts the iterator (python#101896)

Co-authored-by: Oleg Iarygin <[email protected]>
carljm added a commit to carljm/cpython that referenced this pull request Mar 6, 2023
@carljm
Merge branch 'main' into nodeaddict
33dda2a 
* main: (21 commits)
  pythongh-102192: Replace PyErr_Fetch/Restore etc by more efficient alternatives in sub interpreters module (python#102472)
  pythongh-95672: Fix versionadded indentation of get_pagesize in test.rst (pythongh-102455)
  pythongh-102416: Do not memoize incorrectly loop rules in the parser (python#102467)
  pythonGH-101362: Optimise PurePath(PurePath(...)) (pythonGH-101667)
  pythonGH-101362: Check pathlib.Path flavour compatibility at import time (pythonGH-101664)
  pythonGH-101362: Call join() only when >1 argument supplied to pathlib.PurePath() (python#101665)
  pythongh-102444: Fix minor bugs in `test_typing` highlighted by pyflakes (python#102445)
  pythonGH-102341: Improve the test function for pow (python#102342)
  Fix unused classes in a typing test (pythonGH-102437)
  pythongh-101979: argparse: fix a bug where parentheses in metavar argument of add_argument() were dropped (python#102318)
  pythongh-102356: Add thrashcan macros to filter object dealloc (python#102426)
  Move around example in to_bytes() to avoid confusion (python#101595)
  pythonGH-97546: fix flaky asyncio `test_wait_for_race_condition` test (python#102421)
  pythongh-96821: Add config option `--with-strict-overflow` (python#96823)
  pythongh-101992: update pstlib module documentation (python#102133)
  pythongh-63301: Set exit code when tabnanny CLI exits on error (python#7699)
  pythongh-101863: Fix wrong comments in EUC-KR codec (pythongh-102417)
  pythongh-102302 Micro-optimize `inspect.Parameter.__hash__` (python#102303)
  pythongh-102179: Fix `os.dup2` error reporting for negative fds (python#102180)
  pythongh-101892: Fix `SystemError` when a callable iterator call exhausts the iterator (python#101896)
  ...
carljm added a commit to carljm/cpython that referenced this pull request Mar 7, 2023
@carljm
Merge branch 'main' into inlinecomp2
ecb313c 
* main: (37 commits)
  pythongh-102192: Replace PyErr_Fetch/Restore etc by more efficient alternatives in sub interpreters module (python#102472)
  pythongh-95672: Fix versionadded indentation of get_pagesize in test.rst (pythongh-102455)
  pythongh-102416: Do not memoize incorrectly loop rules in the parser (python#102467)
  pythonGH-101362: Optimise PurePath(PurePath(...)) (pythonGH-101667)
  pythonGH-101362: Check pathlib.Path flavour compatibility at import time (pythonGH-101664)
  pythonGH-101362: Call join() only when >1 argument supplied to pathlib.PurePath() (python#101665)
  pythongh-102444: Fix minor bugs in `test_typing` highlighted by pyflakes (python#102445)
  pythonGH-102341: Improve the test function for pow (python#102342)
  Fix unused classes in a typing test (pythonGH-102437)
  pythongh-101979: argparse: fix a bug where parentheses in metavar argument of add_argument() were dropped (python#102318)
  pythongh-102356: Add thrashcan macros to filter object dealloc (python#102426)
  Move around example in to_bytes() to avoid confusion (python#101595)
  pythonGH-97546: fix flaky asyncio `test_wait_for_race_condition` test (python#102421)
  pythongh-96821: Add config option `--with-strict-overflow` (python#96823)
  pythongh-101992: update pstlib module documentation (python#102133)
  pythongh-63301: Set exit code when tabnanny CLI exits on error (python#7699)
  pythongh-101863: Fix wrong comments in EUC-KR codec (pythongh-102417)
  pythongh-102302 Micro-optimize `inspect.Parameter.__hash__` (python#102303)
  pythongh-102179: Fix `os.dup2` error reporting for negative fds (python#102180)
  pythongh-101892: Fix `SystemError` when a callable iterator call exhausts the iterator (python#101896)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@arhadthedev arhadthedev arhadthedev approved these changes
@furkanonder furkanonder furkanonder approved these changes
@kumaraditya303 kumaraditya303 kumaraditya303 approved these changes
+1 more reviewer
@OTheDev OTheDev OTheDev left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@kumaraditya303 kumaraditya303

Labels

interpreter-core (Objects, Python, Grammar, and Parser dirs)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

9 participants

@workingpayload @Eclips4 @arhadthedev @OTheDev @furkanonder @bedevere-bot @miss-islington @kumaraditya303 @AlexWaygood
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.