GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-108851: Fix tomllib recursion tests by vstinner · Pull Request #108853 · python/cpython

Conversation

@vstinner

@vstinner vstinner commented Sep 3, 2023 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member
  • sys.setrecursionlimit() now allows setting the limit to the current recursion depth.
  • Add get_recursion_available() and get_recursion_depth() functions to the test.support module.
  • Change infinite_recursion() default max_depth from 75 to 100.
  • Fix test_tomllib recursion tests for WASI buildbots: reduce the recursion limit and compute the maximum nested array/dict depending on the current available recursion limit.
  • Issue: test_tomllib.test_inline_array_recursion_limit() failed on wasm32-wasi 3.11 buildbot #108851

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@vstinner
vstinner requested review from encukou and hauntsaninja as code owners September 3, 2023 20:39
@bedevere-bot bedevere-bot added the awaiting core review label Sep 3, 2023
@bedevere-bot bedevere-bot mentioned this pull request Sep 3, 2023
test_tomllib.test_inline_array_recursion_limit() failed on wasm32-wasi 3.11 buildbot #108851
Closed
@vstinner

vstinner commented Sep 3, 2023

Copy link
Copy Markdown
Member Author

This PR combines 3 changes at once. IMO it's better to change all of them at the same time to ease backports, rather than having 3 commits (more work for little benefits).

cc @serhiy-storchaka

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Sep 3, 2023

Copy link
Copy Markdown
Member Author

The test_tomllib failure can be reproduced on Linux by reducing the recursion limit. Use this patch:

diff --git a/Lib/test/test_tomllib/test_misc.py b/Lib/test/test_tomllib/test_misc.py
index a477a219fd..43357f2064 100644
--- a/Lib/test/test_tomllib/test_misc.py
+++ b/Lib/test/test_tomllib/test_misc.py
@@ -10,6 +10,8 @@
 import tempfile
 import unittest

+sys.setrecursionlimit(250)
+
 from . import tomllib


On the main branch, I reproduce the issue:

$ ./python -m test test_tomllib -v
(...)
======================================================================
ERROR: test_inline_array_recursion_limit (test.test_tomllib.test_misc.TestMiscellaneous.test_inline_array_recursion_limit)
----------------------------------------------------------------------
(...)
RecursionError: maximum recursion depth exceeded

======================================================================
ERROR: test_inline_table_recursion_limit (test.test_tomllib.test_misc.TestMiscellaneous.test_inline_table_recursion_limit)
----------------------------------------------------------------------
(...)
RecursionError: maximum recursion depth exceeded
(...)

With this PR and the patch to trigger the bug: test_tomllib pass successfully.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

brettcannon
brettcannon reviewed Sep 5, 2023
View reviewed changes
Comment thread Lib/test/support/__init__.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/support/__init__.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/support/__init__.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/support/__init__.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_support.py
# it can raise RecursionError
return
get_depth = support.get_recursion_depth()
print(f"test_recursive: {depth}/{limit}: "

@brettcannon brettcannon Sep 5, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should subTest() used instead of printing out the values?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@vstinner vstinner Sep 6, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a script run by test_get_recursion_depth(). You should not see the output unless the test crash or fails. It's not written with unittest.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/test/test_support.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_support.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_support.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_support.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_support.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@vstinner
pythongh-108851: Fix tomllib recursion tests
a1b3414
* Add get_recursion_available() and get_recursion_depth() functions
  to the test.support module.
* Change infinite_recursion() default max_depth from 75 to 100.
* Fix test_tomllib recursion tests for WASI buildbots: reduce the
  recursion limit and compute the maximum nested array/dict depending
  on the current available recursion limit.
* test.pythoninfo logs sys.getrecursionlimit()
* Enhance test_sys tests on sys.getrecursionlimit()
  and sys.setrecursionlimit().
@vstinner
vstinner force-pushed the support_recursive branch from b0104a7 to a1b3414 Compare September 5, 2023 20:27
@vstinner

vstinner commented Sep 5, 2023

Copy link
Copy Markdown
Member Author

I rebased my PR and I revert the sys.setrecursionlimit().

The sys.setrecursionlimit() change was fun to do, play with the bare minimum. But it introduces a weird behavior (you can no longer call any Python function, not great.) Instead of I changed the support.infinite_recursion() minimum to 3, instead of 2. In practice, this function should be called at least with 50 (default is 100 with my change).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Sep 5, 2023

Copy link
Copy Markdown
Member Author

@brettcannon: Would you mind to review the updated PR?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@brettcannon
brettcannon self-requested a review September 6, 2023 00:22
@serhiy-storchaka
serhiy-storchaka self-requested a review September 6, 2023 10:36
serhiy-storchaka
serhiy-storchaka approved these changes Sep 6, 2023
View reviewed changes

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

You may also add simple docstrings for new functions. Just one line would be enough.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting core review labels Sep 6, 2023
@vstinner
Add docstrings to new test.support functions
d2a77b3
@vstinner

vstinner commented Sep 6, 2023

Copy link
Copy Markdown
Member Author

You may also add simple docstrings for new functions. Just one line would be enough.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Sep 6, 2023

Copy link
Copy Markdown
Member Author

You may also add simple docstrings for new functions. Just one line would be enough.

I added docstrings to the 2 newly added test.support functions.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner
vstinner merged commit 8ff1142 into python:main Sep 6, 2023
@vstinner
vstinner deleted the support_recursive branch September 6, 2023 15:34
@bedevere-bot bedevere-bot added awaiting core review and removed awaiting merge labels Sep 6, 2023
@bedevere-bot

bedevere-bot commented Sep 6, 2023

Copy link
Copy Markdown

There's a new commit after the PR has been approved.

@serhiy-storchaka: please review the changes made to this pull request.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot
bedevere-bot requested a review from serhiy-storchaka September 6, 2023 15:34
@vstinner vstinner added needs backport to 3.11 only security fixes needs backport to 3.12 only security fixes labels Sep 6, 2023
@miss-islington

miss-islington commented Sep 6, 2023

Copy link
Copy Markdown
Contributor

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@miss-islington

miss-islington commented Sep 6, 2023

Copy link
Copy Markdown
Contributor

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@miss-islington

miss-islington commented Sep 6, 2023

Copy link
Copy Markdown
Contributor

Sorry, @vstinner, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 8ff11425783806f8cb78e99f667546b1f7f3428e 3.11

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Sep 6, 2023
@vstinner @miss-islington
pythongh-108851: Fix tomllib recursion tests (pythonGH-108853)
949372b
* Add get_recursion_available() and get_recursion_depth() functions
  to the test.support module.
* Change infinite_recursion() default max_depth from 75 to 100.
* Fix test_tomllib recursion tests for WASI buildbots: reduce the
  recursion limit and compute the maximum nested array/dict depending
  on the current available recursion limit.
* test.pythoninfo logs sys.getrecursionlimit().
* Enhance test_sys tests on sys.getrecursionlimit()
  and sys.setrecursionlimit().
(cherry picked from commit 8ff1142)

Co-authored-by: Victor Stinner <[email protected]>
@miss-islington miss-islington assigned vstinner Sep 6, 2023
@bedevere-bot

bedevere-bot commented Sep 6, 2023

Copy link
Copy Markdown

GH-109012 is a backport of this pull request to the 3.12 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the needs backport to 3.12 only security fixes label Sep 6, 2023
vstinner added a commit to vstinner/cpython that referenced this pull request Sep 6, 2023
@vstinner
pythongh-108851: Fix tomllib recursion tests (python#108853)
e9bbcc5
* Add get_recursion_available() and get_recursion_depth() functions
  to the test.support module.
* Change infinite_recursion() default max_depth from 75 to 100.
* Fix test_tomllib recursion tests for WASI buildbots: reduce the
  recursion limit and compute the maximum nested array/dict depending
  on the current available recursion limit.
* test.pythoninfo logs sys.getrecursionlimit().
* Enhance test_sys tests on sys.getrecursionlimit()
  and sys.setrecursionlimit().

Backport notes:

* Set support.infinite_recursion() minimumum to 4 frames.
* test_support.test_get_recursion_depth() uses limit-2, apparently
  f-string counts for 2 frames in Python 3.11.
* test_sys.test_setrecursionlimit_to_depth() tests depth+2 instead of
  depth+1.

(cherry picked from commit 8ff1142)
vstinner added a commit to vstinner/cpython that referenced this pull request Sep 6, 2023
@vstinner
pythongh-108851: Fix tomllib recursion tests (python#108853)
44a6efd
* Add get_recursion_available() and get_recursion_depth() functions
  to the test.support module.
* Change infinite_recursion() default max_depth from 75 to 100.
* Fix test_tomllib recursion tests for WASI buildbots: reduce the
  recursion limit and compute the maximum nested array/dict depending
  on the current available recursion limit.
* test.pythoninfo logs sys.getrecursionlimit().
* Enhance test_sys tests on sys.getrecursionlimit()
  and sys.setrecursionlimit().

Backport notes:

* Set support.infinite_recursion() minimum to 4 frames.
* test_support.test_get_recursion_depth() uses limit-2, apparently
  f-string counts for 2 frames in Python 3.11.
* test_sys.test_setrecursionlimit_to_depth() tests depth+2 instead of
  depth+1.

(cherry picked from commit 8ff1142)
@bedevere-bot

bedevere-bot commented Sep 6, 2023

Copy link
Copy Markdown

GH-109013 is a backport of this pull request to the 3.11 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the needs backport to 3.11 only security fixes label Sep 6, 2023
@vstinner

vstinner commented Sep 6, 2023

Copy link
Copy Markdown
Member Author

Thanks for the review @brettcannon and @serhiy-storchaka. I will try to keep an eye on the WASI buildbots ;-)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner added a commit that referenced this pull request Sep 6, 2023
@vstinner
[3.11] gh-108851: Fix tomllib recursion tests (#108853) (#109013)
95eb984
gh-108851: Fix tomllib recursion tests (#108853)

* Add get_recursion_available() and get_recursion_depth() functions
  to the test.support module.
* Change infinite_recursion() default max_depth from 75 to 100.
* Fix test_tomllib recursion tests for WASI buildbots: reduce the
  recursion limit and compute the maximum nested array/dict depending
  on the current available recursion limit.
* test.pythoninfo logs sys.getrecursionlimit().
* Enhance test_sys tests on sys.getrecursionlimit()
  and sys.setrecursionlimit().

Backport notes:

* Set support.infinite_recursion() minimum to 4 frames.
* test_support.test_get_recursion_depth() uses limit-2, apparently
  f-string counts for 2 frames in Python 3.11.
* test_sys.test_setrecursionlimit_to_depth() tests depth+2 instead of
  depth+1.

(cherry picked from commit 8ff1142)
@brettcannon

brettcannon commented Sep 6, 2023

Copy link
Copy Markdown
Member

Thanks for the fix!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Sep 7, 2023

Copy link
Copy Markdown
Member Author

The fix works as expected: wasm32-wasi 3.11917 buildbot is back to green (success).

first successful build: https://buildbot.python.org/all/#/builders/1047/builds/908

By the way, I added sys.getrecursionlimit to test.pythoninfo. On WASI, the limit is lower, as you can now see on test.pythoninfo:

sys.getrecursionlimit: 600

On other platforms, the default is 1,000 frames:

$ make pythoninfo|grep getrecursionlimit
sys.getrecursionlimit: 1000
brettcannon reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Sep 8, 2023

Copy link
Copy Markdown
Member Author

Oh, I was sure that I removed #self.assertEqual(available, 2) commented code from test_support.py but it back from death in the merged commit! Well, I will remove it in a following change :-)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Yhg1s pushed a commit that referenced this pull request Sep 8, 2023
@miss-islington @vstinner
[3.12] gh-108851: Fix tomllib recursion tests (GH-108853) (#109012)
55d7e8f
gh-108851: Fix tomllib recursion tests (GH-108853)

* Add get_recursion_available() and get_recursion_depth() functions
  to the test.support module.
* Change infinite_recursion() default max_depth from 75 to 100.
* Fix test_tomllib recursion tests for WASI buildbots: reduce the
  recursion limit and compute the maximum nested array/dict depending
  on the current available recursion limit.
* test.pythoninfo logs sys.getrecursionlimit().
* Enhance test_sys tests on sys.getrecursionlimit()
  and sys.setrecursionlimit().
(cherry picked from commit 8ff1142)

Co-authored-by: Victor Stinner <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@encukou encukou Awaiting requested review from encukou encukou is a code owner
@hauntsaninja hauntsaninja Awaiting requested review from hauntsaninja hauntsaninja is a code owner
@brettcannon brettcannon Awaiting requested review from brettcannon
@serhiy-storchaka serhiy-storchaka Awaiting requested review from serhiy-storchaka

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@vstinner vstinner

Labels

awaiting core review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@vstinner @bedevere-bot @miss-islington @brettcannon @serhiy-storchaka
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.