GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-105699: Add Some Stress Tests for Subinterpreter Creation by ericsnowcurrently · Pull Request #106966 · python/cpython

Conversation

@ericsnowcurrently

@ericsnowcurrently ericsnowcurrently commented Jul 21, 2023

Copy link
Copy Markdown
Member

The tests would have caught the crashes from gh-105699.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@ericsnowcurrently
Add some stress tests.
0681292
@ericsnowcurrently ericsnowcurrently added skip news needs backport to 3.12 only security fixes labels Jul 21, 2023
@bedevere-bot bedevere-bot mentioned this pull request Jul 21, 2023
Crash During Subinterpreter Finalization #105699
Closed
@bedevere-bot bedevere-bot added awaiting core review tests Tests in the Lib/test dir labels Jul 21, 2023
corona10
corona10 reviewed Jul 21, 2023
View reviewed changes
Comment thread Lib/test/test_interpreters.py
interp = interpreters.create()
alive.append(interp)

def test_create_many_threaded(self):

@corona10 corona10 Jul 21, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
def test_create_many_threaded(self):
@requires_resource('cpu')
def test_create_many_threaded(self):

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ericsnowcurrently reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

@ericsnowcurrently ericsnowcurrently Jul 21, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, what is the effect of this decorator? Also, what is it telling readers?

I wouldn't consider this test to be particularly CPU-intensive. We do spin up a couple hundred threads but not so many that this test should take more than a few seconds on a resource-starved host. Likewise, this shouldn't impose a significant memory burden. (Each interpreter uses ~3kB.)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@corona10 corona10 Jul 22, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cpython/Lib/test/libregrtest/cmdline.py

Line 108 in cdeb1a6

cpu - Used for certain CPU-heavy tests.

AFAIK, Conventionally, most CPU-intensive tests consume a few seconds in high-performance computers, we added the flag for low-performance CPUs to skip this test.

I thought that this test look quite CPU-intensive test, so I suggested adding it.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
corona10
corona10 reviewed Jul 21, 2023
View reviewed changes
Comment thread Lib/test/test_interpreters.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@corona10

corona10 commented Jul 21, 2023

Copy link
Copy Markdown
Member

Left some minor suggestions.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ambv
ambv merged commit adda43d into python:main Jul 22, 2023
@miss-islington

miss-islington commented Jul 22, 2023

Copy link
Copy Markdown
Contributor

Thanks @ericsnowcurrently for the PR, and @ambv for merging it 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the awaiting core review label Jul 22, 2023
@bedevere-bot

bedevere-bot commented Jul 22, 2023

Copy link
Copy Markdown

GH-107012 is a backport of this pull request to the 3.12 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jul 22, 2023
@ericsnowcurrently @miss-islington
pythongh-105699: Add some stress tests for subinterpreter creation (p…
0731803
…ythonGH-106966)

(cherry picked from commit adda43d)

Co-authored-by: Eric Snow <[email protected]>
@bedevere-bot bedevere-bot removed the needs backport to 3.12 only security fixes label Jul 22, 2023
@ericsnowcurrently
ericsnowcurrently deleted the interpreter-stress-tests branch July 24, 2023 15:08
ericsnowcurrently added a commit that referenced this pull request Jul 25, 2023
@miss-islington @ericsnowcurrently
[3.12] gh-105699: Add some stress tests for subinterpreter creation (G…
ca42d67
…H-106966) (gh-107012)

gh-105699: Add some stress tests for subinterpreter creation (GH-106966)
(cherry picked from commit adda43d)

Co-authored-by: Eric Snow <[email protected]>
@bedevere-bot

bedevere-bot commented Jul 25, 2023

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows10 3.12 has failed when building commit ca42d67.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1159/builds/222) and take a look at the build logs.
  4. Check if the failure is related to this commit (ca42d67) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1159/builds/222

Failed tests:

  • test_interpreters

Summary of the results of the build (if available):

==

Click to see traceback logs
remote: Enumerating objects: 9, done.        
remote: Counting objects:  11% (1/9)        
remote: Counting objects:  22% (2/9)        
remote: Counting objects:  33% (3/9)        
remote: Counting objects:  44% (4/9)        
remote: Counting objects:  55% (5/9)        
remote: Counting objects:  66% (6/9)        
remote: Counting objects:  77% (7/9)        
remote: Counting objects:  88% (8/9)        
remote: Counting objects: 100% (9/9)        
remote: Counting objects: 100% (9/9), done.        
remote: Compressing objects:  20% (1/5)        
remote: Compressing objects:  40% (2/5)        
remote: Compressing objects:  60% (3/5)        
remote: Compressing objects:  80% (4/5)        
remote: Compressing objects: 100% (5/5)        
remote: Compressing objects: 100% (5/5), done.        
remote: Total 5 (delta 4), reused 0 (delta 0), pack-reused 0        
From https://github.com/python/cpython
 * branch            3.12       -> FETCH_HEAD
Note: checking out 'ca42d6720850967698d6e298e37a4225f0c67d8e'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at ca42d67... [3.12] gh-105699: Add some stress tests for subinterpreter creation (GH-106966) (gh-107012)
Switched to and reset branch '3.12'

Could Not Find D:\buildarea\3.12.bolen-windows10\build\Lib\*.pyc
The system cannot find the file specified.
Could Not Find D:\buildarea\3.12.bolen-windows10\build\PCbuild\python*.zip

Debug memory block at address p=000001D24C96FAA0: API '�'
    1731634056723955712 bytes originally requested
    The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd):
        at p-7: 0x8b *** OUCH
        at p-6: 0x03 *** OUCH
        at p-5: 0x00 *** OUCH
        at p-4: 0xfd
        at p-3: 0xfd
        at p-2: 0xfd
        at p-1: 0xfd
    Because memory is corrupted at the start, the count of bytes requested
       may be bogus, and checking the trailing pad bytes may segfault.
    The 8 pad bytes at tail=180801D24C96FAA0 are Windows fatal exception: access violation

Current thread 0x0000263c (most recent call first):
  File "<frozen importlib._bootstrap_external>", line 1669 in _fill_cache
  File "<frozen importlib._bootstrap_external>", line 1601 in find_spec
  File "<frozen importlib._bootstrap_external>", line 1498 in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1524 in find_spec
  File "<frozen importlib._bootstrap>", line 1168 in _find_spec
  File "<frozen importlib._bootstrap>", line 1228 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1266 in _find_and_load
  File "<frozen importlib._bootstrap>", line 400 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1321 in _handle_fromlist
  File "D:\buildarea\3.12.bolen-windows10\build\Lib\encodings\__init__.py", line 33 in <module>
  File "<frozen importlib._bootstrap>", line 400 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 994 in exec_module
  File "<frozen importlib._bootstrap>", line 841 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1237 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1266 in _find_and_load

Cannot open file 'D:\buildarea\3.12.bolen-windows10\build\test-results.xml' for upload

Could Not Find D:\buildarea\3.12.bolen-windows10\build\PCbuild\python*.zip
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@corona10 corona10 corona10 left review comments

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news tests Tests in the Lib/test dir

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@ericsnowcurrently @corona10 @miss-islington @bedevere-bot @ambv
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.