GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-82054: allow test runner to split test_asyncio by zitterbewegung · Pull Request #103859 · python/cpython

Conversation

@zitterbewegung

@zitterbewegung zitterbewegung commented Apr 25, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Summary:

This runs test_asyncio sub-tests in parallel using sharding by cinder. These two tests are typically the long-poles in runs because they are modules with a lot of further sub-tests run serially. By breaking out the sub-tests as independent modules we can run a lot more in parallel.

After porting we can see the direct impact is extremely large (15% increase in performance):

  • Without this change:
    • Running make test is 5 min 26 sec
  • With this change:
    • Running make test takes 3 min and 45 seconds

The drawbacks are that this implementation is hacky and due to the sorting of the tests it obscures when the asyncio tests occur and involves changing CPython test infrastructure but, the time saved it is worth it . It's not a complicated change and I think the win in productivity with the change above is significant.

  • Issue: unittest: execute tests in parallel #82054

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot bedevere-bot mentioned this pull request Apr 25, 2023
unittest: execute tests in parallel #82054
Closed
@bedevere-bot bedevere-bot added the awaiting review label Apr 25, 2023
@zitterbewegung
zitterbewegung marked this pull request as draft April 25, 2023 23:18
@zitterbewegung
zitterbewegung marked this pull request as ready for review April 25, 2023 23:24
@zitterbewegung zitterbewegung changed the title gh-82054: Porting execution of tests in parallel by sharding (note this only is for asyncio and compiler) gh-82054: Porting parallel test execution from cinder by sharding (note this only is for asyncio and compiler) Apr 25, 2023
@carljm carljm added the skip news label Apr 26, 2023
carljm
carljm requested changes Apr 26, 2023
View reviewed changes

@carljm carljm left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Just a couple comments.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/test/libregrtest/runtest.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Misc/NEWS.d/next/Tests/2023-04-25-23-56-04.gh-issue-gh-82054.3GgB66.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot added awaiting changes and removed awaiting review labels Apr 26, 2023
@bedevere-bot

bedevere-bot commented Apr 26, 2023

Copy link
Copy Markdown

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@zitterbewegung zitterbewegung changed the title gh-82054: Porting parallel test execution from cinder by sharding (note this only is for asyncio and compiler) gh-82054: Porting parallel test execution from cinder by sharding (note this only is for asyncio) Apr 26, 2023
@carljm carljm changed the title gh-82054: Porting parallel test execution from cinder by sharding (note this only is for asyncio) gh-82054: allow test runner to split test_asyncio for better parallel execution Apr 26, 2023
@carljm

carljm commented Apr 26, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

@zitterbewegung in order to make the case for landing this, it would be useful to update the PR description with some current data. Copying the description from the Cinder diff isn't really useful, since it's from a much older version. What would be helpful is to run make test with and without this change, and report any improvement in overall runtime.

It also may be that test_asyncio is no longer the longest pole, and we may need to apply the same treatment to some other modules (test_multiprocessing?) in order to see a significant benefit in overall runtime of make test.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@zitterbewegung

zitterbewegung commented Apr 26, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

@carljm I can see an overall benefit of 100 seconds from running cpython's baseline. I have attached both runs.
To give explicit timings before this change running make test is 5 min 26 sec and with the change it goes to 3 min and 46 seconds (100 seconds saved). Adding multiprocessing gives us a 5 second improvement.

This was executed on a AMD Ryzen Threadripper 3970X 32-Core Processor with 128GB of ram.

Patched_run_of_tests.txt
original_time_cpython.txt

carljm reacted with heart emoji
All reactions
  • ❤️ 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@carljm

carljm commented Apr 26, 2023

Copy link
Copy Markdown
Member

@zitterbewegung Awesome, those are great results! Can you directly update the PR description/summary to eliminate the text copied from the Cinder diff and instead just summarize these results?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@zitterbewegung

zitterbewegung commented Apr 26, 2023

Copy link
Copy Markdown
Contributor Author

@zitterbewegung Awesome, those are great results! Can you directly update the PR description/summary to eliminate the text copied from the Cinder diff and instead just summarize these results?

Updated summary with new results.

carljm reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@carljm

carljm commented Apr 26, 2023

Copy link
Copy Markdown
Member

@zitterbewegung Ok, one last thing I see is that make patchcheck is failing on the Azure Pipelines CI job, due to something it doesn't like about the whitespace in runtests.py. Pretty nitpicky, but we do want to keep the CI green; can you run make patchcheck locally and try to adjust the diff such that patchcheck is happy?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@zitterbewegung

zitterbewegung commented Apr 27, 2023

Copy link
Copy Markdown
Contributor Author

@zitterbewegung Awesome, those are great results! Can you directly update the PR description/summary to eliminate the text copied from the Cinder diff and instead just summarize these results?

Done.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@zitterbewegung zitterbewegung closed this Apr 27, 2023
@zitterbewegung
zitterbewegung force-pushed the main branch from 7848c65 to b701dce Compare April 27, 2023 14:32
@zitterbewegung

zitterbewegung commented Apr 27, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

@carljm I accidentally closed this PR while I was adding test_multiprocessing which saved 6 more seconds the new PR is #103927

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@zitterbewegung zitterbewegung changed the title gh-82054: allow test runner to split test_asyncio for better parallel execution gh-82054: allow test runner to split test_asyncio and test_multiprocessing for better parallel execution Apr 27, 2023
@zitterbewegung zitterbewegung changed the title gh-82054: allow test runner to split test_asyncio and test_multiprocessing for better parallel execution gh-82054: allow test runner to split test_asyncio Apr 27, 2023
@zitterbewegung

zitterbewegung commented Apr 27, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

@zitterbewegung Ok, one last thing I see is that make patchcheck is failing on the Azure Pipelines CI job, due to something it doesn't like about the whitespace in runtests.py. Pretty nitpicky, but we do want to keep the CI green; can you run make patchcheck locally and try to adjust the diff such that patchcheck is happy?

I ran make patch check after I did the changes

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@carljm carljm carljm requested changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting changes skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@zitterbewegung @bedevere-bot @carljm
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.