GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-108388: regrtest runs slowest tests first by vstinner · Pull Request #108391 · python/cpython

Conversation

@vstinner

@vstinner vstinner commented Aug 24, 2023 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

The Python test suite (regrtest) now runs the 20 slowest tests first and
then other tests, to better use all available CPUs when running tests in
parallel.

XXX: for tests, the PR also reverts the commit 7a6cc3e.

  • Issue: Split long running test suites up into smaller sub suites to reduce our regrtest long tail on multi-core systems. #108388

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
vstinner added 2 commits August 24, 2023 02:38
@vstinner
pythongh-108388: regrtest runs slowest tests first
852b6a2
The Python test suite (regrtest) now runs the 20 slowest tests first and
then other tests, to better use all available CPUs when running tests in
parallel.
@vstinner
Revert "test_peg_generator and test_freeze require cpu (python#108386)"
9d5c203
This reverts commit 7a6cc3e.
@vstinner
vstinner requested review from lysnikolaou and pablogsal as code owners August 24, 2023 00:39
@bedevere-bot bedevere-bot mentioned this pull request Aug 23, 2023
Split long running test suites up into smaller sub suites to reduce our regrtest long tail on multi-core systems. #108388
Closed
@bedevere-bot bedevere-bot added the awaiting core review label Aug 24, 2023
@vstinner
DEBUG: Log the 20 slowest tests
3345a17
@gpshead

gpshead commented Aug 24, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

I've thought about doing this in the past. It sort of defeats the purpose of the regrtest -r randomization. It also means people running the testsuite themselves don't get a nice feeling of general feedback on their change immediately because the test run just sits there using all 20 of their cores on long tests for a 30-60s before anything is available to run our 400 fast tests that collectively otherwise complete within the first 20 seconds.

Perhaps have this look at the -j number and only schedule the 0.7*j slowest tests up front.

On Github actions CI systems with basically only one core (two hyperthreads is basically 1.1-1.2 cores), the ordering of fast vs slow tests is not likely to make much difference.

buildbots sometimes have more cores though.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Aug 24, 2023

Copy link
Copy Markdown
Member Author

It we manage to split enough tests so no test takes longer than 1 minute, it should help a lot of the OS scheduler to better use all available CPUs, and so this change becomes less relevant, or even useless.

I close this PR for that. I wanted to give it a try to see how it looks.

But when I used it, it was bad that no test completed in the first 30 seconds, as if something was broken :-( So yeah, for users, it's unpleasant :-( We should investigate other approaches to make tests faster and keep them "responsible" (display progress to users ;-)).

gpshead reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner vstinner closed this Aug 24, 2023
@vstinner
vstinner deleted the regrtest_slowest_first branch August 24, 2023 16:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pablogsal pablogsal Awaiting requested review from pablogsal pablogsal is a code owner
@lysnikolaou lysnikolaou Awaiting requested review from lysnikolaou lysnikolaou is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting core review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@vstinner @gpshead @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.