GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-137242: Add a `--no-randomize` option, and use it in Android CI by mhsmith · Pull Request #138303 · python/cpython

Conversation

@mhsmith

@mhsmith mhsmith commented Aug 31, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Original PR title: "Change CI arguments: remove --randomize, add --fail-rerun"

--randomize can discover real problems, but they're often ordering dependencies between tests, which are difficult to diagnose, and usually have nothing to do with the PR on which they occur.

I assume this was the reason for removing --fail-rerun from the --fast-ci and --slow-ci arguments in #110849. However, this renders --randomize almost useless, because the failing test will usually pass on the rerun in a fresh process, and nobody will ever know that there was a failure.

Also, --rerun without --fail-rerun means that a test which ALWAYS fails the first time and passes the second time will still be treated as a pass. This seems unsafe.

So I propose removing --randomize and restoring --fail-rerun.

This will also allow iOS and Android to switch to --fast-ci on GitHub Actions and --slow-ci on the buildbots. They were previously unable to do this because of the frequent failures caused by --randomize, which were not hidden on the rerun because these platforms use --single-process mode.

  • Issue: Android CI and binary releases #137242

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@mhsmith
Change CI arguments: remove --randomize, add --fail-rerun
e96c92a
@mhsmith
mhsmith requested a review from vstinner August 31, 2025 18:17
@mhsmith mhsmith added the skip issue label Aug 31, 2025
@mhsmith
mhsmith requested a review from freakboy3742 as a code owner August 31, 2025 18:17
@mhsmith mhsmith added skip news needs backport to 3.13 only security fixes needs backport to 3.14 bugs and security fixes labels Aug 31, 2025
@bedevere-app bedevere-app Bot added the awaiting review label Aug 31, 2025
@mhsmith
mhsmith removed the request for review from freakboy3742 August 31, 2025 18:17
@mhsmith
Remove outdated comment
f69095c
@picnixz picnixz added the infra CI, GitHub Actions, buildbots, Dependabot, etc. label Aug 31, 2025
@mhsmith mhsmith changed the title Change CI arguments: remove --randomize, add --fail-rerun gh-137242: Change CI arguments: remove --randomize, add --fail-rerun Aug 31, 2025
@mhsmith mhsmith removed the skip issue label Aug 31, 2025
@bedevere-app bedevere-app Bot mentioned this pull request Aug 31, 2025
Android CI and binary releases #137242
Closed
@mhsmith

mhsmith commented Aug 31, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

Also, --rerun without --fail-rerun means that a test which ALWAYS fails the first time and passes the second time will still be treated as a pass.

It looks like some tests are already doing that on some runners. I'll switch this PR back to draft until this has been resolved.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@mhsmith
mhsmith marked this pull request as draft August 31, 2025 18:57
@bedevere-app bedevere-app Bot removed the awaiting review label Aug 31, 2025
vstinner
vstinner reviewed Sep 3, 2025
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dislike changing CI defaults.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Lib/test/libregrtest/cmdline.py
# -j0 --fail-env-changed --rerun --fail-rerun --slowest --verbose3
if ns.use_mp is None:
ns.use_mp = 0
ns.randomize = True

@vstinner vstinner Sep 3, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to keep randomization for --fast-ci and --slow-ci options.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@vstinner vstinner Sep 3, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer adding an option to disable randomization, option which could be used with --fast-ci / --slow-ci.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Lib/test/libregrtest/cmdline.py Outdated
ns.fail_env_changed = True
if ns.python is None:
ns.rerun = True
ns.fail_rerun = True

@vstinner vstinner Sep 3, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can already pass --fail-rerun. Making it the default would make all CIs way more strict, I dislike this idea. When I tried a few years ago, I discovered tons of flaky tests and it was a pain to fix all of them.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@mhsmith mhsmith changed the title gh-137242: Change CI arguments: remove --randomize, add --fail-rerun gh-137242: Add a --no-randomize option, and use it in Android CI Sep 8, 2025
@mhsmith
Revert changes to default args; add a --no-randomize option instead
b64cace
vstinner
vstinner reviewed Sep 8, 2025
View reviewed changes
Comment thread Lib/test/libregrtest/cmdline.py
self.list_tests = False
self.single = False
self.randomize = False
self.randomize = None

@vstinner vstinner Sep 8, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer avoiding None for randomize and keep a boolean instead. I wrote #138649 which is a clone of this PR with my additional change avoiding None. What do you think?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@mhsmith mhsmith Sep 8, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense; I'll close this PR in favor of that one.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
vstinner reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@mhsmith mhsmith closed this Sep 8, 2025
@mhsmith mhsmith mentioned this pull request Sep 8, 2025
gh-137242: Add a --no-randomize option, and use it in Android CI #138649
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner vstinner left review comments

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

infra CI, GitHub Actions, buildbots, Dependabot, etc. needs backport to 3.13 only security fixes needs backport to 3.14 bugs and security fixes skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@mhsmith @vstinner @picnixz
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.