GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-86275: improve Hypothesis configuration for CI and local runs by Zac-HD · Pull Request #104468 · python/cpython

Conversation

@Zac-HD

@Zac-HD Zac-HD commented May 14, 2023 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Per #22863 (comment), this PR configures Hypothesis to ignore unexpectedly slow tests. Relative to the default of failing, this reduces flakiness due to variable performance of CI machines.

The second commit adds caching to the CI test run, so that if a CI run finds a rare but real failure that input will be replayed in future CI runs (at least until the cache expires, or the test passes and Hypothesis discards the now-passing input). This doesn't come up often, but if it does will convert flaky tests into reliable failures.

Third, we can upload the database as an "artifact" from CI and configure local runs to use that as a pull-through cache - meaning that "reproducing a failure from CI" is just "run the tests locally". No impact on flakiness but it's a rather nice workflow! 1

cc @pganssle for review / checking that it works on your machine

  • Issue: Use hypothesis for testing the standard library, falling back to stubs #86275

Footnotes

  1. Unfortunately the 'dump a patch with explicit examples' workflow relies on libcst and our pytest plugin - if this would be particularly interesting we can talk about making it work with unittest and perhaps even ast. ↩

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@Zac-HD
Zac-HD requested review from ezio-melotti and hugovk as code owners May 14, 2023 06:58
@bedevere-bot bedevere-bot added the awaiting review label May 14, 2023
@bedevere-bot bedevere-bot mentioned this pull request May 14, 2023
Use hypothesis for testing the standard library, falling back to stubs #86275
Closed
@AlexWaygood AlexWaygood added the skip news label May 14, 2023
@AlexWaygood AlexWaygood closed this May 15, 2023
@AlexWaygood AlexWaygood reopened this May 15, 2023
AlexWaygood
AlexWaygood previously requested changes May 15, 2023
View reviewed changes

@AlexWaygood AlexWaygood left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CI on this PR is currently failing to start due to invalid workflow syntax:

https://github.com/python/cpython/actions/runs/4977097067

The workflow is not valid. .github/workflows/build.yml (Line: 378, Col: 11): A sequence was not expected

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot bedevere-bot added awaiting changes and removed awaiting review labels May 15, 2023
@bedevere-bot

bedevere-bot commented May 15, 2023

Copy link
Copy Markdown

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

hugovk
hugovk reviewed May 15, 2023
View reviewed changes
Comment thread .github/workflows/build.yml Resolved

Uh oh!

There was an error while loading. Please reload this page.

@Zac-HD Zac-HD mentioned this pull request May 16, 2023
Incorrect documentation for restore-keys actions/cache#1173
Closed
@Zac-HD

Zac-HD commented May 16, 2023

Copy link
Copy Markdown
Contributor Author

I have made the requested changes; please review again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot added awaiting change review and removed awaiting changes labels May 16, 2023
@bedevere-bot

bedevere-bot commented May 16, 2023

Copy link
Copy Markdown

Thanks for making the requested changes!

@AlexWaygood: please review the changes made to this pull request.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot
bedevere-bot requested a review from AlexWaygood May 16, 2023 02:20
@AlexWaygood
AlexWaygood dismissed their stale review May 16, 2023 07:13

It's valid syntax now

@bedevere-bot bedevere-bot added awaiting review and removed awaiting change review labels May 16, 2023
@Zac-HD
Zac-HD requested a review from hugovk May 17, 2023 20:31
hugovk
hugovk approved these changes May 20, 2023
View reviewed changes
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting review labels May 20, 2023
Zac-HD added 4 commits May 20, 2023 20:30
@Zac-HD
pythongh-86275: Don't fail PBTs for running slowly
f08da66
@Zac-HD
pythongh-86275: Cache Hypothesis' db in CI
59a2793
@Zac-HD
pythongh-86275: use pull-through cache
2d3b811
@Zac-HD
Make restore-keys a string
ccbb010
@Zac-HD
Zac-HD force-pushed the hypothesis-configuration branch from 6476ee0 to ccbb010 Compare May 20, 2023 19:30
@hugovk
hugovk merged commit 014dd30 into python:main May 21, 2023
@bedevere-bot bedevere-bot removed the awaiting merge label May 21, 2023
@Zac-HD
Zac-HD deleted the hypothesis-configuration branch May 21, 2023 13:02
webknjaz
webknjaz reviewed Jul 14, 2024
View reviewed changes
Comment thread .github/workflows/build.yml
key: hypothesis-database-${{ github.head_ref || github.run_id }}
restore-keys:
restore-keys: |
- hypothesis-database-

@webknjaz webknjaz Jul 14, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Zac-HD did you mean to keep that leading hyphen?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@webknjaz webknjaz Jul 14, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes it look up the '- hypothesis-database-' prefix, and I'm pretty sure you wanted 'hypothesis-database-':

Cache not found for input keys: hypothesis-database-9927179210, - hypothesis-database-

(https://github.com/python/cpython/actions/runs/9927179210/job/27421821417#step:21:24)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@webknjaz webknjaz Jul 14, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Zac-HD here's the fix: #121756

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Zac-HD reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@webknjaz webknjaz webknjaz left review comments
@hugovk hugovk hugovk approved these changes
@ezio-melotti ezio-melotti Awaiting requested review from ezio-melotti ezio-melotti is a code owner
@AlexWaygood AlexWaygood Awaiting requested review from AlexWaygood

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@Zac-HD @bedevere-bot @webknjaz @hugovk @AlexWaygood
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.