GeezerVison is showing github.com — Annotate test cases which are timing sensitive · Issue #130363 · python/cpython
Annotate test cases which are timing sensitive #130363DescriptionFeature or enhancement
Proposal:
Many tests in the cpython test suite are sensitive to timing and assume an entirely unloaded system. For example, the TimerfdTests check for 1ms accuracy in expected durations. Some people (eg people building and testing python) may be running the test suite on build machines which are potentially heavily loaded so this sort of expectation isn't feasible.
It seems like a good solution to this would be to annotate the tests which are timing sensitive in some way, so they can be skipped easily.
For reference, we're manually patching python to skip tests which are failing under load but we're now having to rebase these patches on upgrades:
Has this already been discussed elsewhere?
No response given
Links to previous discussion of this feature:
No response
Linked PRs
Activityencukou commented on Feb 21, 2025 Sounds like it could be a new resource, see python -m test --help and test.support.requires. tomasr8 commented on Feb 22, 2025 I think it makes sense to hide the tests that clearly depend on timings (e.g. test_process_time) behind a test resource. Though for some of the other tests (e.g. test_input_no_stdout_fileno) it's not immediately obvious to me why they would fail, so perhaps instead of skipping them we could look into making them more reliable? rossburton commented on Feb 22, 2025 ContributorAuthor More actions Agreed - we've a short list of known flaky tests, and need to triage it to identify the ones which are genuinely sensitive vs those with subtle race conditions which should be fixed.
I will prototype a new resource and see how it works. tomasr8 commented on Feb 22, 2025 If it helps, to add a new resource, you just need to add it here:
|
ALL_RESOURCES = ('audio', 'curses', 'largefile', 'network', |
|
'decimal', 'cpu', 'subprocess', 'urlfetch', 'gui', 'walltime') |
And then you can annotate the tests with @support.requires_resource('your_new_resource') picnixz commented on Feb 22, 2025
Though for some of the other tests (e.g. test_input_no_stdout_fileno) it's not immediately obvious to me why they would fail, so perhaps instead of skipping them we could look into making them more reliable?
How about a flaky resources. At least, we know which tests are meant to be flaky and which are not. And for the CI, we could ask the CI to rerun flaky tests on failures so that contributors don't need to wait for a triager/core dev to rerun a failed workflow when needed.
I think it makes sense to hide the tests that clearly depend on timings (e.g. test_process_time) behind a test resource
I suggest adding stats resources which could cover all tests that are meant to gather timing and check that some condition is statistically verified. Other possible names that are more or less generic: timeprec, measures, experiments. I don't really have better names =/ tomasr8 commented on Feb 22, 2025 I like the flaky test idea! For timing-related tests, I don't have a good suggestion either, though stats seems a bit too generic to me (but I guess how generic we want this resource to be). timeprec is more self-evident, or maybe timing or timings? picnixz commented on Feb 22, 2025
I like the flaky test idea!
Let's open a separate issue for the flaky tests as it could be orthogonal to tests that require an idle system due to stats gathering.
I don't have a good suggestion either, though stats seems a bit too generic to me (but I guess how generic we want this resource to be). timeprec is more self-evident, or maybe timing or timings?
I suggested "stats" because I don't think we should have a test that doesn't check against a population that some hypothesis is verified. However, I'm not sure we have any test that would test something else than timings like this. Memory cannot really be flaky unless the kernel is doing some weird stuff and network can fail but generally it's a timeout issue, not a data issue itself. So timings looks quite promising. tomasr8 commented on Feb 22, 2025
Let's open a separate issue for the flaky tests as it could be orthogonal to tests that require an idle system due to stats gathering.
Issue's here: #130474
I suggested "stats" because I don't think we should have a test that doesn't check against a population that some hypothesis is verified. However, I'm not sure we have any test that would test something else than timings like this.
Yeah, I'm not sure if we assert anything else besides timings that is similarly stochastic.. maybe we can stick with some time-related name for now and adjust it if something else comes up? Reacted by Bénédikt Tran Reacted by Bénédikt Tran rossburton commented on Feb 24, 2025 ContributorAuthor More actions Posted a minimal PR: #130508. This just adds an "idle" resource. vstinner commented on Feb 24, 2025 I'm not comfortable with this issue. "Skip tests because they fail on our CI" sounds too specific to me. IMO a "skip list" for a specific CI is perfectly fine.
Your patches use the description:
This test hangs frequently when run on the Autobuilder. Disable it in testing until the cause can be determined.
Well, yes, the cause should be determined.
0001-test_storlines-skip-due-to-load-variability.patch: it would be interesting to investigate why this specific test "intermittently fails on the Yocto AB when a worker is under heavy load".
We already have the "walltime" resources for "slow tests". You might start by skipping this resource? rossburton commented on Feb 24, 2025 ContributorAuthor More actions Slow tests isn't a problem, it's tests that take 1s but have precise timing restrictions that are. rossburton commented on Sep 24, 2026 ContributorAuthor More actions Another potentially timing sensitive test case in test_asyncio, although it might just be buggy: #158097.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|