GeezerVison

I just copied the timeout-minutes: 60 from reusable-wasi which doesn't need 60 minutes either.

!buildbot emscripten

🤖 New build scheduled with the buildbot fleet by @hoodmane for commit 9da3ed3 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F146198%2Fmerge

The command will test the builders whose names match following regular expression: emscripten

The builders matched are:

• WASM Emscripten PR

So it took 19m 28s with no cache. But the cache is not working yet.

!buildbot emscripten

🤖 New build scheduled with the buildbot fleet by @hoodmane for commit d77a339 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F146198%2Fmerge

The command will test the builders whose names match following regular expression: emscripten

The builders matched are:

• WASM Emscripten PR

!buildbot Emscripten

🤖 New build scheduled with the buildbot fleet by @hoodmane for commit 4276a51 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F146198%2Fmerge

The command will test the builders whose names match following regular expression: Emscripten

The builders matched are:

• WASM Emscripten PR

!buildbot Emscripten

🤖 New build scheduled with the buildbot fleet by @hoodmane for commit 4276a51 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F146198%2Fmerge

The command will test the builders whose names match following regular expression: Emscripten

The builders matched are:

• WASM Emscripten PR

!buildbot emscripten

🤖 New build scheduled with the buildbot fleet by @hoodmane for commit 1520ee8 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F146198%2Fmerge

The command will test the builders whose names match following regular expression: emscripten

The builders matched are:

• WASM Emscripten PR

🤖 New build scheduled with the buildbot fleet by @hoodmane for commit 9d545d1 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F146198%2Fmerge

The command will test the builders whose names match following regular expression: emscripten

The builders matched are:

• WASM Emscripten PR

Alright, let's set the timeout to 25-30 minutes — this will help catch problems early. We just need to cover the cacheless run, plus have some margin for networking + minor flakiness. Otherwise, situations with something getting stuck for 40min+ would waste CI resources..

Well we have:

  1 job  with timeout-minutes: 30
  5 jobs with timeout-minutes: 5
  8 jobs with timeout-minutes: 10
 26 jobs with timeout-minutes: 60

Since most other CI jobs seem to have timeout-minutes: 60 I'd prefer to just keep the setting here as for the rest and then lowering these timeouts can be dealt with systematically as a separate issue.

Thanks @hoodmane for the PR, and @hugovk for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

Sorry, @hoodmane and @hugovk, I could not cleanly backport this to 3.14 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker c94048be025ad9d39cd9307db8f503039094df11 3.14

@hoodmane Please could you check the backport? Thanks!

GH-146331 is a backport of this pull request to the 3.14 branch.

@hugovk unfortunately the github action will be crashing on main because we were crossed up by a commit that broke Emscripten tests. Fixed by this:
#146330

FTR it's usually easier to set the right timeout as a part of the initial integration since the following runs will make use of cache and somebody handling this in the future would need to have privileges to wipe all the cache records to see the slow durations for each job.

Well that makes sense. Do you think 30 minutes makes sense here? That is about 50% higher than the worst run without cache. Or 40 minutes for 2x?

Thanks @hoodmane for the PR, and @hugovk for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

Sorry, @hoodmane and @hugovk, I could not cleanly backport this to 3.14 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker c94048be025ad9d39cd9307db8f503039094df11 3.14

@freakboy3742 freakboy3742 added the needs backport to 3.14 label

@hoodmane already created #146331 to add the CI to the 3.14 branch.

Well that makes sense. Do you think 30 minutes makes sense here? That is about 50% higher than the worst run without cache. Or 40 minutes for 2x?

@hoodmane my personal approach is having tight limits to get responsive cancelations. So I see how it performs when caching is not in use, then add a minute for each networking-related step/operation (things like git clone, various API access and similar), plus other known sources of flaky behaviors. And only increase that if I see that this doesn't account for all the flakiness. One other thing to take into account is that CPython's test runner re-runs the failing tests with verbosity so for the worst-case scenario, I'd add the duration of the rerun of the entire test suite. It's probably fine to add another 5% to the estimate computed like that.

Often, we can make an educated guess that the jobs won't be able to complete early: when git clone starts getting stuck for 10-15 minutes, there's a good chance the CI won't be able to finish. This is one of the indicators that GH (as a platform with multiple components) is having internal problems/outage (not necessarily announced in public), but it's usually a good idea to get those cancelations and wait until things calm down before restarting, which is why I perfect tight timeouts.

So yes, 30 minutes should be fine but if you feel more comfortable with 40, that's good too. I like documenting how I estimated the timeout in a YAML comment. So if you feel like sending a follow-up, plz do that too.

Opened #146378