GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-140189: Temporarily allow CI failures for iOS by savannahostrowski · Pull Request #142365 · python/cpython

Conversation

@savannahostrowski

@savannahostrowski savannahostrowski commented Dec 6, 2025 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Both Android and iOS tests are quite flaky right now. Based on the convo in Discord, it seems like allowing failures is the best option.

  • Issue: iOS CI and binary releases #140189

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@savannahostrowski
Temporarily allow failures iOS/Android
c3b159a
@savannahostrowski
savannahostrowski requested review from AA-Turner, ezio-melotti and hugovk as code owners December 6, 2025 21:49
@bedevere-app bedevere-app Bot added the awaiting core review label Dec 6, 2025
@savannahostrowski savannahostrowski added skip issue skip news labels Dec 6, 2025
@savannahostrowski

savannahostrowski commented Dec 6, 2025

Copy link
Copy Markdown
Member Author

I know the convo over Discord was specifically about Android but I've also been seeing this for iOS as well, so I've included it.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@savannahostrowski
Oops
72a14db
@savannahostrowski savannahostrowski changed the title Temporarily allow failures iOS/Android Temporarily allow CI failures for iOS/Android Dec 6, 2025
@mhsmith

mhsmith commented Dec 7, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

It looks like the Android issue was fixed by #142289, as there have been no Android failures on the main and 3.x branches since that was merged. So I don't think that needs to be an allowed failure now.

On the same branches, iOS has failed about 5 times in the last 2 days because of missing simulator images on the runners. @freakboy3742: did you have any possible solution to this?

savannahostrowski reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

hugovk
hugovk reviewed Dec 7, 2025
View reviewed changes
Comment thread .github/workflows/build.yml Resolved

Uh oh!

There was an error while loading. Please reload this page.

@savannahostrowski @hugovk
Update .github/workflows/build.yml
3f2b5a5
Co-authored-by: Hugo van Kemenade <[email protected]>
@hugovk hugovk changed the title Temporarily allow CI failures for iOS/Android Temporarily allow CI failures for iOS Dec 7, 2025
@picnixz picnixz mentioned this pull request Dec 7, 2025
gh-141732: Fix ExceptionGroup repr changing when original exception sequence is mutated #141736
Merged
@freakboy3742

freakboy3742 commented Dec 7, 2025

Copy link
Copy Markdown
Contributor

On the same branches, iOS has failed about 5 times in the last 2 days because of missing simulator images on the runners. @freakboy3742: did you have any possible solution to this?

The only solution I'm aware of at this point is to switch back to the macOS-14 runner; AFAIK the macOS-14 image isn't subject to the problem that is causing issues for macOS-15 and macOS-26 runners. The downside is that tests would be run on iOS 17, which is a little old at this point - that isn't ideal.

Is anyone aware of a way to get notifications when these failures occur? Or even search recent history for failures? I'm trying to get a sense of how frequent these issues are, if only to report to GitHub as part of the litany of macOS-15 image issues they've been working on.

The thing that is confusing is that we don't appear to see this mode of failure with BeeWare tests (or cibuildwheel tests either, to the best of my knowledge)...

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@freakboy3742

freakboy3742 commented Dec 7, 2025

Copy link
Copy Markdown
Contributor

Is anyone aware of a way to get notifications when these failures occur? Or even search recent history for failures?

Based on a manual search of failures of the Tests task on main, there's been 4 iOS test failures since Dec 29 (the last 100 CI runs), excluding 2 runs that were mass failures across multiple other platforms. 3 of those failures were in the last 24 hours. They're all failing for the same reason - a disk image that is meant to have multiple simulators installed is reporting having no simulators installed.

I do wonder if the increased frequency might be related to rolling out a new macOS-15 image version. Either way, I've posted an update on the GitHub Actions issue tracking the problem.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@hugovk

hugovk commented Dec 8, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

since Dec 29 (the last 100 CI runs)

I'm pretty sure we've had more than 100 runs in ~11 months :)

Or even search recent history for failures?

Here's a script that wraps the gh CLI:

Details
#!/usr/bin/env python3
"""Check recent CPython build.yml workflow runs for iOS build failures.

uv run check-ios-failures.py
"""
# /// script
# requires-python = ">=3.10"
# dependencies = ["prettytable", "rich"]
# ///

import argparse
import datetime as dt
import json
import shlex
import subprocess

from prettytable import PrettyTable, TableStyle
from rich.progress import track


def osc8_link(url: str, text: str) -> str:
    return f"\033]8;;{url}\033\\{text}\033]8;;\033\\"


def run_gh(cmd: str) -> str:
    result = subprocess.run(
        ["gh", *shlex.split(cmd)],
        capture_output=True,
        text=True,
        check=True,
    )
    return result.stdout


def get_recent_runs(repo: str, days_back: float) -> list[dict]:
    cutoff = dt.datetime.now(dt.timezone.utc) - dt.timedelta(days=days_back)
    cutoff_str = cutoff.strftime("%Y-%m-%dT%H:%M:%SZ")
    limit = int(150 * days_back)

    output = run_gh(
        f"run list --repo {repo} --workflow build.yml "
        f"--limit {limit} --json databaseId,conclusion,createdAt,displayTitle"
    )
    runs = json.loads(output)
    if len(runs) >= limit:
        print(f"Warning: fetched {limit} runs, results may be incomplete")
    return [r for r in runs if r["createdAt"] >= cutoff_str]


def get_job_failures(repo: str, run_id: int) -> tuple[bool, bool]:
    """Return (ios_failed, other_failed)."""
    output = run_gh(f"run view {run_id} --repo {repo} --json jobs")
    data = json.loads(output)
    ios_failed = False
    other_failed = False
    for job in data.get("jobs", []):
        name = job.get("name", "")
        if name == "All required checks pass":
            continue
        if job.get("conclusion") == "failure":
            if "ios" in name.lower():
                ios_failed = True
            else:
                other_failed = True
    return ios_failed, other_failed


class Formatter(
    argparse.ArgumentDefaultsHelpFormatter,
    argparse.RawDescriptionHelpFormatter,
):
    pass


def main() -> None:
    parser = argparse.ArgumentParser(
        description="Check recent CPython build.yml runs for iOS build failures",
        formatter_class=Formatter,
    )
    parser.add_argument(
        "-d", "--days", type=float, default=2, help="number of days to look back"
    )
    parser.add_argument(
        "-r", "--repo", default="python/cpython", help="GitHub repository"
    )
    parser.add_argument(
        "-m", "--markdown", action="store_true", help="output tables in markdown format"
    )
    args = parser.parse_args()
    style = TableStyle.MARKDOWN if args.markdown else TableStyle.SINGLE_BORDER

    print(f"Fetching build.yml runs from the last {args.days} days...")
    runs = get_recent_runs(args.repo, args.days)

    total = len(runs)
    failures = [r for r in runs if r["conclusion"] == "failure"]
    num_failures = len(failures)

    print(f"\nTotal runs: {total}")
    print(f"Failures: {num_failures}")

    ios_failure_runs = []
    for run in track(failures, description="Checking failed runs..."):
        run_id = run["databaseId"]
        ios_failed, other_failed = get_job_failures(args.repo, run_id)
        if ios_failed:
            full_title = run["displayTitle"]
            title = full_title[:30] + "…" if len(full_title) > 30 else full_title
            ios_failure_runs.append((run_id, run["createdAt"], title, other_failed))

    ios_only = sum(1 for *_, other in ios_failure_runs if not other)
    ios_plus_other = sum(1 for *_, other in ios_failure_runs if other)

    # Summary table
    table = PrettyTable()
    table.set_style(style)
    table.field_names = ["Metric", "Count"]
    table.align["Metric"] = "l"
    table.align["Count"] = "r"
    table.add_row(["Total runs", total])
    table.add_row(["Failed runs", num_failures])
    table.add_row(["iOS + other failures", ios_plus_other])
    table.add_row(["iOS only failures", ios_only])
    print(f"\nSUMMARY (last {args.days} days)")
    print(table)

    if ios_failure_runs:
        table = PrettyTable()
        table.set_style(style)
        table.field_names = ["Run ID", "Title", "Created", "Other failures"]
        table.align["Title"] = "l"
        table.align["Other failures"] = "l"

        for run_id, created_at, title, other_failed in ios_failure_runs:
            url = f"https://github.com/{args.repo}/actions/runs/{run_id}"
            link = f"[{run_id}]({url})" if args.markdown else osc8_link(url, str(run_id))
            table.add_row([link, title, created_at, "yes" if other_failed else "iOS only"])
        print("\nRuns with iOS failures:")
        print(table)


if __name__ == "__main__":
    main()

Here's 7 days' results:

Metric Count
Total runs 778
Failed runs 173
iOS + other failures 32
iOS only failures 19

Runs with iOS failures:

Run ID Title Created Other failures
20016982627 gh-142207: remove assertions i… 2025-12-08T04:45:09Z iOS only
20010300639 [3.14] gh-141732: Fix `Excepti… 2025-12-07T21:04:17Z yes
20010297873 gh-141732: Fix `ExceptionGroup… 2025-12-07T21:04:08Z iOS only
20009409862 gh-141732: Fix `ExceptionGroup… 2025-12-07T19:46:24Z iOS only
20006669592 gh-138122: Don't sample partia… 2025-12-07T15:53:51Z iOS only
19997546915 gh-68443: Replace debug level-… 2025-12-07T02:13:47Z yes
19994600519 gh-141388: Improve support for… 2025-12-06T21:36:34Z yes
19994299998 GH-142363: Contrast and gradie… 2025-12-06T21:09:42Z iOS only
19994299461 gh-142236: Fix incorrect keywo… 2025-12-06T21:09:39Z iOS only
19991348547 GH-64532: Include parent's req… 2025-12-06T16:43:06Z iOS only
19990985400 gh-142349: Implement PEP 810 2025-12-06T16:10:32Z yes
19978475806 [3.14] Introduce `build-python… 2025-12-05T23:04:53Z iOS only
19976984734 gh-142083: Clarify documentati… 2025-12-05T21:50:42Z yes
19976791392 gh-142083: Clarify documentati… 2025-12-05T21:41:28Z yes
19970993664 gh-81554: Add add_reader suppo… 2025-12-05T17:35:59Z yes
19968999614 [3.14] gh-140482: Avoid changi… 2025-12-05T16:18:44Z iOS only
19968754175 Add explanation comments for t… 2025-12-05T16:09:23Z yes
19966117964 gh-142276: Watch attribute loa… 2025-12-05T14:32:23Z iOS only
19965535904 gh-131372: Include LDVERSION a… 2025-12-05T14:10:53Z iOS only
19956754205 [3.14] gh-142214: Fix two regr… 2025-12-05T08:08:19Z yes
19951049962 spam 2025-12-05T02:50:52Z yes
19947235648 GH-139862: Remove color from… 2025-12-04T23:24:34Z yes
19942618634 Firmament2 rollup: tokenizer/A… 2025-12-04T20:16:07Z yes
19942260019 Firmament2 rollup: tokenizer/A… 2025-12-04T20:03:02Z yes
19942007907 Firmament2 rollup: tokenizer/A… 2025-12-04T19:53:03Z yes
19941519174 Firmament2 rollup: tokenizer/A… 2025-12-04T19:34:09Z yes
19941260940 Firmament2 rollup: tokenizer/A… 2025-12-04T19:24:33Z yes
19941135626 Firmament2 rollup: tokenizer/A… 2025-12-04T19:19:59Z yes
19940690692 Firmament2 rollup: tokenizer/A… 2025-12-04T19:03:52Z yes
19940228017 Firmament2 rollup: tokenizer/A… 2025-12-04T18:47:26Z yes
19937252907 gh-141794: Reduce size of comp… 2025-12-04T17:04:21Z yes
19909113149 gh-139871: Optimize bytearray … 2025-12-03T21:16:48Z iOS only
19906338327 gh-141976: Check stack bounds … 2025-12-03T19:32:28Z iOS only
19875900577 gh-142029: Raise `ModuleNotFou… 2025-12-02T22:40:54Z yes
19863859746 gh-129483: Make `TestLocalTime… 2025-12-02T15:23:59Z iOS only
19862295675 gh-140677 Add heatmap visualiz… 2025-12-02T14:35:18Z yes
19858003358 [3.14] Revert "gh-119452: Fix … 2025-12-02T12:06:28Z iOS only
19855055657 gh-142186: make PY_UNWIND avai… 2025-12-02T10:16:28Z yes
19852792972 Document None for timeout argu… 2025-12-02T08:57:12Z iOS only
19850261887 GH-142050: Jit stencils on Win… 2025-12-02T07:09:10Z iOS only
19849646399 Gh-142174: Explicitly disallow… 2025-12-02T06:38:33Z yes
19849594509 Gh-142174: Explicitly disallow… 2025-12-02T06:35:50Z yes
19849291625 gh-140947: fix contextvars han… 2025-12-02T06:20:51Z yes
19846833554 gh-81554: Add add_reader suppo… 2025-12-02T04:12:15Z yes
19841987508 gh-140677 Add heatmap visualiz… 2025-12-02T00:14:51Z yes
19838324252 gh-140677 Add heatmap visualiz… 2025-12-01T21:35:35Z yes
19836858095 gh-142048: Fix quadratically i… 2025-12-01T20:41:51Z yes
19836186393 gh-139914: Fix another stack a… 2025-12-01T20:16:53Z iOS only
19834719150 gh-54872: Remove default OPT=-… 2025-12-01T19:22:37Z yes
19831639606 gh-81554: Add add_reader suppo… 2025-12-01T17:29:48Z yes
19831279870 gh-142155: Fix infinite recurs… 2025-12-01T17:17:20Z iOS only
savannahostrowski reacted with thumbs up emoji freakboy3742 reacted with heart emoji
All reactions
  • 👍 1 reaction
  • ❤️ 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

colesbury
colesbury approved these changes Dec 8, 2025
View reviewed changes
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Dec 8, 2025
@colesbury
colesbury merged commit 9d39c02 into python:main Dec 8, 2025
48 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the awaiting merge label Dec 8, 2025
@freakboy3742

freakboy3742 commented Dec 9, 2025

Copy link
Copy Markdown
Contributor

since Dec 29 (the last 100 CI runs)

I'm pretty sure we've had more than 100 runs in ~11 months :)

🤦 Sorry - November 29.

Here's a script that wraps the gh CLI:

Thanks for that. The "iOS + other" failures have a lot of "real" problems; but it looks like it shakes out to a 2-4% failure rate.

On that basis, it feels like it might be worth trying the macos-14 runner fix to see if that makes things better.

hugovk reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@freakboy3742 freakboy3742 mentioned this pull request Dec 9, 2025
gh-140189: Use macos-14 runner for iOS CI tests #142444
Merged
@hugovk hugovk changed the title Temporarily allow CI failures for iOS gh-140189: Temporarily allow CI failures for iOS Dec 13, 2025
@hugovk hugovk removed the skip issue label Dec 13, 2025
@bedevere-app bedevere-app Bot mentioned this pull request Dec 13, 2025
iOS CI and binary releases #140189
Closed
3 of 4 tasks
hugovk added a commit to hugovk/cpython that referenced this pull request Dec 13, 2025
@hugovk
Revert "Temporarily allow CI failures for iOS (python#142365)"
ba11262
This reverts commit 9d39c02.
hugovk added a commit that referenced this pull request Dec 14, 2025
@hugovk
gh-140189: Revert "Temporarily allow CI failures for iOS (#142365)" (#…
04a9009
…142660)
fatelei pushed a commit to fatelei/cpython that referenced this pull request Dec 16, 2025
@hugovk @fatelei
pythongh-140189: Revert "Temporarily allow CI failures for iOS (pytho…
66632c6
…n#142365)" (python#142660)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hugovk hugovk hugovk left review comments
@colesbury colesbury colesbury approved these changes
@ezio-melotti ezio-melotti Awaiting requested review from ezio-melotti ezio-melotti is a code owner
@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@savannahostrowski @mhsmith @freakboy3742 @hugovk @colesbury
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.