GeezerVison

Problem

pdm-compatibility.yml is the flakiest workflow right now. From 2026-09-28 to 2026-10-05 it had 29 failed runs (ci.yml had almost no flakes in the same window: 133 runs, and nearly every red one was the PR's own bug). For the 25 failures in the native job, the log shows one random FAIL cell in an otherwise green 33–47-cell matrix, on PRs that don't touch PDM (gem, npm, yarn, uv fixes). The cell is different every time:

Example runs: 37309092656 (2.17.3 space-unicode vendored), 37308344193 (2.29.2 mac extras hosted), 37306321055, 37279240870 (4 legs at once), 37225409452. Each red run costs a re-run of the ~19-leg matrix and blocks the PR's checks.

Root cause

Every failing check judges an operation that goes to production PyPI or the public patch API: a CLI scan, rollback or vex, or a pdm sync/install. Run retries a command once, after 5 s, and only when it exits non-zero. On an exhausted patch API fetch (patch_details_failed / api_batch_failed, a 503, or a 429 its retry loop gave up on) the CLI usually exits zero and reports the failure in its JSON envelope. The cell then just fails a later check. The Poetry harness had exactly this pattern and was fixed in #596. The PDM harness never got that fix. The job log also never says why a cell failed (the notes live only in the artifact), which is why these were never diagnosed.

Fix

Port backtest-poetry.py's evidence-gated, case-level retry to scripts/backtest-pdm.py:

• record_check(..., operation=): each network-bound check names the command it judges. A failed check records transport evidence only from that command: its output if it failed, otherwise only the CLI's JSON error records (error, failed/skipped events, api_batch_failed / patch_details_failed). The patterns are the same TRANSPORT_FAILURE set Poetry uses, plus httpx errors (PDM ≥ 2 uses httpx). Lines saying "retrying" (a retry that recovered) are ignored.
• retry_transport: re-runs a case from a fresh directory, at most 3 attempts, with 10 s/20 s backoff, only when every failed check has transport evidence. An exception is retried only if its text is a transport diagnostic and no check had already failed functionally. A functional failure is never retried.
• Failed attempts' logs go under native-pdm/attempts/<version>-<shape>-<mode>/<n>/, now uploaded. The final row lists them in transportRetries.
• A FAIL cell now prints each failed check's recorded note in the job log, so the next real failure names its cause without downloading the artifact.

The existing one-shot Run(retry=True) stays as is. Nothing is skipped, ignored or loosened: a persistent outage still turns the cell red after 3 attempts.

Proof

• New PdmTransportRetryTests (7 tests, offline) cover:

  • a zero-exit CLI patch_details_failed 503 is retried from a fresh case, and the evidence is kept and saved;
  • non-zero CLI request error / exhausted 429 / httpx DNS errors are retried;
  • refusals, recovered "retrying" warnings, transport text outside error records, and checks with no operation are not retried;
  • one unexplained failed check blocks the retry;
  • a transport exception after a functional failure is not retried;
  • a persistent failure stays red after 3 attempts.

  python3 -B -m unittest discover -s scripts/tests → 237 tests OK.

• End to end with the real harness and CLI (--versions 2.17.3 --shapes direct --modes vendored agent). The sandbox blocks patches-api.socket.dev, so the real CLI returns "error": "Network error: error sending request for url (https://patches-api.socket.dev/patch/batch) …". Both cases were classified as transport, re-run from fresh case dirs (attempts 1/3, 2/3 kept under attempts/), and stayed FAIL after the third attempt, with appliedExactlyOne: {"applied": 0, … "status": "error"} printed in the log.

• ruff check --select F shows the same single pre-existing warning as main. The workflow YAML parses.

Where tests run

Nothing removed or moved. Every cell still runs in pdm-compatibility.yml on the same triggers. The only workflow change adds native-pdm/attempts/** to the results artifact. Required-check names are unchanged.

Also in this PR: 80538db ports #851's test-only fix for two commands::vex_consumed tests that main has failed since #605. The port is needed because those tests make coverage/test/test-release red on every PR. It becomes a no-op once #851 lands. 6b0302a addresses Bugbot's finding: the rollback and VEX checks are now judged by the commands they check.

🤖 Generated with Claude Code

https://claude.ai/code/session_01N8YeCUhdg2tKdqyyY7z3sV

Generated by Claude Code

Generated by Claude Code