GeezerVison

Problem

Pipenv compatibility is the flakiest PR workflow by re-run count. Between 2026-09-18 and 2026-10-01, 10 runs went red on attempt 1 and green on re-run of the same SHA, covering 15 failed matrix legs (9 on the 8-release ubuntu direct leg, 3 on the macOS leg, 3 on the ubuntu crlf/marker-excluded/extras/category leg). Each re-run costs a full matrix leg (5–7 min each, on top of the cargo build) plus a human noticing and re-running it.

Examples of red-then-green runs:

• https://github.com/SocketDev/socket-patch/actions/runs/36699362953 (3 legs red on attempt 1)
• https://github.com/SocketDev/socket-patch/actions/runs/36549335640
• https://github.com/SocketDev/socket-patch/actions/runs/36472053681
• https://github.com/SocketDev/socket-patch/actions/runs/36428895598
• https://github.com/SocketDev/socket-patch/actions/runs/35632449754 (3 legs red on attempt 1)

Clustering the attempt-1 logs of all 15 legs:

Example (job 109834889963):

| 2023.12.1 | direct | hosted | in-dir | ERROR | ...urllib3-1.26.18-py2.py3-none-any.whl (Caused by ResponseError('too many 503 error responses')) ERROR: Couldn't install package ...

Root cause (confirmed cluster)

scripts/backtest-pipenv.py installs real wheels, both from PyPI and from the public patch service's hosted wheel URLs. Unlike the Bun (retry_network_cell), PDM (NETWORK_RE + Run(retry=True)) and vlt (run_with_retries) harnesses, it has no transport retry. One 503 burst or DNS blip fails the case, so the leg exits 1.

The applied=0 cluster looks like the same outage reaching the CLI's hosted fetch, but the summary table hides the CLI's output. The case logs are not uploaded as an artifact; only summary.* is. The retry recorded here (transportRetries) will show in summary.json whether those rows were transport failures. The other two clusters are left for a follow-up.

Fix

retry_transport wraps each case in the scheduler. When a case raises or fails its checks and its error text or one of its *.log* files matches a narrow transport signature, it re-runs the case from a fresh case directory, with at most 3 attempts and a 10s/20s backoff. backtest_case already rmtrees the case dir, and the shared native lock is only cached after pipenv lock succeeds. Signatures: pip's too many 5xx error responses / Max retries exceeded with url / NewConnectionError / ConnectTimeoutError / ReadTimeoutError / DNS (Temporary failure in name resolution, macOS nodename nor servname provided) / connection reset, plus the CLI's own error sending request for url ( / API request failed with status 5xx (the same regex the Bun and vlt harnesses use).

• Functional failures (hash mismatch, resolution failures, wrong bytes) are never retried, and they still fail on the first attempt.
• A transport failure that persists through all 3 attempts still fails the leg.
• The logs from failed attempts are kept under attempts/<case>/<n>/, and the final row records transportRetries, so a retried pass shows up in summary.json.
• case_dir was factored out so the scheduler and the case runner agree on the path.

No assertions were weakened and no tests were removed or moved. The workflow and job names are unchanged.

Proof

• New unit tests (PipenvTransportRetryTests in scripts/tests/test_backtest_harnesses.py, run by the workflow's "Harness unit tests" step):
  • A PyPI 503 error is retried from a fresh case and its evidence is kept.
  • A failed check whose log shows a 503 is retried.
  • A patch API 5xx is retried.
  • A macOS DNS failure is retried.
  • Hash-mismatch and resolution failures are not retried.
  • A persistent 503 stays red after 3 attempts.
  • A passing case runs exactly once.
• python3 -B -m unittest scripts/tests/test_backtest_harnesses.py: 59 tests OK (re-run after merging main at acbce34).
• ruff check: same 4 errors as on main, none new.
• Real Pipenv matrix on this PR: all 3 legs green (https://github.com/SocketDev/socket-patch/actions/runs/36817010228), e.g. 16/16 macOS cases PASS.
• Cursor Bugbot: no issues (on head 3ef2997).
• CI: green on 3ef2997. main was merged in at acbce34 because Fix Poetry venv discovery to match Poetry (#327, #329) #330 changed python_crawler.rs venv discovery, which the Pipenv matrix exercises. CI is re-running on that head.

Where tests run

The tests are unchanged. The Pipenv matrix still runs on PRs that touch its paths and on every push to main.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TrR5uqMbj9sJcU2Gpm9y9R