GeezerVison

Summary

Customers running many concurrent upload_file calls through a single SDK instance were hitting bursts of 408 Request timed out waiting for request data from mux/Jetty. Triage on 2026-05-14 traced this to three things, all introduced by recent SDK changes:

• feat: share HTTP connection pool across SDK instances; refactor polling #797 lowered the default httpx pool from (100, 20) → (20, 10) and made it process-global. With many parallel multipart bodies sharing 20 TCP connections instead of 100, per-stream bandwidth dropped to where the server's 30 s request-body idle timeout fires before the body finishes arriving.
• _should_retry blanket-retries 408, so a stalled upload immediately queues another large multipart body onto the same exhausted pool — amplifying the stall instead of recovering from it.
• _files.py read the entire PathLike into memory via read_bytes() before the request was even built, contradicting the docstring's claim of streaming. Holds the pool slot during disk I/O for large files.

Changes

Usage of the new option

import httpx
from runloop_api_client import AsyncRunloop

# Raise the cap for upload-heavy workloads:
client = AsyncRunloop(
    connection_limits=httpx.Limits(max_connections=300, max_keepalive_connections=50)
)

Test plan

• uv run pytest tests/ — 1079 passed locally
• New regression tests: test_upload_file_408_does_not_retry, test_non_upload_file_408_still_retries, TestSyncConnectionLimits, TestAsyncConnectionLimits
• Manual repro on examples/stress_upload_file.py against prod with --concurrency 50 --total 200:
  • Pin to v1.20.3, expect a burst of 408s at ~30 s each
  • Switch to this branch, expect 0 408s and lower wall time
• Optional: customer pre-release try-out, watch their 408 rate in Honeycomb mux

Notes for reviewers

• Three independent fixes are bundled because they share a single customer-visible symptom and a single repro path — separating would force whoever is bisecting in production to back-port them individually. Easy to split if preferred.
• _constants.py is Stainless-generated; the (100, 20) value just restores the pre-feat: share HTTP connection pool across SDK instances; refactor polling #797 default and we should probably move this off the generated file longer-term so it stops being a release-time merge conflict candidate.
• HTTP/2 is left enabled (from Improve devbox polling: eliminate client-side sleep, enable HTTP/2 #793). It may still hurt multipart throughput under contention; happy to add http2: bool = True as a sibling knob if that's a follow-up we want.

🤖 Generated with Claude Code