Is there an existing issue for this?

• I have searched the existing issues

Closely related: #27954, #19269 and #10626. This report adds the server-side impact: since #25207, stale clients loop on authenticated /coordinate → 404 after a workspace rebuild.

Current Behavior

Since #25207 (v2.34.0), agents from prior builds are soft-deleted when a new build completes, so GET /api/v2/workspaceagents/{old-agent-id}/coordinate returns 404.

Client processes that resolved the old agent ID keep redialing /coordinate for it forever. That's about once every 10 s per process once backoff hits its cap. They never exit, never re-resolve the workspace's current agent and, without --verbose, print nothing.

AI desktop apps make this much worse by leaking coder ssh --stdio ProxyCommand processes. We found ~30 orphaned processes for a single workspace on one developer's machine.

Every one of these 404s authenticates successfully and does real database work: API key validation, role and custom-role lookups, and the agent lookup. Rate limiting doesn't stop them. Across a few dozen developers, this saturated our database CPU for days and made the API slow for everyone.

Observed impact

• /coordinate responses (from coderd_api_requests_processed_total): ~96% were 404, ~2% 101 (connected), ~1% 401. That was a sustained ~8–15 req/s of 404s, plus ~3–4 req/s of /derp/latency-check. We think the latency checks come from the same stuck clients' netcheck, but haven't verified that.
• Each 404 authenticated successfully. Postgres showed the API key lookup, GetAuthorizationUserRoles and custom-role queries for every request, followed by an agent lookup that returned no rows.
• On one developer's macOS machine: ~30 coder ssh --stdio --hostname-suffix coder <agent>.<workspace>.<user>.coder processes, all for one workspace. They had no TTY, sequential PIDs and ~45–60 s of CPU each, and no parent ssh process remained, so they were orphaned ProxyCommand children. This looks like bug: Windows orphaned coder ssh --stdio processes after interrupted Codex connection retries #27954, which was reported for Codex on Windows, here with Claude Desktop on macOS.
• Killing them dropped the 404 rate by ~2.5–3 req/s. That matches ~30 processes × ~0.1 req/s at the 10 s backoff cap. The remaining rate matches similar leaked processes on other developers' machines.
• Killing one user's stale coder port-forward processes on another occasion dropped the 404 rate to near zero.
• No 429s were observed.

Relevant Log Output

# client, with --verbose (repeats roughly every 10 s, forever)
failed to dial tailnet v2+ API

# server: coderd_api_requests_processed_total, sustained rate
{code="404",method="GET",path="/api/v2/workspaceagents/{workspaceagent}/coordinate"}  ~8–15 req/s

Expected Behavior

Any of these, ideally more than one:

1. Client: treat a 404/403 on reconnect as terminal after a few consecutive attempts. Either re-resolve the workspace's current agent and redial, or close the connection, so coder port-forward / coder ssh exits with a clear error such as "agent no longer exists; workspace was rebuilt". Today 404 is in permanentErrorStatuses, but that only applies to the first dial.
2. coder port-forward: watch the workspace the way coder ssh does (watchAndClose), and exit or redial when it stops or rebuilds. This is the outcome requested in Port-forward stays active after workspace stop/restart, causing dead forward and timeouts #19269 / coder port-forward hangs upon workspace restart, ssh -L exits cleanly #10626.
3. coder ssh --stdio: exit when stdin closes or the parent goes away, even before the session is established (bug: Windows orphaned coder ssh --stdio processes after interrupted Codex connection retries #27954 / fix(cli): exit coder ssh --stdio when stdin closes before connection #28597), so ProxyCommand children can't be orphaned.
4. Backoff: for 4xx errors, use a much higher ceiling than 10 s, with jitter.
5. Server: make dead agent IDs cheap to reject:
   • a distinct status, such as 410 Gone, for soft-deleted agents;
   • per-caller rate limiting on /coordinate (e.g. RateLimitPerCaller) instead of per-path keys that include the agent UUID;
   • and/or a short negative cache.

Steps to Reproduce

Steps 1–3 are from the code and production behavior above; we haven't yet run a minimal isolated repro.

1. Start coder port-forward <ws> --tcp 8080:8080 --verbose, or open an SSH session through the coder config-ssh ProxyCommand.
2. Stop and start (or rebuild) the workspace so it gets a new agent ID.
3. The client keeps sending GET /api/v2/workspaceagents/<old-id>/coordinate → 404 about every 10 s, indefinitely. With --verbose, it repeatedly logs failed to dial tailnet v2+ API. Port-forward never exits, and localhost:8080 hangs (as in Port-forward stays active after workspace stop/restart, causing dead forward and timeouts #19269 / coder port-forward hangs upon workspace restart, ssh -L exits cleanly #10626).
4. N leaked processes produce about N/10 req/s of authenticated 404s.

coder ssh --stdio probably hits the same loop if the rebuild happens before its session is established (inside conn.AwaitReachable). The workspace watcher isn't running yet at that point, and stdin EOF isn't observed (#27954 / #28597). Leaked orphans make this easy to hit.

Environment

• Coder server and CLI: a recent release. Still reproducing after upgrading to a newer patch release. The relevant client and server code is unchanged through the latest release and main (2116682e); see the code pointers below.
• Self-hosted on Kubernetes, with a managed Postgres instance.
• Workspaces are frequently stopped/started or rebuilt. Every new build gets new agent IDs.
• Clients: macOS developers using coder config-ssh (ProxyCommand coder ssh --stdio), driven by AI desktop apps (Claude Desktop, Codex desktop), plus some long-lived coder port-forward processes.

Additional Context

Code pointers (main @ 2116682e)

• codersdk/workspacesdk/dialer.go L21: 404 is in permanentErrorStatuses. But the check is gated on w.isFirst (L103), which is cleared after the first successful dial (L129). Later failures are only logged and returned (L141).
• tailnet/controllers.go L1531: retry.New(50*time.Millisecond, 10*time.Second). Every dial error loops again. The only terminal condition is context cancellation, and retrier.Reset() runs only after a successful dial (L1579).
• codersdk/workspacesdk/workspacesdk.go L256: DialAgent fixes the agent ID into the coordinate URL (L292). It runs the controller on a context.Background()-derived context (L285), and only the first dial's result is surfaced (L359).
• cli/portforward.go L182-185: waits only for a signal or context cancellation, with no workspace watch. The dial logger is attached only with --verbose (L116-118).
• cli/ssh.go: watchAndClose (L981) only starts after conn.AwaitReachable (L519).
• coderd/coderd.go: PrecheckAPIKey validates the API key against the database on every request at the router root (L1185), before any rate limiting. The coordinate route is at L1877.
• coderd/httpmw/ratelimit.go L101: RateLimit keys on user plus the normalized request path (L168). The path includes the agent UUID, so each stale agent ID gets its own bucket. RateLimitPerCaller (L32) exists but isn't used here.
• coderd/httpmw/workspaceagentparam.go L37-39: returns 404 when GetWorkspaceAgentAndWorkspaceByID finds no row. The query excludes deleted = TRUE agents.
• coderd/provisionerdserver/provisionerdserver.go L2319: SoftDeletePriorWorkspaceAgents runs when a build completes.

Possibly relevant history (inferred, not verified): before #25207, prior-build agents weren't soft-deleted, so a stale client's redial likely got a 101 and sat idle instead of looping on 404s. If so, #25207 turned the existing leaks (#10626 / #19269 / #27954) into a steady authenticated request load.

Related

• bug: Windows orphaned coder ssh --stdio processes after interrupted Codex connection retries #27954 (open) and fix(cli): exit coder ssh --stdio when stdin closes before connection #28597 (closed, unmerged): orphaned coder ssh --stdio processes from Codex Desktop retries.
• Port-forward stays active after workspace stop/restart, causing dead forward and timeouts #19269 (open) and coder port-forward hangs upon workspace restart, ssh -L exits cleanly #10626 (closed by the stale bot): coder port-forward survives workspace stop/restart.
• fix(cli): exit port-forward when workspace agent becomes unreachable #24011 / feat: spawn heartbeat for agent connection during port-forward #24350 (closed, unmerged): port-forward watchdog. The review concern was surviving network loss; a 404 for a deleted agent isn't network loss and could be handled separately.
• bug: v2.33.1 aws-instance-identity auth fails with HTTP 409 for instances that have had multiple builds on the same workspace #25155 / fix: soft-delete stale workspace agents on new build #25207: soft-delete prior-build agents (v2.34.0).
• feat: add ServerTailnet logs and metrics for unreachable agents #30133: server-to-agent unreachable metrics. It covers the server side only.