GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — feat: add ServerTailnet logs and metrics for unreachable agents by BobbyHo · Pull Request #30133 · coder/coder

Conversation

@BobbyHo

@BobbyHo BobbyHo commented Sep 29, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Part of PLAT-680, a sub-issue of PLAT-360.

TL;DR

PLAT-360: after coderd has been up for a long time, its ServerTailnet loses reachability to a subset of workspace agents. Workspace app and web terminal requests to those agents fail with 502 while other agents on the same coderd keep working. A restart clears it, and the customer restarts once the 502 rate crosses their threshold, so the state is gone before anyone can look at it.

Today the default-level logs and the existing metrics only say that an agent was unreachable. They do not say where along the path the connection broke: coordinator, WireGuard tunnel, embedded DERP relay, or the agent's own API connection. The Tailscale internals that would tell us only log at debug, and coderd's log level cannot be changed at runtime.

This PR adds metrics and request log fields at the default level so the next incident can be narrowed down from a normal Prometheus scrape and warn-level logs, without restarting coderd and without running with debug logging on all the time.

chatd also dials agents through the ServerTailnet, so it needs two small changes. It has no request log line, so its failures get their own warn line. It also dials new agents every few seconds while they boot, and abandons dials to agents that a rebuild replaced. Those failures are expected, so chatd marks them and they are not counted.

Change

  • When an agent doesn't answer, the 502 request log line gets the agent ID, a reason, how long the request waited, and peer state (node received, lost flag, handshake time, DERP regions, byte counts).
  • The PTY route and callers with no request log, such as chatd, log one warn line with the same fields.
  • Reverse proxy log lines also record which dial step failed, so a failure after the agent answered a ping is not counted as unreachable.
  • New metrics:
    • coder_servertailnet_agent_unreachable_total{reason}
    • coder_servertailnet_await_reachable_seconds
    • coder_servertailnet_derp_connects_total
    • coder_workspaceapps_agent_offline_total{status,access_method}, since the offline page moved from 502 to 404 in chore: switch agent gone response from 502 to 404 #23090.
  • Peer diagnostics gain TX and RX byte counts. A peer with no handshake reports zero time, not 1970, which also fixes coder ping.
  • Reverse proxy dials stop after one minute.
  • Chat readiness polls, which dial while a new agent boots, are not counted.
New log fields on unreachable-agent lines
Field What it is What it tells us
agent_id The agent the request tried to reach. Lets us group failures per agent and check whether the same agents fail every time or the set moves.
reason Why the attempt was classified as unreachable. Values are in the next table. The single field that says which layer to look at first.
unreachable_after How long the request waited for the agent's first ping before giving up. Separates a slow agent from one that never answers, and shows whether the wait hit the request timeout or the client left.
dial_phase Reverse proxy only. await_reachable means waiting for the first ping. tcp_dial means the ping succeeded and the TCP dial to the app port failed. tcp_dial failures are above the tunnel (the app is not listening) and are not counted as unreachable. Only await_reachable points at the network path.
tcp_dial_after Reverse proxy only, with dial_phase=tcp_dial. Time until the TCP dial failed. Distinguishes an immediate connection refused from a dial that hung inside the tunnel.
peer_node_received Whether the coordinator delivered the agent's node to this coderd. false means the failure is in the control plane: the coordinator never told us where the agent is, so WireGuard was never configured for it.
peer_lost Whether the coordinator marked the peer lost, or this coderd's own coordination stream ended. true means the agent's coordinator connection dropped, or ours did. Many agents lost at once points at this coderd, not the agents.
server_preferred_derp The DERP region this coderd uses as its home relay. Confirms which relay our side of the tunnel goes through. A change over time means coderd re-homed.
peer_preferred_derp The DERP region the agent reported as its home relay. Compared with the server region, shows whether both sides agree on a relay and whether the failing agents share one.
peer_last_handshake Time of the last completed WireGuard handshake with this agent. Omitted when there has never been one. An old handshake with recent traffic means the tunnel was working and then stopped, which is the PLAT-360 shape. Missing means the tunnel never came up.
peer_tx_bytes / peer_rx_bytes Bytes sent to and received from the agent over WireGuard since the peer was added. RX at zero means nothing ever came back from the agent. Both at zero means our packets never left coderd. Large counts on a failing peer mean it worked earlier. Compare two lines for the same agent to see whether the counts still move.
Values of reason
Reason Condition Where to look next
no_node The coordinator never sent the agent's node, or it was removed after a disconnect or lost timeout. Control plane: coordinator, agent API connection, pubsub. WireGuard was never configured for this agent.
peer_lost The coordinator reported the peer lost, or this coderd's coordination stream ended. The agent's coordinator connection or this coderd's. A spike across many agents means this coderd lost its coordinator stream.
no_handshake The node arrived but WireGuard never completed a handshake. The path between coderd and the agent: DERP relay, the in-memory pipe to the embedded relay, or the agent's DERP connection. Check the byte counts to see which direction is silent.
handshake_stale The last handshake is older than 180 seconds, WireGuard's session limit. The tunnel worked and then our handshakes went unanswered. This is the PLAT-360 symptom. Byte counts show whether anything still flows.
handshake_ok The last handshake is under 180 seconds old. The agent answered a handshake recently. It may have stopped answering since then, or the failure may be above WireGuard. Compare the byte counts on two lines for the same agent.
diagnostics_timeout The peer state could not be read within one second. The WireGuard engine lock was busy. On its own it is a sign of a stuck engine, which is itself evidence.
client_canceled The client went away within five seconds. Not a network failure. Counted separately so it does not inflate the other reasons.
New metrics
Metric What it is What it tells us
coder_servertailnet_agent_unreachable_total{reason} Counter of connection attempts where the agent did not answer its first ping, labeled by the reason above. The rate is the alert signal. The reason split says whether an incident is a control plane failure, a tunnel failure, or something above the tunnel, without reading logs. No per-agent label, so cardinality is fixed.
coder_servertailnet_await_reachable_seconds Histogram of time until an agent answered the first ping on a new connection. A drift toward the upper buckets before agents become unreachable is the early warning of a slow loss of reachability, and shows whether the failure is sudden or gradual.
coder_servertailnet_derp_connects_total Counter of times the ServerTailnet connected to the embedded DERP relay over the in-memory pipe. Reads 1 after startup, or stays 0 when the embedded relay is disabled. A rise means the relay connection was rebuilt. If it rises alongside unreachable failures, the in-memory pipe to the embedded relay is the suspect. If it stays flat, the relay side is stable.
coder_workspaceapps_agent_offline_total{status,access_method} Counter of app and terminal requests that got the offline page because the agent's database status was not connected. These requests never reach the tunnel. A rise means agents are dropping their API connection to coderd, which is a different failure from the tunnel losing them. Separates the two 502 writers the customer sees.
Where each signal sits on the network path

The diagram shows the DERP-only path, which is the customer's setup. With direct connections enabled, WireGuard traffic can bypass the relay, but the coordinator, offline check, and handshake fields are the same.

  browser / CLI
       |
       |  HTTPS: workspace app, web terminal, chat
       v
+-------------------------------------------------------------------------------+
|  coderd                                                                        |
|                                                                                |
|  app proxy / PTY route / chatd                                                 |
|    agent DB status check                                                       |
|    [M4] agent_offline_total ........ agent not "connected" in the database,    |
|         |                            request stops here (agent API is down)    |
|         v                                                                      |
|  ServerTailnet.AgentConn / ReverseProxy                                        |
|    wait for first ping                                                         |
|    [M2] await_reachable_seconds .... time to first answer on success           |
|    [M1] agent_unreachable_total .... no answer; request log line gets          |
|    [L]  reason, unreachable_after,   agent_id, reason, dial_phase, peer_*      |
|         dial_phase, tcp_dial_after                                             |
|         |                                                                      |
|         |   control plane: coordinator (node exchange)                         |
|         |   [L] peer_node_received, peer_lost                                  |
|         v                                                                      |
|  WireGuard / magicsock                                                         |
|    [L] peer_last_handshake, peer_tx_bytes, peer_rx_bytes,                      |
|        server_preferred_derp, peer_preferred_derp                              |
|         |                                                                      |
|         |  in-memory pipe                                                      |
|         v                                                                      |
|  embedded DERP relay                                                           |
|    [M3] derp_connects_total ........ times this pipe was (re)built             |
+---------|----------------------------------------------------------------------+
          |                                          ^
          |  DERP over HTTPS / WebSocket             |  agent API over WebSocket:
          v                                          |  coordination, stats, status
     workspace agent  -------------------------------+
       (WireGuard peer, netstack, apps, PTY)

  M1..M4  new Prometheus metrics      L  new fields on the warn-level log line

Reading top to bottom: a rise in M4 means agents are losing their API connection before any tunnel is involved. M1 with no_node or peer_lost means the coordinator step failed. M1 with no_handshake or handshake_stale means the WireGuard step failed, and the byte counts say in which direction. A matching rise in M3 puts the fault at the relay pipe. M1 with handshake_ok means the last handshake was recent; it does not show whether the tunnel still works.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@BobbyHo
feat: add ServerTailnet logs and metrics for unreachable agents
5c0f171
When the ServerTailnet cannot reach an agent, the only record today is
the error page text in the request log body. This adds enough default
logging and metrics to tell a control plane failure from a data plane
failure without a coderd restart or verbose logging (PLAT-680).

- Attach peer diagnostics to the request log line when AwaitReachable
  fails: agent_id, unreachable_after, peer_node_received,
  peer_last_handshake, preferred_derp, peer_tx_bytes, peer_rx_bytes.
  The read is capped at one second so a busy engine lock cannot delay
  the response. The PTY route logs one warn line with the same fields
  since its 101 response keeps the request log at debug.
- Add TxBytes and RxBytes to tailnet.PeerDiagnostics.
- Add coder_servertailnet_agent_unreachable_total{reason} and
  coder_servertailnet_await_reachable_seconds.
- Add coder_servertailnet_derp_reconnects_total, incremented in the
  embedded DERP region dialer.
- Add coder_workspaceapps_offline_total{status,access_method} for the
  agent offline page. The existing debug log is unchanged.

No new log lines on successful connections and no per-agent labels.
@github-actions

github-actions Bot commented Sep 29, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Docs preview

Check off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here.

  • docs/admin/integrations/prometheus.md
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@linear-code

linear-code Bot commented Sep 29, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

PLAT-680

PLAT-696

PLAT-697

PLAT-698

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@github-actions github-actions Bot assigned BobbyHo Sep 29, 2026
@BobbyHo

BobbyHo commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

/coder-agents-review

coder-agents-review[bot] reacted with rocket emoji
All reactions
  • 🚀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@coder-agents-review

coder-agents-review Bot commented Sep 29, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Chat: Review posted | View chat
Requested: 2026-10-13 17:53 UTC by @BobbyHo

Review history
  • R1 (2026-09-29): 12 reviewers, 3 Nit, 2 P1, 1 P2, 8 P3, 6 P4, REQUEST_CHANGES. Review
  • R2 (2026-09-30): 12 reviewers, 5 Nit, 2 P1, 3 P2, 14 P3, 10 P4, COMMENT. Review
  • R3 (2026-09-30), 5 Nit, 2 P1, 3 P2, 14 P3, 10 P4, COMMENT. Review
  • R4 (2026-09-30): 13 reviewers, 7 Nit, 2 P1, 3 P2, 18 P3, 14 P4, COMMENT. Review
  • R5 (2026-10-01): 14 reviewers, 10 Nit, 2 P1, 3 P2, 20 P3, 19 P4, COMMENT. Review
  • R6 (2026-10-13): 15 reviewers, 11 Nit, 2 P1, 3 P2, 21 P3, 22 P4, COMMENT. Review

deep-review v0.13.0 | Round 6 | adcce66..c1fd7e9

Last posted: Round 6, 59 findings (2 P1, 3 P2, 21 P3, 22 P4, 11 Nit), COMMENT. Review

Finding inventory

Findings

# Sev Status Location Summary Round Reviewer Posted
CRF-1 P3 Author fixed (cb69c0e) tailnet.go:372 no_handshake, handshake_ok, diagnostics_timeout reasons and peer_tx/rx_bytes fields untested R1 Netero Yes
CRF-2 P3 Author fixed (bc12263) tailnet.go:344 Client that gives up during AwaitReachable is counted in agent_unreachable_total and logged at WARN R1 Netero Yes
CRF-3 P4 Author fixed (b7f4705; PR description edited) tailnet.go:79 derp_reconnects_total counts the first connect; Help "1 after startup" is false when embedded DERP is disabled R1 Netero Yes
CRF-4 P4 Author fixed (426bc02) proxy.go:813 PTY Warn branch for AgentUnreachableError has no test R1 Netero Yes
CRF-5 Note Dropped by orchestrator (verification, no action) tailnet.go:1 Mechanical checks (dashes, docs regen, build, vet, tests) found nothing R1 Netero No
CRF-6 OOS Deferred (PLAT-696) wsproxy.go:300 Workspace proxy ServerTailnet is not registered with Prometheus, so the new servertailnet metrics are absent on proxies R1 Netero No
CRF-7 P1 Author fixed (65ba780) tailnet.go:344 Reverse-proxy dial context has no deadline, so the 502 line gets no fields and agent_unreachable_total does not increment R1 Takumi, Ryosuke, Melody, Pariston, Mafuuu, Bisky, Chopper Yes
CRF-8 P1 Author fixed (PR description edited R6; Help text b7f4705) tailnet.go:381 Never-handshaked (1970 timestamp) and stale-handshake peers are counted as handshake_ok R1 Luffy P1, Chopper P2, Pariston P3, Mafuuu P3, Gon P3, Leorio P3 Yes
CRF-9 P3 Author fixed (65ba780) tailnet.go:398 rl.WithFields runs on the transport dial goroutine and races with WriteLog R1 Mafuuu Yes
CRF-10 P3 Author fixed (cb69c0e) tailnet.go:368 Blocked GetPeerDiagnostics leaves one goroutine per failed dial, unbounded R1 Killua, Takumi Yes
CRF-11 P3 Author fixed (0ce724c) db.go:253 offline_total duplicates coderd_agents_connections{status} for the PR's stated question, with more noise R1 Luffy Yes
CRF-12 P3 Author fixed (74e9d79) conn.go:969 TxBytes/RxBytes are cumulative per peer; description's "rx at zero" rule applies only to peers that never carried traffic R1 Luffy P3, Pariston Note, Mafuuu Note Yes
CRF-13 P2 Author fixed (0ce724c) db.go:52 appOfflineTotal comment says stopped workspaces feed the baseline; they never reach the counter R1 Gon Yes
CRF-14 P3 Author fixed (0ce724c) db.go:84 offline_total name covers both offline pages; only the disconnected-agent page increments it R1 Gon Yes
CRF-15 P4 Author fixed (cb69c0e) tailnet.go:360 agentUnreachableError name hides metric increment and request logger mutation R1 Gon Yes
CRF-16 P2 Dropped by orchestrator (merged into CRF-15; the comment is the only statement of the side effects while the name stays) tailnet.go:356 agentUnreachableError doc comment narrates the body R1 Gon No
CRF-17 Nit Author fixed (cb69c0e) tailnet.go:351 peerDiagnosticsTimeout comment repeats itself R1 Gon P2 Yes
CRF-18 P2 Dropped by orchestrator (the sentence states what Fields contains and when it was collected; the rewrite drops that) proxy.go:85 AgentUnreachableError doc comment restates the field R1 Gon No
CRF-19 Nit Author fixed (426bc02) proxy.go:811 PTY comment omits why Warn uses s.Logger instead of log R1 Gon P2 Yes
CRF-20 P2 Dropped by orchestrator (proposed rewrite has the same content and length) tailnet_test.go:75 Test comment longer than the setup fact R1 Gon No
CRF-21 P2 Dropped by orchestrator (merged into CRF-3; the comment changes with the metric rename) tailnet_test.go:132 Test comment "for the life of the process" wrong for a test R1 Gon No
CRF-22 Nit Author fixed (cb69c0e) tailnet.go:393 peer_diagnostics_timeout field and diagnostics_timeout reason name one condition two ways; log line lacks reason R1 Gon Yes
CRF-23 Note Dropped by orchestrator (verification, no action) tailnet.go:149 Inline histogram buckets match repo convention R1 Gon No
CRF-24 P3 Author fixed (cb69c0e) tailnet.go:376 preferred_derp is coderd's own home region, logged among peer_* fields R1 Leorio Yes
CRF-25 P4 Author fixed (cb69c0e) tailnet.go:375 peer_last_handshake logs year 1 or 1970 for "never" R1 Leorio Yes
CRF-26 OOS Deferred (PLAT-697) proxy.go:93 AgentUnreachableError.Error() keeps "agent is unreachable" with no wait time or next step R1 Leorio No
CRF-27 Note Dropped by orchestrator (merged into CRF-8 fix: list reason values in Help text) tailnet.go:351 Copy the PR's field-to-subsystem mapping into docs R1 Leorio No
CRF-28 P4 Author fixed (6567d0a) tailnet_test.go:66 Success test checks only that no_node is absent R1 Bisky Yes
CRF-29 Note Dropped by orchestrator (verification, no action) tailnet_test.go:1 Remaining new assertions are real and pass R1 Bisky No
CRF-30 P4 Author fixed (426bc02) proxy.go:813 Use errors.AsType instead of var plus errors.As R1 Ging-go Yes
CRF-31 Note Dropped by orchestrator (verification, no action) db.go:253 New metrics have bounded labels and no hot-path cost R1 Killua No
CRF-32 Note Dropped by orchestrator (verification, no action) db.go:253 offline_total cardinality bounded; db_test label order correct R1 Chopper No
CRF-33 Note Note (body) conn.go:969 TxBytes/RxBytes also appear in support bundle peer_diagnostics.json R1 Melody No
CRF-34 OOS Author fixed (65ba780, proxyDialTimeout) tailnet.go:298 dialContext runs AwaitReachable unbounded; abandoned app requests leak dial goroutines and coordinator tickets R1 Takumi, Ryosuke, Melody, Pariston, Chopper No
CRF-35 OOS Author fixed (568c5e7, epoch normalized in fillPeerDiagnostics) cli/cliui/agent.go:459 Same IsZero check shows a 1970 handshake instead of "Wireguard is not connected" R1 Orchestrator (from Luffy's reproduction) No
CRF-36 P3 Author fixed (f7bec69) tailnet.go:416 Request that got a pooled conn while its own dial runs is counted unreachable when the pooled conn fails R2 Netero Yes
CRF-37 P3 Author fixed (ed06d32) tailnet.go:422 tcp_dial and errAgentUnreachable branches of reportProxyDialFailure untested R2 Netero Yes
CRF-38 P4 Author fixed (6dcae29) proxy_internal_test.go:169 lockedBuffer duplicates testutil.WaitBuffer R2 Netero Yes
CRF-39 Nit Author fixed (6dcae29) tailnet_test.go:135 Test comment still says "for the life of the process" after the derp_connects rename R2 Netero Yes
CRF-40 Note Dropped by orchestrator (verification, no action) tailnet.go:1 Round 1 fixes and mechanical checks verified R2 Netero No
CRF-41 P2 Author fixed (bc12263) tailnet.go:519 Skipped diagnostics reads (another read in flight) are counted as diagnostics_timeout; introduced by the CRF-10 fix R2 Takumi P2, Ryosuke P3, Melody P3, Gon P3, Leorio P3 Yes
CRF-42 P3 Author fixed (b7f4705) tailnet.go:143 agent_unreachable_total Help lists reason values but explains only handshake_stale and client_canceled R2 Leorio Yes
CRF-43 P4 Author fixed (dcad2e9) tailnet.go:387 ensureAgent failure on the proxy path is counted as unreachable and prefixed "agent is unreachable"; AgentConn does not count it R2 Melody, Mafuuu Yes
CRF-44 P3 Author fixed (70b54f8) tailnet.go:511 chatd cancels stale dials after its 5s validation delay; each is counted as unreachable with a diagnostics reason R2 Kite Yes
CRF-45 P3 Author fixed (bc12263) configmaps.go:652 PeerDiagnostics omits the peer lost flag, so lost-coordinator failures look like WireGuard failures for 15 min R2 Mafuuu Yes
CRF-46 P4 Author fixed (bc30966) configmaps.go:659 Zero-time change sends last_handshake seconds=-62135596800 to Coder Desktop instead of the proto zero value R2 Kite P4, Melody Note Yes
CRF-47 P3 Author fixed (dcad2e9) tailnet.go:541 No test proves the diagnostics gate is released after a read, or covers the 1s timeout branch R2 Bisky Yes
CRF-48 P4 Author fixed (facf0b6) tailnet.go:145 await_reachable_seconds answers no question in the PR's triage R2 Luffy Yes
CRF-49 P2 Author fixed (f33df09) tailnet.go:380 dialContext comment says the error handler reports a dial that fails after the request ended; nothing reports that R2 Gon P2, Leorio P4 Yes
CRF-50 P2 Dropped by orchestrator (the transport clause is the reason for the request-goroutine constraint; rewrite keeps the same three facts) tailnet.go:407 reportProxyDialFailure doc comment is six lines R2 Gon No
CRF-51 P2 Dropped by orchestrator (12-word sentence pairs logging with not counting; rewrite is the same length) tailnet.go:423 tcp_dial comment restates the code R2 Gon No
CRF-52 P2 Dropped by orchestrator (text applies the R1 suggestion; Leorio R2 endorses it; "failed request" is defined by the prior sentence) db.go:52 appOfflineTotal comment carries alerting advice R2 Gon No
CRF-53 Nit Author fixed (6dcae29) db.go:57 appOfflineTotal field and local keep the old name after the agent_offline_total rename R2 Gon Yes
CRF-54 Nit Dropped by orchestrator (metricsdocgen scanner resolves Help only from literals and constants, scanner.go:303; fmt.Sprintf would drop the metric from docs) tailnet.go:143 Help hard-codes 180s and 5s R2 Gon No
CRF-55 Note Dropped by orchestrator (only consumer reads theErr.Error(); fields are already on the log line) tailnet.go:421 xerrors wrap flattens AgentUnreachableError to text R2 Ging-go No
CRF-56 Note Note (body) tailnet.go:414 Failures on a reused pooled connection get no agent_id, reason, or count R2 Pariston No
CRF-57 Note Note (body) tailnet.go:308 60s proxy dial cap can turn slow first pings into new 502s after upgrade R2 Kite No
CRF-58 Note Dropped by orchestrator (merged into CRF-36 fix) tailnet.go:416 Record GotConn via httptrace to know whether the request got a connection R2 Ryosuke No
CRF-59 Note Dropped by orchestrator (verification, no action) tailnet.go:1 R1 fixes verified (CRF-7, 9, 10, 24, 25, 28, 30, 34) R2 Ging-go, Killua, Takumi, Ryosuke, Melody, Bisky, Leorio, Pariston No
CRF-60 P3 Author fixed (60da3a2) tailnet_internal_test.go:152 TestRecordAgentUnreachable_DiagnosticsWait passes when peerDiagnostics skips a busy slot R4 Netero Yes
CRF-61 P4 Author fixed (PR description) tailnet.go:146 PR description omits peer_lost and says overlapping failures report diagnostics_timeout at once R4 Netero Yes
CRF-62 Note Note (body) conn.go:970 Lost also appears in support bundle peer_diagnostics.json R4 Netero No
CRF-63 Note Dropped by orchestrator (verification, no action) tailnet.go:1 R4 mechanical checks and CRF-2/41/45 fix verification R4 Netero No
CRF-64 P3 Author fixed (dfd13e2) tailnet.go:490 chatd waitForAgentReady dials a booting agent in 5s attempts for up to 2m; each attempt is counted as no_node R4 Melody, Knov Yes
CRF-65 P3 Author fixed (60da3a2) tailnet.go:565 chatd dials have no request logger, so their unreachable counts have no log line with agent_id or peer fields R4 Ryosuke Yes
CRF-66 P4 Author fixed (60da3a2) tailnet.go:449 gotConn stays true across a transport retry, so a retry dial that waits on an unreachable agent gets no fields or count R4 Takumi Yes
CRF-67 P3 Author fixed (60da3a2) tailnet_internal_test.go:197 TestReportProxyDialFailure_PooledConn passes with the hadConn check removed R4 Bisky Yes
CRF-68 P4 Author fixed (60da3a2) conn.go:969 Lost comment says node and config stay for lostTimeout; removal is lostTimeout after the last handshake and re-arms on new handshakes R4 Gon P3, Mafuuu P4, Kite P4 Yes
CRF-69 P4 Author fixed (60da3a2) tailnet.go:146 Help text: no_node also covers a delivered node removed after DISCONNECTED or lost expiry; peer_lost does not say whose stream ended R4 Mafuuu, Kite, Leorio Yes
CRF-70 Nit Author fixed (60da3a2) tailnet_internal_test.go:263 Test name and one test comment call peerDiagnosticsSlot a "gate" R4 Gon Yes
CRF-71 Nit Author fixed (60da3a2) proxy.go:85 AgentUnreachableError doc omits the client_canceled exception, which decides PTY Warn vs Debug R4 Mafuuu Yes
CRF-72 P2 Dropped by orchestrator (lines 577-578 already say other callers wait for the slot "within the same timeout"; the rewrite adds no fact) tailnet.go:575 peerDiagnostics doc comment repeats the engine-lock fact and omits the shared timer R4 Gon No
CRF-73 P4 Dropped by orchestrator (text reaches only a debug log and a response to a client that already left; the request log line carries reason=client_canceled) tailnet.go:570 client_canceled error text still says "agent is unreachable" R4 Gon, Leorio No
CRF-74 Nit Dropped by orchestrator (field, setter, and getter names are each clear; renaming is churn) tailnet.go:392 dialState names one flag three ways (gotConn, markGotConn, hadConn) R4 Gon No
CRF-75 P4 Dropped by orchestrator (both values are set together in director; a new type for no behavior change) tailnet.go:303 Put agent ID and dialState under one context key R4 Ryosuke No
CRF-76 P4 Dropped by orchestrator (proposed home coderd/workspaceapps depends on coderd/x/chatd/chattool per go list -deps, so chattool, where the CRF-64 fix would set a cause, could not import it; workspacesdk is the package both sides share) workspacesdk.go:34 ErrDialAbandoned is an exported SDK name whose effect exists only in coderd R4 Knov No
CRF-77 Note Dropped by orchestrator (verification, no action) tailnet.go:1 R4 panel checks: slot bounds blocked goroutines, CRF-44 test still guards, targeted tests pass, no Go idiom findings R4 Ging-go, Killua, Takumi, Melody, Bisky, Kite, Knov, Mafuuu No
CRF-78 P4 Author fixed (b7f4705) tailnet.go:580 ServerTailnet has a quartz clock, but AgentConn, acquireAgent, and recordAgentUnreachable still use time.Now/time.Since R5 Netero Yes
CRF-79 P4 Author fixed (b7f4705) tailnet.go:395 startDial can run after GotConn for the same getConn and clear gotConn, reopening the pooled-connection misclassification R5 Netero, Takumi, Gon Yes
CRF-80 Note Dropped by orchestrator (verification, no action) tailnet.go:1 R5 mechanical checks, generated docs, mutation checks of the R4 fixes R5 Netero No
CRF-81 OOS Out of scope; author accepted R6 (lines not changed by this PR) createworkspace.go:338 Em dashes in unchanged comment lines of files the PR touches (createworkspace.go, dialvalidation_internal_test.go, workspacesdk.go) R5 Netero No
CRF-82 P3 Author fixed (b7f4705) createworkspace.go:640 No test fails when waitForAgentReady stops setting ErrReadinessProbeTimeout on the attempt context R5 Knov P3, Ryosuke P4, Melody P4, Bisky P4, Kite P4, Netero Note Yes
CRF-83 P4 Author fixed (b7f4705) createworkspace.go:640 ErrReadinessProbeTimeout is set on start_workspace polls of an already connected agent, so a broken tunnel there is counted once per 2m call instead of per attempt R5 Kite P3, Melody P4, Mafuuu P4, Knov P4, Luffy P4 Yes
CRF-84 P3 Author fixed (b7f4705); premise disproved R6: toolsdk WithCleanContext drops the request logger, so the fix duplicates the warn line (CRF-99) tailnet.go:585 MCP HTTP tool calls have a request logger but return 200, so their unreachable counts get the fields only on a debug line R5 Mafuuu P3, Kite P4, Razor P4 Yes
CRF-85 P4 Author fixed (b7f4705) tailnet.go:561 recordAgentUnreachable doc omits the new warn line; AgentUnreachableError tells every caller to log the fields again R5 Gon P3, Melody Note Yes
CRF-86 Nit Author fixed (b7f4705) proxy.go:86 AgentUnreachableError doc omits the ErrDialAbandoned and ErrReadinessProbeTimeout cases that also return a plain error R5 Razor P4 Yes
CRF-87 P4 Author fixed (b7f4705) tailnet_internal_test.go:330 TestRecordAgentUnreachable_DiagnosticsSlot runs on the real clock and needs the slot freed within 1s of wall time R5 Komugi Yes
CRF-88 Nit Author fixed (b7f4705) tailnet_internal_test.go:253 PooledConn test comment says the error is from the connection; the test passes context.Canceled after cancel() R5 Gon P4 Yes
CRF-89 Nit Author fixed (b7f4705) tailnet.go:590 New warn line says "agent is unreachable" while the PTY warn line for the same event says "dial workspace agent" R5 Gon Yes
CRF-90 Note Deferred (PLAT-698) tailnet.go:350 One dialState per request is shared by every dial; an earlier dial finishing sets done while a retry dial waits R5 Takumi No
CRF-91 Note Author accepted R6 (both labels mean the peer state was not read within 1s) tailnet.go:607 A failure that times out waiting for the diagnostics slot is labeled diagnostics_timeout like a hung read R5 Komugi P3 No
CRF-92 Note Author fixed (b7f4705, testutil.RandomPort) tailnet_test.go:344 TCPDialFailure test needs host port 65000 free; inside the ephemeral range on macOS and Windows R5 Komugi P4 No
CRF-93 P3 Dropped by orchestrator (R1 review asked for recording in the ErrorHandler; the dial-level alternative drops the fields from the 502 line, records up to 60s after the 502, and skips requests whose dial succeeds after the request ended) tailnet.go:460 Record unreachable dials in dialContext with s.logger and drop dialState, the trace hook, and reportProxyDialFailure R5 Luffy No
CRF-94 Note Dropped by orchestrator (verification, no action) tailnet.go:1 R5 panel checks: R4 fixes verified by mutation, targeted tests pass under -race and constrained GOMAXPROCS, no Go idiom findings R5 Ging-go, Takumi, Ryosuke, Melody, Bisky, Gon, Leorio, Luffy, Mafuuu, Kite, Komugi No
CRF-95 P4 Open createworkspace_internal_test.go:202 TestWaitForAgentReady/AttemptTimeoutCause waits the real 5s agentAttemptTimeout in each subtest R6 Netero Yes
CRF-96 P4 Dropped by orchestrator (merged into CRF-99: the MCP branch should be removed, and the test should assert one warn line) mcp_http.go:80 The MCP warn line for AgentUnreachableError (CRF-84 fix) has no test R6 Netero No
CRF-97 Note Dropped by orchestrator (R6 Ryosuke P4 and Meruem P4 re-raised: no production behavior differs, the author R5 reply states the choice, and no current test needs the mock clock on the proxy path) tailnet.go:359 dialState set/get still use time.Now/time.Since R6 Netero No
CRF-98 Note Dropped by orchestrator (verification, no action) tailnet.go:1 R6 mechanical checks, generated docs, mutation checks of the R5 fixes R6 Netero No
CRF-99 P3 Open mcp_http.go:80 MCP tool context has no request logger, so the CRF-84 fix writes a second identical warn line per MCP failure R6 Takumi P3, Razor P4 Yes
CRF-100 P4 Open createworkspace.go:642 Readiness poll decides from a FirstConnectedAt snapshot: a disconnected agent is counted on every attempt, a new agent that connects mid-loop is never counted R6 Melody, Knov, Kite, Luffy, Komugi Note Yes
CRF-101 P4 Open tailnet.go:519 await_reachable_seconds records successful readiness probes of booting chat agents R6 Mafuuu Yes
CRF-102 Nit Open tailnet.go:381 hadConn doc says the request had a connection; since GetConn resets it, it is per attempt R6 Gon Yes
CRF-103 P4 Dropped by orchestrator (the next line shows setGotConn(false); "clears the connection the previous attempt used" reads as clearing the record of it) tailnet.go:295 director comment says a retry clears the connection, not the flag R6 Gon No
CRF-104 Nit Dropped by orchestrator (a context whose deadline passes is canceled with its cause; the doc wording is accurate) proxy.go:87 AgentUnreachableError doc says canceled with ErrReadinessProbeTimeout, a timeout cause R6 Gon No
CRF-105 P4 Dropped by orchestrator (superseded by CRF-99: once the MCP branch is removed, MCP falls under "callers with no request log") mcp_http.go:80 PR description does not mention the MCP warn line R6 Leorio No
CRF-106 P3 Dropped by orchestrator (R5 offered always-warn as one CRF-84 option and the author chose per-caller; after CRF-99 only PTY re-logs and the doc states the rule; the MCP evidence is disproved) tailnet.go:559 recordAgentUnreachable equates a request logger with a warn-level line; add a level floor or always warn R6 Meruem No
CRF-107 OOS Out of scope (body) toolsdk.go:367 MCP tool dials have no deadline and WithCleanContext keeps only cancellation, so an MCP call to an unreachable agent waits in AwaitReachable and is rarely counted R6 Takumi, Razor No
CRF-108 Note Dropped by orchestrator (verification, no action) tailnet.go:1 R6 panel checks: GetConn/GotConn order verified, R5 fixes verified by mutation, mock-clock tests pass 200x under GOMAXPROCS 1/2/4, port test analysis R6 Ging-go, Takumi, Ryosuke, Melody, Pariston, Mafuuu, Bisky, Gon, Leorio, Luffy, Knov, Kite, Komugi, Razor No

Contested and acknowledged

CRF-91 (Note, tailnet.go:607) - Slot-wait timeout shares the diagnostics_timeout label

  • Finding: A failure that times out while waiting for the single diagnostics slot is labeled diagnostics_timeout, the same as a read that hung on the engine lock, so in a burst of several hundred failures within one second the label reflects queueing, not the engine (Komugi R5, recorded as a Note).
  • Author accepted (R6, IC_kwDOGkVX1s8AAAABYQKwww): "Both mean the peer state was not read within 1s." The R3 review offered the shared-timeout design, and the author applied it.

Law analysis

Not run. Effective additions 245, below the 1000 threshold. Head 5c0f171.

R4: effective +1150 -34 (426 production, 724 test), head bc12263. Verdict: Don't split. Enforcement: Advisory. Optional vertical order if the author splits: V1 (PeerDiagnostics fields and epoch plus vpn), V2 (60s proxy dial cap), V3 (AgentConn classification, with the DERP and offline counters folded in), V4 (proxy dial-phase reporting).

Round log

Round 1

Netero first pass: 2 P3, 2 P4, 1 Note, 1 OOS. No P0, so the panel ran. Law skipped (245 effective additions).

Round type: round 1, full panel. Classified focused (Go only; docs and generated_metrics are metricsdocgen output). Target 6-8, exceeded to 12 plus wildcard: specialists ging-go (Go files), killua (AgentConn hot path, goroutine on failure path), takumi (goroutine, channel, time.After select), ryosuke (exported AgentUnreachableError, NewDBTokenProvider signature, loggermw import), melody (Describe/Collect pairing, error fields crossing AgentProvider), luffy (new metrics feature) plus core pariston, mafuuu reach 8; bisky required (tests changed); gon and leorio for the once-per-PR floor. Wildcard chopper (shuf draw from meruem, chopper, kite, zoro, knov, razor). Not selected: pen-botter (docs diff is generated from Help strings; leorio covers Help text), mafu-san (human-account author, pool already over target), kurapika (no auth, input, or secrets), knuckle/nami (no SQL or frontend).

Orchestrator checks: read Go 1.26.5 net/http/transport.go:1529 and confirmed cancelCtx is called only from CloseIdleConnections (line 912), so the reverse-proxy dial context has no deadline (CRF-7). Confirmed SlogRequestLogger.WithFields has no lock (CRF-9). Confirmed getPeerStatusLite sets LastHandshake = time.Unix(0, ...) in the pinned tailscale fork (CRF-8). Confirmed PreferredDERP comes from nodeUpdater (our node), and that tailcfg.Node carries the peer region in DERP ("127.3.3.40:N"), not PreferredDERP, so Leorio's suggested field name is corrected in the posted comment (CRF-24). Confirmed errWorkspaceStopped path returns before the counter (CRF-13). Confirmed PTY passes the websocket-watched request context with no deadline, while several workspaceagents.go and exp_chats.go callers use a 30s dialCtx; so Netero's first fix for CRF-2 (skip context.Canceled) would drop every PTY failure, and the posted fix labels by ctx.Err() instead. errors.As has 82 non-test uses under coderd/ and errors.AsType 26, so CRF-30 stays P4, not Nit.

Cross-check:

  • CRF-7 convergent across 7 reviewers, all P1, 5 independent scratch-test reproductions. Chain with CRF-9, CRF-10, CRF-34: on the app path each abandoned request leaves a dial running; at shutdown Close cancels all pending dials at once, each increments the counter, starts a diagnostics goroutine, and calls WithFields on an already-written logger.
  • CRF-8 severities P1 (Luffy) to P3. Upward case: Luffy reproduced the 1970 value, so every node-received failure is labeled handshake_ok, the value the PR's triage reads as "WireGuard is fine", and the PLAT-360 stale case lands there too. Downward case: the log line carries peer_last_handshake, so humans reading logs can see it. Kept P1: the reason label is the metric the PR adds for alerting, and it reports the opposite state. Gon's handshake_seen rename and Leorio's docs note folded into the fix.
  • CRF-1 interacts with CRF-8: a node-received test would have exposed the 1970 value.
  • CRF-11, CRF-13, CRF-14 interact: removing the counter (CRF-11) makes the other two moot.
  • CRF-16 contradicts CRF-15 as raised (Gon asks to cut the comment that states the side effects, and separately says the name hides them). Keep-argument for CRF-16: the comment restates the body. Merged into CRF-15, whose fix renames first and then trims the comment.
  • CRF-17 and CRF-19 downgraded P2 to Nit. Keep-argument: AGENTS.md requires concise comments, and CRF-19 has a real trap (log vs s.Logger). Downgrade reason: the comments are accurate; the severity definitions put comment wording under Nit.
  • CRF-18 dropped. Keep-argument: the sentence restates the field. Drop reason: it says the fields are peer diagnostics collected at failure time, which []slog.Field does not show.
  • CRF-20 dropped. Keep-argument: concise rule. Drop reason: proposed text has the same content and nearly the same length.
  • CRF-21 merged into CRF-3.
  • CRF-26 (Leorio P4) moved to out of scope: the string is unchanged from the removed xerrors.New, so the diff does not cause the missing detail; Leorio's 502-page example is also contradicted by CRF-7.
  • CRF-10: Killua's memory estimate used an 8 KiB minimum stack; Go's minimum is 2 KiB. The posted comment omits the number. Neither reviewer reproduced a held lock; posted as unverified.
  • CRF-35 added by orchestrator from Luffy's reproduction and cli/cliui/agent.go:459; CLI output not reproduced.

Reviewed against 6c6b6fe..5c0f171.

Round 2

Churn guard: PROCEED, 20 of 20 open findings addressed (author claims fixed, commits 65ba780..426bc02). Netero (advisory, post-panel): CRF-36 to CRF-40. Law skipped (819 effective additions, below 1000).

Round type: restructure (files added: coderd/tailnet_internal_test.go, coderd/workspaceapps/proxy_internal_test.go; agentUnreachableError rewritten into recordAgentUnreachable, acquireAgent, peerDiagnostics, unreachableFields, unreachableReason, plus dialState; effective additions +574 since R1). Full panel, focused (Go only): specialists ging-go, killua, takumi, ryosuke, melody, luffy; core pariston, mafuuu; bisky (tests changed); gon, leorio (restructure round floor). Wildcard kite (shuf draw from meruem, chopper, kite, zoro, knov, razor). Size 12 for the same reason as R1. No contested findings, so all agents were new.

Inventory validation: CRF-34 (OOS, unbounded proxy dial) now fixed in-PR by proxyDialTimeout (confirmed by Killua, Takumi, Ryosuke, Pariston). CRF-35 (OOS, cliui 1970) fixed by the epoch normalization in fillPeerDiagnostics. CRF-6 and CRF-26 unchanged; still need a human decision and are re-listed in the body.

Orchestrator checks: net/http/transport.go tryPutIdleConn delivers a returned idle conn to a waiter that may still be dialing (supports CRF-36). scripts/metricsdocgen/scanner/scanner.go:303 resolveStringExpr handles only literals, identifiers, concatenation, and selectors, so a fmt.Sprintf Help would resolve to "" and the scanner skips the metric (CRF-54 drop). coderd/x/chatd/chatd.go:72 workspaceDialValidationDelay = 5s and dialvalidation.go:185 dialCancel() on a stale binding (CRF-44). configmaps.go:526 sets lc.lost, controllers.go:335 calls SetAllPeersLost, lostTimeout = 15m (CRF-45). vpn/vpn.proto:133 documents the zero value for last_handshake (CRF-46).

Cross-check:

  • CRF-41 convergent across 5 reviewers; Takumi P2 with reproduction (six concurrent calls on a real tailnet.Conn: no_node=1 diagnostics_timeout=5, read time 166-674us), others P3. Upward case: the description tells operators a rise in diagnostics_timeout is evidence the engine lock was held, and concurrent failures are the normal shape of an incident, so the label produces false magicsock evidence and drops the peer-state split for those requests. Kept P2. Posted as a new finding on the CRF-10 thread because the CRF-10 fix introduced it.
  • CRF-8 re-raised at P2 (Pariston P2, Mafuuu P3, both inferred from wireguard-go timers). Upward case: the first failures after an active agent dies fall inside the 180s window and get handshake_ok, which the description reads as "session live, failure above WireGuard", the same misdirection CRF-8 raised. Kept P2. Leorio's Help-meaning gap is a separate finding (CRF-42) because it is a new line, not the original misclassification.
  • CRF-2 re-raised at P4 for the PTY WARN half only (Luffy P4). Gon P3 asked to stop counting client_canceled or rename the metric. Keep-argument: the metric name claims an agent failure and a sum(rate(...)) alert fires on tab closes. Downgrade reason: the R1 review itself proposed the client_canceled reason in this counter, the author applied it, and the Help text defines the value; reversing it now is churn caused by the review. The re-raise mentions that alerts must exclude the value.
  • CRF-49: Gon P2, Leorio P4, same comment. Upward case: the comment describes a report path that does not exist, and R1 rated a false comment P2 (CRF-13). Kept P2.
  • CRF-50, CRF-51 (Gon P2 comment length) dropped; keep-argument for each: concise rule. Drop reasons in the table.
  • CRF-52 dropped. Keep-argument: operator advice in a struct comment, and "failed request" is loose. Drop reason: the alerting sentence is the R1 suggested text the author applied, and Leorio R2 cites the comment as the model for this PR; changing it again is churn.
  • CRF-54 dropped. Keep-argument: Help and generated docs go stale if a constant changes. Drop reason: the proposed fix breaks docs generation (scanner cannot resolve fmt.Sprintf).
  • CRF-55 dropped. Keep-argument: a future errors.AsType caller on the proxy path would not find the fields. Drop reason: the only consumer reads theErr.Error(), and the fields are already on the request log line.
  • CRF-46: Kite P4, Melody Note. Melody checked Desktop: macOS maybeDate treats both values as nil; Windows TrayWindowViewModel.cs:229 treats year 1 as "connecting". Kept P4 because it asks for a change in vpn/tunnel.go, whose output this PR changes.
  • CRF-44 fix touches chatd (not in the PR); kept in scope because the defect is the PR's 5s heuristic, and the fix pairs a chatd cause with a check in AgentConn.
  • Interactions: CRF-41 and CRF-47 (the skip path and the untested gate release); CRF-36 and CRF-56 (dialState tracks this request's dial, not whether it got a connection; Ryosuke's GotConn trace addresses both); CRF-8 re-raise and CRF-42 (the Help text must not call handshake_ok "live").

Event: COMMENT (no P0/P1 this round; the R1 REQUEST_CHANGES stays until the remaining findings are resolved).

Reviewed against 6c6b6fe..426bc02.

Post result: review 5368597034, 12 inline comments and 3 replies (CRF-41 on the CRF-10 thread, CRF-8 and CRF-2 re-raises). post folded CRF-45 (tailnet/configmaps.go:652, not on the RIGHT side of the diff) into the review body. All three unresolveReviewThread calls failed with "Resource not accessible by integration", so those threads are still marked resolved on GitHub although their findings are open here.

Round 3 update

BLOCKED. 3 of 16 open findings silent (CRF-2 re-raise, CRF-41, CRF-45); 13 addressed (commits f7bec69..6dcae29). No Netero, no panel, no review of the new commits. Reviewed against: 46265fc..7a208ae (base moved: the branch merged origin/main).

The three silent findings were hard for the author to see: CRF-2 and CRF-41 were R2 replies on threads the author had already resolved, and the R2 unresolveReviewThread calls failed, so both threads still show Resolved; CRF-45 was folded into the R2 review body by post because tailnet/configmaps.go:652 is not on the diff. Verified at head: coderd/tailnet.go:548 still defaults reason to diagnostics_timeout and :561-562 only adds peer_diagnostics_skipped; coderd/workspaceapps/proxy.go:813-814 still logs WARN for any *AgentUnreachableError, including client_canceled; no Lost field or peer_lost log field exists.

CRF-8 closure is the Help-text option the R2 re-raise offered; the next panel should confirm the PR description's triage line for handshake_ok ("the WireGuard session is live") matches the new Help text.

The author asked in IC_kwDOGkVX1s8AAAABYLvmDQ to drop the request-changes review if nothing blocks. The R1 REQUEST_CHANGES stays: the skill allows dismissal only when all findings are resolved. The body says so and says the R1 P1s are fixed. Advisor consulted on BLOCKED vs override; recommended following BLOCKED.

Post result: review 5370891908 (COMMENT, body only).

Round 4

Churn guard: PROCEED; CRF-2, CRF-41, CRF-45 addressed in bc12263. Netero (advisory, post-panel): CRF-60 to CRF-63. Law ran (effective +1150, above 1000 and +905 since the R1 analysis at 245): Don't split, Advisory.

Round type: restructure. Five files joined the PR since the R2 panel (coderd/x/chatd/dialvalidation.go and its test, codersdk/workspacesdk/workspacesdk.go, vpn/tunnel.go and its test), and peerDiagnostics and recordAgentUnreachable were rewritten. Full panel of 13: specialists ging-go, killua, takumi, ryosuke, melody, luffy; core pariston, mafuuu; bisky (tests changed); gon, leorio (restructure floor); kite (pool: chatd, vpn, SDK in one PR); wildcard knov (shuf draw). Above the target because of the required specialists, core, bisky, and the floor. No contested findings, so all agents were new.

Orchestrator checks: pinned coder/wireguard-go device/constants.go:17,22 has RekeyAfterTime 120s and RejectAfterTime 180s (CRF-8). meta/pr.json at setup still has the description line "reason=handshake_ok means the WireGuard session is live and the failure is above it" and the Cost bullet "report diagnostics_timeout at once" (CRF-8, CRF-61). createworkspace.go:34-42,638-643 retries 5s attempts for 2m with no status check; agentConnFn is api.agentProvider.AgentConn (coderd.go:955, chatd.go:3815,3824); the five workspaceagents.go AgentConn callers check WorkspaceAgentConnected first (CRF-64). tasks.go:764-767 logs a chatd dial failure with only slog.Error(err); chatd dialTimeout is 30s (chatd.go:78), so its cancel-cause timeout is never client_canceled (CRF-65). configmaps.go:519-529,585-605,737-747 confirm DISCONNECTED deletes the peer and the lost timer is lostTimeout - since(lastHandshake), re-armed on handshake (CRF-68, CRF-69). TestReportProxyDialFailure_PooledConn (tailnet_internal_test.go:197-223) uses an uncanceled httptest context and a plain error, so tailnet.go:458 returns before hadConn matters (CRF-67). go list -deps ./coderd/workspaceapps includes coderd/x/chatd/chattool, and ErrSkipClose is acted on in workspacesdk/agentconn.go:193, so it is not a precedent for CRF-76 either way.

Cross-check:

  • CRF-8 re-raised: Pariston P2, Melody P3, Luffy P3, Leorio P3, Knov P3. Gon's Note says the Help text and the description agree; that is true of the two texts and does not address whether the peer answered, so it does not contradict the finding. Keep-argument for P2: the misdirection R2 rated P2 is unchanged, since the R2 fix option asked the Help to say handshake_ok means only a handshake under 180s old, facf0b6 added the conclusion, and the description line is unchanged. Downward case: the Help now states the 180s measurement, and a persistent incident moves to handshake_stale within 180s. Disagreement, higher wins: kept P2. Posted as a reply on the CRF-8 thread with unresolve, and named in the body with the thread URL because R2 replies on resolved threads were missed.
  • CRF-64: Melody P3, Knov P3, sibling of CRF-44 (P3). Upward case: P2 like CRF-41, a label producing false evidence. Kept P3: it adds a no_node baseline during normal chat workspace creation, the same class as CRF-44, and CRF-41 was false evidence during incidents. Posted on Knov's tailnet.go:490 (the check the cause fix pairs with) rather than Melody's createworkspace.go:641, which is outside the diff.
  • CRF-65 (Ryosuke P3) interacts with CRF-64: a warn line per counted chatd failure fires on each boot attempt until CRF-64 is fixed; the comment says so. Ryosuke's broader alternative (ServerTailnet always logs its own line and AgentUnreachableError goes away) is recorded here, not posted. Removed Ryosuke's "no test covers the nil request logger branch" from the comment: internal tests call recordAgentUnreachable on contexts without a request logger.
  • CRF-66: Takumi P4 with a scratch-test reproduction; needs the app to close a pooled connection and the agent to stop answering before the retry dial.
  • CRF-67: Bisky P3 with a mutation run; same class as CRF-60 (Netero P3).
  • CRF-68: Gon P3, Mafuuu P4, Kite P4; Leorio's Note calls the Lost comment accurate without tracing setLostTimer. Keep-argument for P3: a reader expects peer_lost for 15 minutes after LOST, but a lost peer with an old or no handshake is removed at once and reads no_node. Downgrade reason: the comment is right for a lost peer that handshook recently, it changes no code path, operators read the Help text (CRF-69) rather than this comment, and the R1/R2 P2 comment findings (CRF-13, CRF-49) described behavior that did not exist at all. Set P4.
  • CRF-69 merges Mafuuu P4 (no_node Help), Kite P4 (no_node and peer_lost scope), Leorio P4 (whose stream ended).
  • CRF-72 dropped. Keep-argument: two adjacent comments state the engine-lock fact, and the shared budget between slot wait and read is what a reader misses. Drop reason: lines 577-578 already say other callers wait for the slot "within the same timeout".
  • CRF-73 dropped. Keep-argument: the text says the agent is unreachable in the one case the code classifies as the client leaving, and it lands in the proxy error chain and the PTY debug log. Drop reason: its readers are a debug log line and a 502 page or websocket close sent to a client that already left, and the request log line for the same failure carries reason=client_canceled.
  • CRF-74 dropped. Keep-argument: three identifiers for one flag. Drop reason: field, setter, and getter is the common Go shape and each name is clear.
  • CRF-75 dropped. Keep-argument: nil guards cover a state the director never produces. Drop reason: a new type and key for no behavior change; both values are set together in director (tailnet.go:301-313).
  • CRF-76 dropped. Keep-argument: external Go programs import codersdk, and removing an exported name later breaks them. Drop reason: the proposed home cannot be imported by chattool, where the CRF-64 cause would be set; workspacesdk is the package chatd, chattool, and coderd share. Interaction with CRF-64 recorded.
  • Verification without findings: ging-go none; killua none (one slot keeps at most one goroutine blocked on the engine lock); takumi Note, peerDiagnostics timing converges; kite ran go test -race ./coderd/ -run 'TestRecordAgentUnreachable|TestReportProxyDialFailure|TestUnreachable|TestServerTailnet' and go test ./tailnet/ -run TestConfigMaps_fillPeerDiagnostics, both passed, and confirmed TestServerTailnet_AgentConn_Abandoned still guards the CRF-44 check; bisky confirmed the other new tests catch their regressions by mutation.

Event: COMMENT (no P0/P1). The R1 REQUEST_CHANGES stays: CRF-8 (P2 re-raise) and the new findings are open, and the skill dismisses only when all findings are resolved.

Reviewed against 46265fc..bc12263.

Post result: review 5371572377, 10 inline comments and 1 reply (CRF-8 re-raise). The unresolveReviewThread call failed again with "Resource not accessible by integration", so the CRF-8 thread still shows Resolved; the body links it.

Round 5

Churn guard: PROCEED; 11 of 11 open findings addressed (dfd13e2, 60da3a2, PR description). The author filed the two out-of-scope items as PLAT-696 (CRF-6) and PLAT-697 (CRF-26); both now Deferred. Netero (advisory, post-panel): CRF-78 to CRF-81 and a CRF-42 re-raise at P4. Law skipped (effective +1282, +132 since the R4 analysis).

Inventory validation: dropped items re-checked against 60da3a2 and dfd13e2. CRF-54 (Help hard-codes numbers) is moot, the Help has none. CRF-73, CRF-75, CRF-76 unchanged in assumption; CRF-76's drop reason (workspacesdk is the shared package) also covers the new ErrReadinessProbeTimeout. No reopenings.

Round type: fix round. The two commits only address R4 findings: no file was created, renamed, or moved; createworkspace.go joined the diff with a one-line fix for CRF-64; the new exported ErrReadinessProbeTimeout adds a name without changing a signature; effective additions grew 132; the rewritten description keeps the same goal. Panel: raisers of findings the commits touch (pariston, melody, luffy, leorio, knov for CRF-8; ryosuke CRF-65; takumi CRF-66; bisky CRF-67; gon, mafuuu, kite CRF-68 to CRF-71), specialists for the files changed since R4 (ging-go; takumi, ryosuke, melody already in), komugi from the pool (the CRF-60 fix is a mock-clock test), wildcard razor (shuf draw from meruem, chopper, zoro, razor). 14 reviewers, all new agents (no contested findings).

Orchestrator checks: context.md description still has "The last handshake is under 180 seconds old, so the tunnel is up. | Above WireGuard" and "M1 with handshake_ok means the tunnel is fine", although reply PRRC_kwDOGkVX1s73XEX_ says the description was changed (CRF-8). The description's metrics table says derp_connects "Reads 1 after startup" with no disabled-relay case (CRF-3). Go 1.26.5 net/http/transport.go:1516-1565,1639-1648: GetConn fires on the request goroutine at the start of getConn, the dial goroutine reads getCtxForDial and then calls dialConn, and an idle-conn delivery (tryPutIdleConn, :1103-1110) does not cancel the dial; cancelCtx runs only from CloseIdleConnections (:912). So startDial can run after GotConn (CRF-79). startworkspace.go:128-132 calls waitForAgentAndRespond for a succeeded start build with no status check, while createworkspace.go:497-512 passes nil for a connected agent (CRF-83). WithDeadlineCause returns WithCancel(parent) when the parent deadline is earlier, so the last attempt of the 2m window carries DeadlineExceeded and is counted (CRF-83 severity). MCP: coderd/mcp/mcp.go:49-55 is stateless, go-sdk v1.8.0 streamable.go:700-703 passes req.Context(), jsonrpc2/conn.go:231-233 wraps it in notDone (values kept), mcp.go:164-167 turns tool errors into IsError results (HTTP 200), and loggermw/logger.go:182-186 logs non-5xx at debug (CRF-84). newTestServerTailnet uses quartz.NewReal(), so TestRecordAgentUnreachable_DiagnosticsSlot waits two real 1s timers (CRF-87). CI runs ./coderd only on Linux; macOS and Windows jobs run ./cli/... ./enterprise/cli/... ./agent/... (ci.yaml), so CRF-92 is a local-run risk. Em dashes: 7 in createworkspace.go, 1 each in dialvalidation_internal_test.go and workspacesdk.go, none on added lines (CRF-81).

Cross-check:

  • CRF-8 re-raised at P3 (Pariston P3, Leorio P3). Keep-argument for P2 (the R4 severity): the description is now the only definition of the reason values, it lands in the squash commit, and the author's reply says it was fixed when it was not. Downgrade reason: the Help text and generated docs no longer carry the conclusion, and both reviewers that re-raised chose P3. Leorio's replacement row and diagram sentence are in the comment.
  • CRF-42 re-raised at P3 (Leorio P3, Netero P4). Higher wins. Churn check: R2 asked for definitions, R4 asked to correct two of them and remove one conclusion, and 60da3a2 removed all of them; the re-raise gives exact replacement text with no conclusion for handshake_ok, following the chatloop/metrics.go:179 form.
  • CRF-3 re-raised at P4 (Leorio). Same class: 60da3a2 removed the qualifier that fixed CRF-3, and the description's "Reads 1 after startup" is the text CRF-3 called false.
  • CRF-82: six reviewers (Knov P3, four P4, Netero Note), all with the same mutation (revert line 640, chattool tests pass). Higher wins; P3 matches CRF-60 and CRF-67, tests that do not guard a fix. Interaction with CRF-83: if the author replaces the cause with a status check, the test should cover the status check instead.
  • CRF-83: Kite P3, Melody, Mafuuu, Knov, Luffy P4. Keep-argument for P3: the CRF-64 fix hides failures on the connected-but-unreachable agent, the PLAT-360 state. Downgrade reason: Kite's "neither counted nor logged" is contradicted by the last-attempt analysis four reviewers made and the orchestrator confirmed from WithDeadlineCause: each start_workspace call still gives one count and one warn line with fields, so the incident stays visible; app and PTY paths are unaffected. Set P4. Luffy's alternative (check agent status before each attempt, remove the sentinel) is folded into the fix text.
  • CRF-84: Mafuuu P3, Kite P4, Razor P4. Melody's walk said every HTTP caller including mcp_http.go writes a 500; mcp.go:164-167 shows MCP returns a 200 tool error, so Melody's statement is wrong for MCP. Kept P3, same class as CRF-65 (P3). MCP HTTP is behind the mcp-server-http experiment; chatd, the CRF-65 caller, is under coderd/x as well. Interaction with CRF-85: if recordAgentUnreachable always writes its own warn line, its doc changes with it.
  • CRF-85: Gon P3 set to P4. Keep-argument: a caller that follows "Callers that log the failure should include them" writes a second warn line, and tasks.go:766 already writes one without the fields (Melody Note). Downgrade reason: comment accuracy with no code path affected, same as CRF-68 (P4).
  • CRF-86: Razor P4 set to Nit. Keep-argument: the doc is the AgentProvider contract and this PR wrote the exception list. Downgrade reason: the two omitted cases return a plain error only to the caller that set the cause, so no caller misreads its own result; same class as CRF-71 (Nit).
  • CRF-78: Netero P3 set to P4. Keep-argument: repo guidance (TESTING.md:27) points to quartz, and a struct whose clock field half the code ignores invites tests that mix mock and real time. Downgrade reason: no production behavior changes, the tests pass after and now explicitly, and the one concrete test consequence (CRF-87) is its own finding.
  • CRF-79: Netero P4, Takumi confirms from the transport source with the same GetConn fix; Gon's Note (the startDial comment states the false ordering) is folded into the comment.
  • CRF-88: Gon P4 set to Nit. Keep-argument: a reader who trusts the comment could restore a connection error and lose the mutation coverage CRF-67 added. Downgrade reason: the comment describes the scenario, cancel() and context.Canceled are on the next lines, and changing the input needs a code edit that the test's require.Same would show.
  • CRF-91: Komugi P3 as a re-raise of CRF-41, recorded as Note (body). Keep-argument: in a burst, queued failures are labeled diagnostics_timeout, which the description reads as a stuck engine, and a coordinator-stream loss is such a burst. Downgrade reason: R3 offered "let concurrent callers wait up to 1s for the in-flight read" and the author applied exactly that; the residual needs more failures in one second than 1s divided by the read time (about 400 to 1250 at Komugi's measured 0.8 to 2.4ms), and Komugi's 26 of 50 used a 40ms stub. Re-raising the review's own option is churn.
  • CRF-92: Komugi P4 set to Note (body). Keep-argument: local macOS runs can take port 65000. Downgrade reason: CI runs ./coderd only on Linux, where 65000 is outside the ephemeral range, and no cheap fix was offered.
  • CRF-93 dropped (Luffy P3). Keep-argument: seven findings in four rounds (CRF-9, 36, 37, 49, 66, 67, 79) came from dialState and reportProxyDialFailure, and a dial-level record removes that code. Drop reason: the R1 review's CRF-7 fix text asked for this design, the PR description documents the fields on the 502 line, and the alternative records up to proxyDialTimeout after the 502 and does not count a request whose dial succeeds after the request ended.
  • CRF-90 (Takumi Note): recorded as Note (body); a fix needs a per-dial record chosen at GetConn.

Event: COMMENT (no P0, P1, or P2). The R1 REQUEST_CHANGES stays: the skill dismisses it only when all findings are resolved.

Reviewed against 0d12256..39a12bf.

Post result: review 5373339953, 10 inline comments and 3 replies (CRF-8, CRF-42, CRF-3 re-raises). All three unresolveReviewThread calls failed with "Resource not accessible by integration"; the body links each thread.

Round 6

Churn guard: PROCEED; 13 of 13 open findings addressed (b7f4705, 8fde7df, PR description). Author also answered the R5 body notes: CRF-90 deferred to PLAT-698, CRF-91 accepted (entry in Contested and acknowledged), CRF-92 fixed with testutil.RandomPort, CRF-81 (OOS em dashes) left because the lines are not changed. Netero (advisory): CRF-95 to CRF-98. Law skipped (effective +1360, +210 since the R4 analysis). The author also commented "Please limit the review to P0, P1, and P2 issues only" (IC_kwDOGkVX1s8AAAABbqx37A).

Inventory validation: CRF-93 (Luffy R5 proxy restructure) unchanged in assumption. CRF-76 drop reason (workspacesdk is the shared package) still holds. No reopenings from the drop list; CRF-84's own premise is disproved (below).

Round type: fix round. b7f4705 patches existing functions; no file created, renamed, or moved (createworkspace_internal_test.go and coderd/mcp_http.go existed before and joined the diff); no exported signature changed; effective additions +78 since R5. Panel: every persona that raised a finding the commits touch (pariston, leorio for CRF-8/42/3; takumi, gon for CRF-79; knov, ryosuke, melody, bisky, kite for CRF-82; mafuuu, luffy for CRF-83; razor for CRF-84/86; komugi for CRF-87; gon for CRF-85/88/89), ging-go for the Go files changed since R5, wildcard meruem (shuf draw from meruem, chopper, zoro). 15 reviewers, all new agents (no contested findings).

Orchestrator checks: codersdk/toolsdk/toolsdk.go:280 wraps every tool handler in WithCleanContext, and :367-386 builds the child from context.Background(), copying only deadline and cancellation; openAgentConn (toolsdk.go:147) calls the dial function with that child, so loggermw.RequestLoggerFromContext is nil in the MCP dial and recordAgentUnreachable already writes the warn line (CRF-99; Takumi and Razor each confirmed with a scratch test). In R5 the orchestrator checked only the go-sdk side (streamable.go, jsonrpc2/conn.go) and missed the toolsdk wrapper, so CRF-84 was posted on a false premise. createworkspace.go:381 has AgentInactiveDisconnectTimeout in the create options; startworkspace.go does not (CRF-100 fix needs it plumbed). tailnet.go:520 observes await_reachable_seconds on every successful AwaitReachable, with no cause check (CRF-101). The CLI accepts dismiss: true with a COMMENT event that has inline comments (internal/review/post.go:348-355).

Cross-check:

  • CRF-99: Takumi P3, Razor P4, both verified by scratch test. Keep-argument for P3 (higher wins): every MCP failure writes two identical warn lines, which doubles the per-agent counts the description tells operators to compare, and the review caused it. Downward case: logs only, mcp-server-http is an experiment, no metric changes. Kept P3. Several Notes (Mafuuu, Kite, Ryosuke, Melody, Pariston, Leorio, Meruem) assume the MCP context has a request logger; that premise is wrong, so they carry no weight on this point. CRF-96 (no test for the MCP warn line) folded in: the fix should add a test that asserts one warn line. Posted inline at mcp_http.go:80 instead of a reply on the CRF-84 thread, which is resolved and cannot be reopened.
  • CRF-100: Melody, Knov, Kite, Luffy all P4, Komugi Note. Upward case: like CRF-64 (P3), it puts about 24 no_node/peer_lost counts and warn lines per start_workspace call into M1 for an agent the app path counts in agent_offline_total. Kept P4: it needs a workspace whose succeeded start build has a dead agent, far less common than every chat workspace boot. Not churn: R5 offered "set the cause only while the agent's status is not connected, or check the status before each attempt", and the author used FirstConnectedAt, which is neither. Luffy's second case (parent deadline gives the last attempt DeadlineExceeded) was already recorded in R5 as intended.
  • CRF-101: Mafuuu P4. Interaction with CRF-100: dialing only after the agent shows connected removes most boot time from the successful sample (inferred). Posted at Mafuuu's line 519 (the Observe call is line 520).
  • CRF-102: Gon P4 set to Nit. Keep-argument: a reader of reportProxyDialFailure expects a retried request that once had a pooled connection to return early. Downgrade reason: one word ("request" for "attempt"), and the director comment 85 lines above states the per-attempt reset; comment wording is Nit as in CRF-88.
  • CRF-95: Netero P4, TESTING.md:31-33 rule. Interaction with CRF-100: a status-check poll may remove the 5s wait.
  • CRF-97 re-raised by Ryosuke P4 and Meruem P4 (proxy-path tests cannot cross the 5s client_canceled boundary with a mock clock). Keep-argument: sibling of CRF-78 (P4, fixed), and one decision reads two clocks. Drop reason: no production behavior differs, the author's R5 reply on the CRF-78 thread states the choice, and no current test needs it. Stays dropped.
  • CRF-103 (Gon P4), CRF-104 (Gon Nit), CRF-105 (Leorio P4), CRF-106 (Meruem P3) dropped; keep-arguments: CRF-103 "clears the connection" could read as closing the pooled connection; CRF-104 two verbs for one mechanism; CRF-105 the squash body would not mention the MCP line; CRF-106 the contract that non-5xx callers must re-log is enforced only by a doc comment, and the review itself got it wrong once. Drop reasons in the table.
  • CRF-107 (Takumi, Razor OOS): toolsdk is not changed by the PR.
  • Author request for P0-P2 only: the skill posts every in-scope finding, and PR comments do not override the operator's workflow. Advisor consulted: post all validated findings after removing unsupported ones, and say plainly that none is P0 to P2. Posted 1 P3, 3 P4, 1 Nit.
  • Dismissal: every finding raised in rounds 1 to 5 is resolved (fixed, deferred with a ticket, or accepted), which is the condition the R3 to R5 bodies gave the author for dismissing the R1 REQUEST_CHANGES, and the R1 P1s were fixed in R2. This round's findings are P3 and below, which the event rule does not treat as blocking. The skill's text ("re-review rounds where all findings are resolved") can also be read as "no open findings at all"; under that reading the review would stay forever while each round adds P4s, which contradicts "REQUEST_CHANGES for P0-P1". Advisor: dismiss if the condition refers to earlier findings. Decision: COMMENT with dismiss: true.

Reviewed against adcce66..c1fd7e9.

About deep-review

CRF = Coder Review Finding (P0-P4, Nit, Note)

Reviewer Focus
Bisky tests
Chopper ops/errors
Churn-guard change verification
Ging language modernization
Gon naming
Hisoka edge cases
Killua perf
Kite change integrity
Knov contracts
Knuckle SQL
Komugi flake/determinism
Kurapika security
Law decomposition
Leorio docs
Luffy product
Mafu-san process
Mafuuu contracts
Melody dispatch/pairing
Meruem structural
Nami frontend
Netero mechanical checks
Pariston premise testing
Pen-botter product gaps
Razor verification
Robin duplication
Ryosuke Go arch
Takumi concurrency
Zoro shape

🤖 Managed by Coder Agents.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

coder-agents-review[bot]
coder-agents-review Bot suggested changes Sep 29, 2026
View reviewed changes

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds peer diagnostics to ServerTailnet unreachable-agent failures, four Prometheus metrics, and TxBytes/RxBytes on tailnet.PeerDiagnostics. Findings: 2 P1, 1 P2, 8 P3, 6 P4, 3 Nit. Because of the two P1 findings, the 502 line and the reason label do not yet separate the PLAT-360 cases the description lists.

Out of scope (needs a ticket or explicit acceptance by a human):

  • coderd/tailnet.go:298: dialContext runs AwaitReachable on the transport's detached dial context with no bound, so each app request to an unreachable agent leaves a dial goroutine, its ping goroutines, and a coordinator ticket in place until the agent answers or coderd shuts down.
  • enterprise/wsproxy/wsproxy.go:300: the workspace proxy's ServerTailnet is never registered with Prometheus, so proxies export none of the coder_servertailnet_* metrics, including the three this PR adds.
  • coderd/workspaceapps/proxy.go:93: AgentUnreachableError.Error() keeps the old text "agent is unreachable", so PTY and agent API users see no wait time or next step.
  • cli/cliui/agent.go:459: the same IsZero check on LastWireguardHandshake, so for a peer that is in the WireGuard device but never completed a handshake, the CLI diagnostics print a handshake time from 1970 instead of "Wireguard is not connected" (inferred from the code, not run).

Notes:

  • tailnet/conn.go:969: TxBytes and RxBytes also appear in support bundles, because support.Bundle serializes tailnet.PeerDiagnostics into agent/peer_diagnostics.json without JSON tags; no code in the repository parses that file.

🤖 This review was automatically generated with Coder Agents.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread coderd/tailnet.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/tailnet.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/tailnet.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/tailnet.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/tailnet.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/tailnet.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/tailnet.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/tailnet.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/tailnet_test.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/workspaceapps/proxy.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

BobbyHo added 8 commits September 29, 2026 17:19
@BobbyHo
fix(coderd): report unreachable agents from the app proxy error handler
65ba780
The HTTP transport keeps dialing after the request ends, so the 502 was
written before AgentConn returned. The request log line had no peer
diagnostics and agent_unreachable_total only moved at shutdown.

Report the failure from ReverseProxy's ErrorHandler instead, using a
dial state shared through the request context. Bound the dial so an
abandoned one stops pinging, and add a test through ReverseProxy.
@BobbyHo
fix: count never-handshaked and stale peers as unreachable reasons
568c5e7
WireGuard reports a peer with no handshake as the Unix epoch, which
IsZero does not catch, so every known peer was counted as handshake_ok.
Peers that worked earlier kept their old handshake time and were counted
the same way.

Report no handshake as the zero time in fillPeerDiagnostics, add a
handshake_stale reason for handshakes older than WireGuard's 180 second
session limit, and list the reason values in the metric help text.
@BobbyHo
fix: rename workspace app offline counter and correct its comment
0ce724c
Only the disconnected-agent page increments the counter; stopped
workspaces return before it. Name it agent_offline_total, say what it
adds over coderd_agents_connections, and describe the real baseline.
@BobbyHo
fix(coderd): limit peer diagnostics reads and clarify unreachable log…
cb69c0e
… fields

Allow one diagnostics read at a time so a held engine lock parks one
goroutine instead of one per failed request. Name the server's relay
server_preferred_derp, log the agent's relay from its node, omit the
handshake time when there was none, and rename the recording function.
Cover the node-received fields and the busy gate with unit tests.
@BobbyHo
fix(coderd): do not count quick client cancellations as unreachable a…
9b1b1a2
…gents

A context canceled within five seconds of the dial is the client leaving,
not the agent failing. Count it under client_canceled without reading
peer diagnostics. Longer waits count as unreachable however the context
ended, since a load balancer giving up also cancels the request. Add the
reason to the log line.
@BobbyHo
docs(tailnet): say the peer byte counters are cumulative
74e9d79
@BobbyHo
fix(coderd): rename the DERP counter and tighten the success test
6567d0a
The counter counts the first connection too and stays 0 when the
embedded relay is disabled. Name it derp_connects_total and say so in
the help text. Assert that a reachable agent leaves no series in
agent_unreachable_total.
@BobbyHo
test(coderd/workspaceapps): cover the PTY unreachable agent log line
426bc02
Use errors.AsType and say why the warn branch uses the server logger.
@BobbyHo

BobbyHo commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

/coder-agents-review

coder-agents-review[bot] reacted with rocket emoji
All reactions
  • 🚀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@BobbyHo
Merge branch 'main' into bobby/plat-680-servertailnet-observability-m…
a4edb96
…ake-agent-unreachable-failures
coder-agents-review[bot]
coder-agents-review Bot reviewed Sep 30, 2026
View reviewed changes

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 2. 18 of 20 round-1 findings are fixed, including the proxy-path P1 (the ErrorHandler now reports on the request goroutine). Two are partially fixed and re-raised on their threads: a dead peer within 180s of its last handshake still counts as handshake_ok (P2; the 1970 value is fixed), and a PTY client_canceled still logs at WARN (P4). New findings: 2 P2, 6 P3, 4 P4, 2 Nit. The diagnostics_timeout finding comes from the single-read gate, so it is posted on that thread.

Out of scope (needs a ticket or explicit acceptance by a human), still open from round 1:

  • enterprise/wsproxy/wsproxy.go:300: the workspace proxy's ServerTailnet is never registered with Prometheus, so proxies export none of the coder_servertailnet_* metrics.
  • coderd/workspaceapps/proxy.go:93: AgentUnreachableError.Error() keeps the text "agent is unreachable", so PTY and agent API users see no wait time or next step.

Notes:

  • coderd/tailnet.go:414: a request that reuses a pooled connection to an agent that stopped answering gets a 502 with no agent_id or reason and no count, because its dialState reads dialPhaseNone. With MaxIdleConnsPerHost = 6, reused connections carry the first failures on an agent in active use, so on the app path a 502 without agent_id means a reused connection failed.
  • coderd/tailnet.go:308: the one-minute proxyDialTimeout turns an app request that waits more than 60s for the first ping into a 502 where it used to keep waiting, which operators may see as new 502s after upgrading.

tailnet/configmaps.go:652

P3 [CRF-45] PeerDiagnostics does not report the peer's lost flag, so for up to 15 minutes a lost coordinator connection produces the same reasons as a WireGuard failure. (Mafuuu)

configMaps sets lc.lost = true when the coordinator sends LOST for the agent (tailnet/configmaps.go:526), and SetAllPeersLost runs when ServerTailnet's own coordination stream ends (tailnet/controllers.go:335); lost peers keep their node and WireGuard config for lostTimeout (15 minutes), so failures in that window read no_handshake, handshake_stale, or handshake_ok, not no_node. The description maps handshake_stale with large byte counters to the PLAT-360 signature, so a dropped coordination stream reads as that signature. Fix: add Lost bool to PeerDiagnostics in fillPeerDiagnostics, which this PR already changes, and log it as peer_lost.

🤖

🤖 This review was automatically generated with Coder Agents.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread coderd/tailnet.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/tailnet.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/workspaceapps/proxy_internal_test.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/tailnet_test.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/tailnet.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread tailnet/configmaps.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/tailnet.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/tailnet.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/tailnet.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/workspaceapps/db.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

BobbyHo added 9 commits September 30, 2026 10:26
@BobbyHo
fix(coderd): do not count pooled connection errors as unreachable agents
f7bec69
The transport can give a request a pooled connection while the dial it
started is still waiting on the agent. An error on that connection then
reached reportProxyDialFailure with the dial phase still at
await_reachable and was counted as an unreachable agent.

Install a GotConn trace in the director and return early from the error
handler once it has fired.
@BobbyHo
test(coderd): cover the tcp_dial and dial timeout branches of the pro…
ed06d32
…xy error handler

A registered agent with no listener on the target port exercises the
tcp_dial branch, which names the phase on the log line without counting
an unreachable agent. A unit test covers a dial that already returned
errAgentUnreachable, which must not be prefixed a second time.
@BobbyHo
fix: do not count a dial chatd abandoned as an unreachable agent
70b54f8
chatd cancels its dial to a cached agent when validation finds the
workspace has a newer one. That cancel arrived after clientCanceledWait
and was counted in agent_unreachable_total, usually as no_node.

Cancel the abandoned dial with workspacesdk.ErrDialAbandoned as the
cause and skip the count in AgentConn when that cause is set.
@BobbyHo
docs(coderd): say what the proxy error handler reports about a dial
f33df09
The old comment implied a dial that fails after its request ended is
reported later. Only the phase at the time the request ends is recorded.
@BobbyHo
fix(coderd): count only agents that did not answer on the proxy path
dcad2e9
A request still waiting on its dial can get an error from before the
first ping, such as a failed coordinator send. AgentConn does not count
those, so the proxy error handler now skips them too.

Also prove the peer diagnostics gate is released after a read and that
the timeout branch reports diagnostics_timeout, by dialing a second
unknown agent and by injecting a blocking read in an internal test.
@BobbyHo
fix(vpn): leave last_handshake unset for a peer with no handshake
bc30966
The zero time now returned for a never-handshaked peer would encode as
year 1, which the Windows client shows as connecting. The proto
documents an unset value as no handshake, so send that instead.
@BobbyHo
docs: explain each agent_unreachable_total reason and the await_reach…
facf0b6
…able histogram

The Help text listed six reason values but explained two. It now has
one clause per value, and the histogram names the question it answers.
@BobbyHo
chore(coderd): reuse testutil.WaitBuffer and rename the offline count…
6dcae29
…er field

The PTY test had its own locked buffer, the agent_offline_total field
kept its old name, and a test comment still described the DERP counter
as process-wide.
@BobbyHo
Merge remote-tracking branch 'origin/main' into bobby/plat-680-server…
7a208ae
…tailnet-observability-make-agent-unreachable-failures

# Conflicts:
#	docs/admin/integrations/prometheus.md
@BobbyHo

BobbyHo commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

/coder-agents-review review the latest changes to see if there are more P0 to P2 issues. Please drop the request for change label if there are no block issues.

coder-agents-review[bot] reacted with rocket emoji
All reactions
  • 🚀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

coder-agents-review[bot]
coder-agents-review Bot reviewed Sep 30, 2026
View reviewed changes

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

13 of the 16 findings open after round 2 are fixed in f7bec69 through 6dcae29, and the round-1 P1s that prompted the request-changes review are fixed. Three findings have no reply or code change. Two were easy to miss: the first and third below are replies on threads you had already resolved (reopening them failed on our side, so they still show Resolved), and the second was in the round-2 review body, not inline.

  • P2, skipped diagnostics reads counted as diagnostics_timeout (#30133 (comment)): recordAgentUnreachable still defaults reason to diagnostics_timeout (coderd/tailnet.go:548) and the skipped case only adds peer_diagnostics_skipped, so failures that overlap an ordinary read rise under the label the description reads as a held engine lock. Count skipped reads under their own reason, or let concurrent callers wait up to 1s for the in-flight read.
  • P3, PeerDiagnostics has no lost flag (#30133 (review)): for up to 15 minutes after the coordinator sends LOST, or after ServerTailnet's coordination stream ends, failures read no_handshake, handshake_stale, or handshake_ok instead of pointing at the control plane. Add Lost in fillPeerDiagnostics and log it as peer_lost.
  • P4, PTY logs client_canceled at WARN (#30133 (comment)): workspaceAgentPTY still logs WARN for any *AgentUnreachableError (coderd/workspaceapps/proxy.go:813), including a client that left within 5s. Return an untyped error for client_canceled.

For each one, a fix, a linked issue, or a reply with the reason to keep the current behavior is enough. Review of the latest commits is on hold until then; this round did not review them.

The round-1 request-changes review (a review state, not a repository label) stays for now because this workflow dismisses it only after every earlier finding is resolved, not because the three findings above meet the blocking threshold.

Out of scope (needs a ticket or explicit acceptance by a human), still open from round 1:

  • enterprise/wsproxy/wsproxy.go:300: the workspace proxy's ServerTailnet is never registered with Prometheus, so proxies export none of the coder_servertailnet_* metrics.
  • coderd/workspaceapps/proxy.go:93: AgentUnreachableError.Error() keeps the text "agent is unreachable", so PTY and agent API users see no wait time or next step.

🤖 This review was automatically generated with Coder Agents.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@BobbyHo
fix: label lost peers and skipped diagnostics reads correctly in agen…
bc12263
…t_unreachable_total

Overlapping failures were counted as diagnostics_timeout even when no read
was attempted. Callers now wait up to 1s for the single read slot and read
with the remaining time, so the label only means the peer state could not
be read in time.

PeerDiagnostics gains a Lost flag. A peer the coordinator reported lost, or
one left behind by a dropped coordination stream, keeps its node and
WireGuard config for 15 minutes and could read as no_handshake or
handshake_ok. It now counts as peer_lost and the log line carries the flag.

A client that leaves within 5s is still counted as client_canceled but gets
a plain error, so the PTY handler logs it at debug like other dial errors
instead of WARN.
@BobbyHo
fix: address ServerTailnet review feedback
60da3a2
- Log a warn line for unreachable agents when there is no request
  log, such as chatd dials.
- Clear gotConn when the transport retries on a new dial.
- Use an injectable clock for the diagnostics timeout so the slot
  wait test is deterministic.
- Fix tests that passed with their checks removed.
- Shorten the ServerTailnet metric Help texts, and fix the Lost and
  AgentUnreachableError comments.
@BobbyHo

BobbyHo commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

Filed the out-of-scope items as follow-ups: PLAT-696 (proxy ServerTailnet metrics) and PLAT-697 (unreachable error message).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@BobbyHo
Merge branch 'main' into bobby/plat-680-servertailnet-observability-m…
39a12bf
…ake-agent-unreachable-failures
@BobbyHo

BobbyHo commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

/coder-agents-review review the latest changes to see if there are more P0 to P2 issues. Please drop the request for change label if there are no block issues.

coder-agents-review[bot] reacted with rocket emoji
All reactions
  • 🚀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

coder-agents-review[bot]
coder-agents-review Bot reviewed Oct 1, 2026
View reviewed changes

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds reason labels, peer diagnostics, and counters for workspace agents that ServerTailnet cannot reach. This round re-raises three findings (2 P3, 1 P4) and adds new ones: 2 P3, 5 P4, 3 Nit.

All 11 findings from round 4 have code or description changes in dfd13e2 and 60da3a2, and the two out-of-scope items are tracked in PLAT-696 and PLAT-697. Three are re-raised as replies on threads that show as resolved, because reopening them fails on our side:

  • #30133 (comment): the description's reason table and diagram still say handshake_ok means the tunnel is up.
  • #30133 (comment): the Help text now names the seven reasons without defining any of them.
  • #30133 (comment): the derp_connects_total Help text lost "stays 0 when the embedded relay is disabled", and the description says it reads 1 after startup.

No finding this round is P0, P1, or P2. The round-1 request-changes review stays until the open findings are resolved, because this workflow dismisses it only then.

Out of scope (needs a ticket or explicit acceptance by a human):

  • coderd/x/chatd/chattool/createworkspace.go:338: seven em dashes in comment lines this PR does not change, plus one each in coderd/x/chatd/dialvalidation_internal_test.go and codersdk/workspacesdk/workspacesdk.go; scripts/check_emdash.sh checks only added lines, so CI does not report them.

Notes:

  • coderd/tailnet.go:350: one dialState per request is shared by every dial the request starts, so if an earlier dial finishes while a retry dial waits on the agent, the phase reads done and the failure gets no fields and no count.
  • coderd/tailnet.go:607: a failure that times out while waiting for the diagnostics slot is labeled diagnostics_timeout, the same as a read that hung on the engine lock; at 1 to 2ms per read this takes several hundred failures within one second.
  • coderd/tailnet_test.go:344: TestServerTailnet_ReverseProxy_TCPDialFailure needs host port 65000 to be free. It is outside Linux's default ephemeral range, where CI runs ./coderd, and inside it on macOS and Windows.

🤖 This review was automatically generated with Coder Agents.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread coderd/x/chatd/chattool/createworkspace.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/x/chatd/chattool/createworkspace.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/tailnet.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/tailnet.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/workspaceapps/proxy.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/tailnet_internal_test.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/tailnet_internal_test.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/tailnet.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/tailnet.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread coderd/tailnet.go Resolved

Uh oh!

There was an error while loading. Please reload this page.

BobbyHo added 3 commits September 30, 2026 17:27
@BobbyHo
Merge branch 'main' into bobby/plat-680-servertailnet-observability-m…
1266266
…ake-agent-unreachable-failures
@BobbyHo
fix: address ServerTailnet review feedback, round 5
b7f4705
- Count chat readiness dial failures for agents that connected before
- Log unreachable agents at warn on MCP tool calls
- Use "agent is unreachable" for the PTY warn line
- Clear gotConn in a GetConn trace hook on the request goroutine
- Use the ServerTailnet clock in AgentConn and recordAgentUnreachable
- Define each reason in the agent_unreachable_total Help text
- Note that derp_connects_total stays 0 without the embedded relay
- Run the diagnostics slot test on a mock clock
- Use a free port in TestServerTailnet_ReverseProxy_TCPDialFailure
@BobbyHo
Merge remote-tracking branch 'origin/bobby/plat-680-servertailnet-obs…
bd172fa
…ervability-make-agent-unreachable-failures' into bobby/plat-680-servertailnet-observability-make-agent-unreachable-failures
@BobbyHo

BobbyHo commented Oct 1, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

On the notes:

  • Em dashes in createworkspace.go, dialvalidation_internal_test.go, and workspacesdk.go: those lines are not changed by this PR, so I left them.
  • Shared dialState across retries: GetConn now clears gotConn per attempt, but a phase set to done by an earlier dial can still hide a retry. Tracked in PLAT-698.
  • Waiting for the diagnostics slot shares the diagnostics_timeout label: accepted. Both mean the peer state was not read within 1s.
  • Port 65000 in TestServerTailnet_ReverseProxy_TCPDialFailure: fixed in b7f4705. It now uses testutil.RandomPort.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@BobbyHo
Merge branch 'main' into bobby/plat-680-servertailnet-observability-m…
e90b8c6
…ake-agent-unreachable-failures
@BobbyHo
BobbyHo marked this pull request as ready for review October 1, 2026 01:00
@BobbyHo
BobbyHo requested review from a team and Emyrk October 1, 2026 01:04
@matifali
matifali requested a review from code-asher October 1, 2026 05:11
@BobbyHo
Merge branch 'main' into bobby/plat-680-servertailnet-observability-m…
7fdf16b
…ake-agent-unreachable-failures
@BobbyHo
BobbyHo requested a review from nickvigilante as a code owner October 6, 2026 17:06
nickvigilante
nickvigilante reviewed Oct 6, 2026
View reviewed changes

@nickvigilante nickvigilante left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docs LGTM!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
BobbyHo added 2 commits October 6, 2026 19:52
@BobbyHo
test(coderd/workspaceapps): add tracing middleware to PTY unreachable…
8fde7df
… test

- workspaceAgentPTY now reads tracing.ClientSessionID, which panics
  without the tracing middleware
@BobbyHo
Merge branch 'main' into bobby/plat-680-servertailnet-observability-m…
c1fd7e9
…ake-agent-unreachable-failures
@BobbyHo

BobbyHo commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

/coder-agents-review Please limit the review to P0, P1, and P2 issues only.

coder-agents-review[bot] reacted with rocket emoji coder-agents-review[bot] reacted with eyes emoji
All reactions
  • 🚀 1 reaction
  • 👀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@BobbyHo
fix(coderd/workspaceapps): explain nolint directive in PTY test
3c6dfa1
coder-agents-review[bot]
coder-agents-review Bot reviewed Oct 13, 2026
View reviewed changes

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds reason labels, peer diagnostics, and counters for workspace agents that ServerTailnet cannot reach. This round has 1 P3, 3 P4, and 1 Nit. None is P0, P1, or P2, which is the range you asked about; this workflow posts every finding, and none of these blocks merging.

All 13 findings from round 5 are fixed in b7f4705 and the description, and the round-5 notes are answered or tracked in PLAT-698. The round-5 comment about MCP request logs was wrong: the MCP tool context has no request logger, so the warn line added for it is now a duplicate (comment on coderd/mcp_http.go:80). The round-1 request-changes review is dismissed with this review, because every finding from rounds 1 to 5 is resolved.

Out of scope (needs a ticket or explicit acceptance by a human):

  • codersdk/toolsdk/toolsdk.go:367: MCP tool dials have no deadline, and WithCleanContext keeps only the parent's cancellation, so an MCP call to an agent that is connected in the database but unreachable over the tunnel waits in AwaitReachable until the client cancels and is rarely counted.

🤖 This review was automatically generated with Coder Agents.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread coderd/mcp_http.go
}
return api.agentProvider.AgentConn(ctx, agentID)
conn, release, err := api.agentProvider.AgentConn(ctx, agentID)
if unreachable, ok := errors.AsType[*workspaceapps.AgentUnreachableError](err); ok {

@coder-agents-review coder-agents-review Bot Oct 13, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 [CRF-99] The MCP tool context has no request logger, so recordAgentUnreachable already writes the warn line, and this branch writes a second one with the same fields for every MCP failure. (Takumi, Razor)

Every toolsdk handler runs under WithCleanContext (codersdk/toolsdk/toolsdk.go:280), which builds the handler context from context.Background() (toolsdk.go:367-386), and openAgentConn dials with that context (toolsdk.go:147); a scratch test that passed a request logger through WithCleanContext found none in the handler. The round-5 comment that asked for this branch assumed the request logger reached the tool context, and that assumption was wrong. Remove lines 79-84, return api.agentProvider.AgentConn(ctx, agentID) as before, and add a test that an MCP dial failure writes exactly one warn line.

🤖

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread coderd/x/chatd/chattool/createworkspace.go
// unreachable. An agent that connected before should answer, so its
// failures are counted.
var attemptCause error
if !agent.FirstConnectedAt.Valid {

@coder-agents-review coder-agents-review Bot Oct 13, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P4 [CRF-100] The readiness poll decides from FirstConnectedAt, read once before the loop, so a start_workspace call on an agent that connected once and is now disconnected counts every 5s attempt, and a new agent that connects during the loop is not counted. (Melody, Knov, Kite, Luffy)

startworkspace.go:132 runs the poll for any succeeded start build, so an agent whose VM died adds about 24 no_node or peer_lost counts and warn lines per 2-minute call (inferred from the timers, not run), while the app path counts the same agent in agent_offline_total (workspaceapps/db.go:252-254). Re-read the agent before each attempt and dial only when its Status(now, inactiveTimeout) is connected, as createworkspace.go:494-512 does; that also removes the need for ErrReadinessProbeTimeout. StartWorkspaceOptions needs the AgentInactiveDisconnectTimeout field that CreateWorkspaceOptions has at line 70, and the test needs a case for a disconnected agent with FirstConnectedAt set.

🤖

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread coderd/tailnet.go
ret()
return nil, nil, xerrors.New("agent is unreachable")
return nil, nil, errAgentUnreachable
}

@coder-agents-review coder-agents-review Bot Oct 13, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P4 [CRF-101] await_reachable_seconds records the successful readiness probe of a booting chat agent, so agent boot time goes into the histogram the description uses as the early warning of slow reachability. (Mafuuu)

acquireAgent calls Observe on every successful AwaitReachable (line 520), and the readiness cause exists only once an attempt times out, so it cannot exclude a probe that succeeds; each chat that creates or starts a workspace adds one sample between 0 and 5s (inferred from the code). If the readiness poll dials only after the agent shows connected, most of the boot time leaves the sample; otherwise mark readiness probes with a context value and skip Observe for them.

🤖

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread coderd/x/chatd/chattool/createworkspace_internal_test.go

var cause error
connFn := func(ctx context.Context, _ uuid.UUID) (workspacesdk.AgentConn, func(), error) {
<-ctx.Done()

@coder-agents-review coder-agents-review Bot Oct 13, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P4 [CRF-95] TestWaitForAgentReady/AttemptTimeoutCause waits for the real 5s agentAttemptTimeout in each subtest. (Netero)

.claude/docs/TESTING.md:31-33 says to inject quartz.NewMock(t) instead of waiting on real time when a deadline is under test, and go test ./coderd/x/chatd/chattool/ -run TestWaitForAgentReady takes 5.07s. Give waitForAgentReady a quartz.Clock and end each attempt with clock.AfterFunc on a WithCancelCause context; if the poll switches to a status check, test that check without a deadline instead.

🤖

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread coderd/tailnet.go
d.gotConn = gotConn
}

// hadConn reports whether the transport gave the request a connection.

@coder-agents-review coder-agents-review Bot Oct 13, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit [CRF-102] The hadConn doc says it reports whether the transport gave the request a connection, but the GetConn hook resets the flag before each attempt, so it reports only the current attempt. (Gon)

TestReportProxyDialFailure_RetryDial asserts hadConn() is false for a request whose first attempt had a pooled connection. Write "hadConn reports whether the transport gave the current attempt a connection."

🤖

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
BobbyHo added 4 commits October 13, 2026 16:13
@BobbyHo
Merge branch 'main' into bobby/plat-680-servertailnet-observability-m…
2cc8137
…ake-agent-unreachable-failures
@BobbyHo
Merge branch 'main' into bobby/plat-680-servertailnet-observability-m…
989d4da
…ake-agent-unreachable-failures
@BobbyHo
Merge branch 'main' into bobby/plat-680-servertailnet-observability-m…
bf0acc0
…ake-agent-unreachable-failures
@BobbyHo
Merge branch 'main' into bobby/plat-680-servertailnet-observability-m…
d37a255
…ake-agent-unreachable-failures
Emyrk
Emyrk approved these changes Oct 14, 2026
View reviewed changes

@Emyrk Emyrk left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is all logs and metrics

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@BobbyHo
BobbyHo merged commit 603dd42 into main Oct 14, 2026
32 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@BobbyHo
BobbyHo deleted the bobby/plat-680-servertailnet-observability-make-agent-unreachable-failures branch October 14, 2026 21:55
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 14, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@nickvigilante nickvigilante nickvigilante left review comments
@Emyrk Emyrk Emyrk approved these changes
@code-asher code-asher Awaiting requested review from code-asher
+1 more reviewer
@coder-agents-review coder-agents-review[bot] coder-agents-review[bot] requested changes
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@BobbyHo BobbyHo

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@BobbyHo @Emyrk @nickvigilante
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.