GeezerVison

Problem

Right after a pipeline (re)start the api-server can fail to open a connection to it:

• cluster DNS keeps answering with the previous pod's IP for the record's TTL (30 s with kube-dns and NodeLocal DNSCache on GKE), measured in feldera/cloud#2179;
• a cached negative answer can outlive the new Service;
• a pod can refuse connections for a moment before its server listens.

Since 004e386 the proxy answers such failures with 503 PipelineInteractionConnectFailed and the SDK and fda retry on it, but the manager itself makes a single attempt, so every other client still sees the 503. feldera/cloud#2179 removes the restart case by giving each deployment its own host name; a pod replaced within a deployment (preemption, the fault-tolerance tests' pod kills, feldera/cloud#1470) keeps its name and still hits the window.

Change

A connect failure means the request was never sent, so repeating it cannot apply it twice. forward_http_request_to_pipeline and streaming_proxy now retry connect-phase failures (timeout, refused, resolver error; not a disconnect after the connection was open) every 500 ms for up to 35 s, which outlasts the DNS window with an attempt to spare.

• A streaming request body is shared across attempts (SharedPayload) and only replayed while no attempt has read from it; awc opens the connection before it reads the body, so a connect failure leaves it untouched.
• When the proxy gives up, the PipelineInteractionConnectFailed message says how many attempts it made over how long. When a retry rescues a request, the proxy logs it, which also gives the observability asked for in feldera/cloud#2125 a first data point.
• The classification lives in one place, is_connect_failure, used by both the retry loop and the error mapping.
• Before each further attempt the proxy re-checks the pipeline through the same check_pipeline the first attempt used (cached, evicted on every pipeline change). A pipeline that was stopped or became Unavailable meanwhile ends the loop with its status error instead of running down the budget; a pipeline that was restarted meanwhile is followed to its new location.

Not changed: the runner's own status polling, which has its own semantics for an unreachable pipeline, the websocket proxy, and the static forward_http_request_to_pipeline used by the metrics scrape, which walks every pipeline in sequence from a one-time snapshot and therefore keeps making a single attempt (ConnectRetry::NONE).

Testing

• test_streaming_proxy_retries_until_the_pipeline_listens: nothing listens on the port for 1.5 s, then a server appears; the POST succeeds and the server receives the whole body after the refused attempts.
• test_streaming_proxy_connect_retry_budget: nothing ever listens; PipelineInteractionConnectFailed arrives only after the 1 s budget and names the attempts.
• test_streaming_proxy_connect_retry_stops_when_the_pipeline_is_gone: a re-check that reports the pipeline unavailable ends the loop at once with that error.
• test_streaming_proxy_connect_retry_follows_a_moved_pipeline: a re-check that returns a new location sends the next attempt there.
• Each pair fails with its mechanism disabled (the retry loop, the re-check). The existing refused-connection test now runs with a zero budget to keep testing the single-attempt classification.
• cargo test -p pipeline-manager --lib runner::interaction: 14 passed. clippy and pre-commit clean.