GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Define which layer retries each kind of request failure · Issue #7879 · feldera/feldera

Define which layer retries each kind of request failure #7879

New issue
Copy link
New issue
Copy link
Open
Open
Define which layer retries each kind of request failure#7879
Copy link
Assignees
blp
Labels
CLIFeldera CLI (fda) - related issuesFeldera CLI (fda) - related issuesPipeline managerPipeline manager (API, API server, runner, compiler server)Pipeline manager (API, API server, runner, compiler server)RFCRequest for CommentsRequest for Commentspython-sdkIssues related to the feldera python sdkIssues related to the feldera python sdk

Description

@blp
blp
opened on Nov 6, 2026
Issue body actions

We have added retries for "pipeline unreachable" one layer at a time, usually after a CI flake:

Layer PR / issue
Python SDK #6851
fda #6890
One test (dropped) #7621, feldera/cloud#2124
Coordinator (multihost) feldera/cloud#2079
Error code clients can key on #7860
Manager proxy #7868

The result is retries that stack. With #7868, one unreachable pipeline can cost about 154 s with default client settings instead of about 35 s (see #7868 (comment)). The websocket proxy and the metrics scrape still make a single attempt. is_connect_failure in the manager and HostUnreachable in the coordinator classify the same failures separately.

Proposed rule: handle each kind of failure in exactly one layer, the lowest one that can tell (1) that a repeat is safe and (2) when to stop.

Failure Owner Reason
Stale DNS after a restart Kubernetes runner: remove the cause (feldera/cloud#2179) Fixing the source is better than retrying
Pod replaced in place, or not yet listening Manager proxy It knows the request was never sent and the pipeline's status and location
Manager unreachable, ingress 502 Client No layer below can see it
Manager gave up Client reports the error; no new retry loop Prevents stacking

Work:

  • Manager retries connect failures on all proxy paths, including websocket, with one budget.
  • Manager reports the attempt count in the error details. Clients do not retry when the manager already retried.
  • Document the policy (what is retried, the budget, which layer owns it) and the manager budget in the client timeout docs.
  • Decide whether the manager and the coordinator should share one connect-failure classification.
Reacted by Gerd Zellweger
Reacted by Gerd Zellweger

Activity

  1. blp
    self-assigned this
    on Nov 6, 2026
  2. blp
    added
    RFCRequest for CommentsRequest for Comments
    Pipeline managerPipeline manager (API, API server, runner, compiler server)Pipeline manager (API, API server, runner, compiler server)
    python-sdkIssues related to the feldera python sdkIssues related to the feldera python sdk
    CLIFeldera CLI (fda) - related issuesFeldera CLI (fda) - related issues
    on Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

  • @blp
    blp

Labels

CLIFeldera CLI (fda) - related issuesFeldera CLI (fda) - related issuesPipeline managerPipeline manager (API, API server, runner, compiler server)Pipeline manager (API, API server, runner, compiler server)RFCRequest for CommentsRequest for Commentspython-sdkIssues related to the feldera python sdkIssues related to the feldera python sdk

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

    You are visitor 01402503 — GeezerVison does not store the pages you look up.
    Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.