GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — py: retry error-dismissal ingress on pipeline connect failure by mythical-fred-ops[bot] · Pull Request #7621 · feldera/feldera

Conversation

mythical-fred-ops Bot commented Oct 23, 2026

Copy link

Copy Markdown

Contributor

Problem

test_pipeline_error_dismissal can flake. The panic-inducing ingress POST sometimes gets 503 PipelineInteractionUnreachable: Failed to connect to host even though wait_for_pipeline_reachable() has just seen /stats return 200. One successful connection does not guarantee that the next one succeeds.

Change

cause_error_and_wait_for_stopped now retries the ingress POST every 0.5s, for up to 30s, when the failure is a connect failure.

503 message	Retried?	Why
`Failed to connect to host...`	yes	the request never reached the pipeline
timeout / disconnect mid-exchange	no	the data may already have been applied

This matches is_never_dispatched_503 in crates/rest-api/src/retry.rs.

Testing

pre-commit (ruff, ruff-format): passed.
Ran a local mock of post_json (no cluster):
- one connect-failure 503, then 200: passes with the fix; without the fix it raises the same RuntimeError seen in the flaky runs
- timeout 503: not retried; fails as before
I did not run the full platform test against a live cluster.

Generated with Claude Code

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions


          py: retry error-dismissal ingress on pipeline connect failure

ab62b5c

A successful /stats probe via wait_for_pipeline_reachable() does not
guarantee that the next request reaches the pipeline. The API server can
still fail to open a connection to the pod and answer 503
PipelineInteractionUnreachable, which made the deliberate panic-inducing
ingress in test_pipeline_error_dismissal flaky.

Retry the ingress for up to 30s, but only for connect-phase failures
("Failed to connect to host"), which guarantee the data was never
delivered. Other unreachable errors, such as a mid-exchange timeout, may
already have been applied and still fail the test.

Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>

gz closed this

Nov 6, 2026

blp mentioned this pull request

Nov 6, 2026

Define which layer retries each kind of request failure #7879

Open

4 tasks

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

1 participant

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.