Description

mythical-fred-oss

bot

opened on Oct 30, 2026

Issue body actions

Verdict

Pre-existing bug on main, unrelated to the PR that triggered this run (PR #7696 only touches the NATS connector transport, crates/connectors/src/transport/nats/*). Moderately confident in the mechanism below; the test itself and the code path it exercises were last touched by an earlier, unrelated commit.

Failure

• Job: Integration Tests / Runtime Integration Tests (gke-runners-amd64)
• Test: tests/runtime/test_adaptive_joins.py::TestAdaptiveJoins::test_adaptive_joins_balancer_policies
• Log lines 505-517:

self.client.patch_pipeline(sql=SQL_JOIN_ABC)   # test_adaptive_joins.py:179 (pipeline.modify)
wait_for_program_success(name, prev_ver + 1)   # test_adaptive_joins.py:180
...
RuntimeError: program version (3) != expected program version (2)

The test stops the pipeline, reads prev_ver, calls pipeline.modify(sql=...) once, and waits for exactly prev_ver + 1. The server ended up two versions ahead instead of one, meaning the mutating PATCH that bumps program_version (crates/pipeline-manager/src/db/operations/pipeline.rs:761) was applied twice for a single client-side call.

Suspected mechanism

python/feldera/rest/_httprequests.py::_is_retryable retries FelderaAPIErrors whose status is in retryable_status_codes (408/429/502/503/504 by default) unconditionally, regardless of the idempotent flag:

if idempotent and isinstance(exc, requests.exceptions.ConnectionError):
    return True
if isinstance(exc, FelderaAPIError):
    return exc.status_code in self.config.retry_config.retryable_status_codes  # no idempotent check

patch_pipeline (python/feldera/rest/feldera_client.py:479) doesn't pass idempotent=True, matching the doc comment that PATCH "may already have been applied" — but that protection only covers ConnectionError. A 502/503/504 returned by the ingress/gateway after the pipeline-manager already applied the PATCH (plausible on the known-flaky GKE CI runners, see #7465) would cause the client to silently retry a non-idempotent, counter-bumping PATCH, applying it twice.

Next step

Gate the status-code retry branch on idempotent too (or make patch_pipeline's program-mutating fields excluded from blind retries), so a gateway hiccup after a successful PATCH can't double-apply it. python/feldera/rest/_httprequests.py:159-160.

Seen once so far, in this run. Re-queueing the PR is likely to succeed since the PR itself doesn't touch this code path.

Activity

1. mythical-fred-oss

   added

   python-sdkIssues related to the feldera python sdkIssues related to the feldera python sdk

   fred-observeTest failure diagnosed by the fred-observe botTest failure diagnosed by the fred-observe bot

   on Oct 30, 2026
2. mythical-fred-oss

   assigned

   blp

   on Oct 30, 2026
3. blp commented on Nov 2, 2026

   

   blp

   on Nov 2, 2026

   Member

   More actions

   I think the mechanism in the description is not the cause of this failure.

   A retried PATCH cannot bump program_version twice

   The manager bumps program_version only when the program is different from the stored one (crates/pipeline-manager/src/db/operations/pipeline.rs:701-711). A replayed PATCH has the same SQL as the first one, so it changes version but not program_version.

   Step	program_version
   First PATCH, SQL_JOIN_AB → SQL_JOIN_ABC	1 → 2
   Retried PATCH, SQL_JOIN_ABC → SQL_JOIN_ABC	stays 2

   What can give 3

   Only one statement bumps program_version (pipeline.rs:761). Two paths reach it:

   1. A user edit (PATCH/PUT). It sets the pipeline's platform_version to the API server's version.
   2. The compiler, before each compile. clear_ongoing_sql_compilation_for_worker (crates/pipeline-manager/src/compiler/sql_compiler.rs:186-194) selects pending pipelines whose platform_version is different from the compiler's (pipeline.rs:1952-1957) and calls update_pipeline(..., bump_platform_version=true) (crates/pipeline-manager/src/db/storage_postgres.rs:1574-1589). That bumps program_version again.

   So if the API server and the compiler have different platform versions when the test calls modify(), one edit gives 1 → 2 → 3. The first compile in the test went normally, so the mismatch probably started during the run. A rolling redeploy of the shared amd64-gke.ci.feldera.io instance would do this. I did not confirm that a redeploy happened at that time.

   Fixes

   • _wait_for_compilation now accepts program_version >= expected (an older version still raises).
   • The retry concern in the description is real for other requests, so it has its own issue: Python client retries non-idempotent requests after failures where the server may have applied them #7793. The manager also sends 503 after a proxied request may have been applied, so the fix covers more than 502/504.

   Both are in one PR (to follow).

   Posted by Claude (AI assistant) on behalf of @blp.
4. blp

   added a commit that references this issue on Nov 3, 2026

   [python] Accept a newer program version after compilation.

   This issue will close once commit c4e1f51 is merged into the 'main' branch.

   c4e1f51
5. blp

   closed this as completedin #7795on Nov 3, 2026
6. AnnaZhand

   added a commit that references this issue on Nov 3, 2026

   [python] Accept a newer program version after compilation.

   This issue will close once commit eb19160 is merged into the 'main' branch.

   eb19160

Issue actions