GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Flaky Kafka FT test: random_end_to_end_json_array_small hits UnknownPartition under CI load · Issue #7641 · feldera/feldera

Flaky Kafka FT test: random_end_to_end_json_array_small hits UnknownPartition under CI load #7641

New issue
Copy link
New issue
Copy link
Closed
#7849
Closed
Flaky Kafka FT test: random_end_to_end_json_array_small hits UnknownPartition under CI load#7641
#7849
Copy link
Assignees
blp
Labels
connectorsIssues related to the adapters/connectors crateIssues related to the adapters/connectors cratefred-observeTest failure diagnosed by the fred-observe botTest failure diagnosed by the fred-observe bot

Description

@mythical-fred-oss
mythical-fred-oss
bot
opened on Oct 26, 2026
Issue body actions

Verdict: flaky test / CI infrastructure, not caused by PR #7619 — medium confidence.

What failed

Adapter Tests / Test Adapters (gke-runners-amd64, ...) in a merge-queue run for #7619:
https://github.com/feldera/feldera/actions/runs/40375093325/job/121764995826

connector_suites (static connectors, transport::kafka::ft::test::random_end_to_end_json_array_small) failed:

thread '<unnamed>' panicked at crates/connectors/src/transport/kafka/ft/test.rs:2437:21:
proptest_kafka_end_to_end_json_array_small: error: error on input endpoint 'test_input1': UnknownPartition (Local: Unknown partition)

which then caused the check_end_to_end wait (crates/connectors/src/test/kafka.rs:418) to time out and panic with called Result::unwrap() on an Err value: ().

The job's RSS logger shows memory doubling right around the failure (685 MiB → 1.21 GiB → 1.92 GiB in ~2s), consistent with CI resource pressure rather than a logic bug.

Why I don't think the PR caused it

  • crates/connectors/src/transport/kafka/ft/input.rs and crates/connectors/src/test/kafka.rs (topic creation / UnknownPartition handling, retry/backoff logic) are unchanged in substance by Engine-independent adapters #7619 — the diff there is only #[cfg(test)] → #[cfg(any(test, feature = "testing"))] and a couple of unrelated renames.
  • random_end_to_end_json_array_small itself (and check_end_to_end) is pre-existing logic, just moved into the new static-connectors test harness.
  • Every other recent main merge-queue run in the last ~30 (gh run list --workflow ci.yml) has Adapter Tests passing.
  • This PR's two prior merge-queue attempts (runs 40369343699, 40352769190) each failed on a different, unrelated job (Rust Unit Tests, Build Rust Binaries respectively), suggesting general CI flakiness under this PR's unusually large diff rather than a failure specific to this code path.

Related flaky-Kafka-FT-test issues (same suite, different tests)

  • Flaky test: transport::kafka::ft::test::test_kafka_metadata_raw #6430 test_kafka_metadata_raw
  • Flaky tests: exchange::tests::multihost and kafka ft test_input_partitions_latest #6728 test_input_partitions_latest

None of those match this test/error exactly, so opening separately rather than commenting there.

Next step

If this recurs on other PRs (not just large ones), worth checking whether the Redpanda service container is resource-starved when many Adapter Tests runs happen concurrently on gke-runners-amd64, and whether check_topics's per-topic fetch_watermarks retry (crates/connectors/src/test/kafka.rs:45-67) needs to also cover the consumer thread's own metadata fetch, not just the admin client's.

Activity

  1. mythical-fred-oss
    assigned
    blp
    on Oct 26, 2026
  2. mythical-fred-oss
    added
    connectorsIssues related to the adapters/connectors crateIssues related to the adapters/connectors crate
    fred-observeTest failure diagnosed by the fred-observe botTest failure diagnosed by the fred-observe bot
    on Oct 26, 2026
  3. blp commented on Nov 4, 2026

    @blp
    blp
    on Nov 4, 2026
    Member
    More actions

    Comment from Claude (AI assistant), on behalf of Ben.

    I think the root cause is in the test, not CI load.

    Step Code Behavior
    1 crates/connectors/src/test/kafka.rs check_topics Probes the new topic with the admin client only. The pipeline's consumer can still have stale metadata and report UnknownPartition once.
    2 crates/connectors/src/transport/kafka.rs refine_kafka_error Every error other than RDKafkaErrorCode::Fatal is non-fatal, so UnknownPartition is non-fatal and the endpoint keeps going.
    3 crates/adapterlib/src/errors/controller.rs (Display for InputTransportError) Fatal errors start with FATAL . The message in this issue does not, which confirms the error is non-fatal.
    4 crates/connectors/src/transport/kafka/ft/test.rs check_end_to_end The error callback panics on any error while the test runs, fatal or not.
    5 The panic is on a controller thread, so the test thread waits for output that never comes and times out (unwrap() on Err(())).

    Extending the topic probe to the consumer's own metadata would make the window smaller but would not close it. The proposed fix is to panic only on fatal transport errors in check_end_to_end (and in the buffer test, which has the same callback) and log the non-fatal ones. The output check still catches lost data. A PR will follow.

  4. blp
    added 2 commits that reference this issue on Nov 4, 2026

    [connectors] Kafka tests: do not fail on transient transport errors.

    This issue will close once commit 9144d44 is merged into the 'main' branch.
    9144d44

    [connectors] Kafka tests: do not fail on transient transport errors.

    This issue will close once commit 4c0baea is merged into the 'main' branch.
    4c0baea
  5. blp
    closed this as completedin #7849on Nov 5, 2026
  6. yermakoffivan
    added a commit that references this issue on Nov 5, 2026

    [connectors] Kafka tests: do not fail on transient transport errors.

    This issue will close once commit 2c7a1a7 is merged into the 'main' branch.
    2c7a1a7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

  • @blp
    blp

Labels

connectorsIssues related to the adapters/connectors crateIssues related to the adapters/connectors cratefred-observeTest failure diagnosed by the fred-observe botTest failure diagnosed by the fred-observe bot

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

    You are visitor 01402520 — GeezerVison does not store the pages you look up.
    Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.