Description

mythical-fred-oss

bot

opened on Oct 31, 2026

Issue body actions

Verdict

Pre-existing bug on main, unrelated to any specific PR. Seen in a merge-queue run that merged PR #7696 (which only touches NATS input, not Kafka) with current main.

Failing test

transport::kafka::ft::output::tests::an_outage_aborts_and_replays_the_transaction in crates/connectors/src/transport/kafka/ft/output/tests.rs:260

• Job: Adapter Tests / Test Adapters (arm64)
• Run: gh-readonly-queue/main/pr-7696-111d1cd9d... (merge queue run 41174148017)

Evidence

thread 'transport::kafka::ft::output::tests::an_outage_aborts_and_replays_the_transaction' panicked at crates/connectors/src/transport/kafka/ft/output/tests.rs:260:5:
assertion `left == right` failed: the spool must be released at commit
  left: 190
 right: 0

The test drives a mock Kafka broker through an outage (timed-out deliveries, abort, replay), brings the broker back up, and asserts the commit loop finishes successfully (result.expect("the commit loop must succeed once the broker is back") passes) — but endpoint.memory() still reports 190 bytes in the replay spool instead of 0. So the commit reports success while the spool isn't fully released.

Where it came from

The test and the code it exercises were added by commit 6ad915e99 ("[connectors] Recover Kafka output transactions through broker outages"), merged to main on 2026-10-28 — after PR #7696's branch point. It is not present on the PR branch at all.

Next step

Likely a race between reporting the successful commit and clearing the replay spool after an abort+replay cycle in crates/connectors/src/transport/kafka/ft/output.rs. Worth checking whether the spool-clear happens unconditionally on the success path, or only for records replayed in the last attempt (leaving earlier-replayed bytes counted). Re-running this test a few times locally (it uses rdkafka::mocking::MockCluster, no external broker needed) should show if it's a deterministic logic bug or timing-sensitive.

Activity

1. mythical-fred-oss

   assigned

   bestouff

   on Oct 31, 2026
2. mythical-fred-oss

   added

   fred-observeTest failure diagnosed by the fred-observe botTest failure diagnosed by the fred-observe bot

   connectorsIssues related to the adapters/connectors crateIssues related to the adapters/connectors crate

   on Oct 31, 2026
3. mythical-fred-oss commented on Oct 31, 2026

   

   mythical-fred-ossBot

   on Oct 31, 2026

   ContributorAuthor

   More actions

   Re-failure

   Same failure as before, in a different merge-queue run.

   • Job: Adapter Tests / Test Adapters (amd64)
   • PR in queue: [SQL] Rebuild a keyed table's whole row in an index closure under --gen2 #7739 (gz/gen2-window-keyed-row, SQL compiler only — unrelated to Kafka)

   thread 'transport::kafka::ft::output::tests::an_outage_aborts_and_replays_the_transaction' panicked at crates/connectors/src/transport/kafka/ft/output/tests.rs:260:5:
   assertion `left == right` failed: the spool must be released at commit
     left: 190
    right: 0
   

   Confirms this is not tied to a specific PR — still open, still unfixed.
4. mythical-fred-oss commented on Oct 31, 2026

   

   mythical-fred-ossBot

   on Oct 31, 2026

   ContributorAuthor

   More actions

   Re-failed in another merge-queue run, again unrelated to the PR being merged.

   • Job: Adapter Tests / Test Adapters (arm64)
   • Run: merge queue run 41220218215 (PR [profiler] Remove support for legacy profiles produced pre 0.244 (Feb 2026) #7756, which only touches profiler JS/TS files — no connectors code)
   • Same assertion, same test:

   thread 'transport::kafka::ft::output::tests::an_outage_aborts_and_replays_the_transaction' panicked at crates/connectors/src/transport/kafka/ft/output/tests.rs:260:5:
   assertion `left == right` failed: the spool must be released at commit
     left: 190
    right: 0
   

   Confirms this isn't a one-off: it's dropping unrelated PRs from the merge queue on repeat.
5. mythical-fred-oss commented on Nov 2, 2026

   

   mythical-fred-ossBot

   on Nov 2, 2026

   ContributorAuthor

   More actions

   Re-failure

   Same failure, this time in PR #7778's merge-queue run (sharded-accumulator-lag-notices, which only touches crates/dbsp exchange/sharded-accumulator code — unrelated to Kafka connectors).

   • Job: Adapter Tests / Test Adapters (gke-runners-amd64, x86_64, x86_64-unknown-linux-gnu)
   • Same assertion, same test:

   thread 'transport::kafka::ft::output::tests::an_outage_aborts_and_replays_the_transaction' panicked at crates/connectors/src/transport/kafka/ft/output/tests.rs:260:5:
   assertion `left == right` failed: the spool must be released at commit
     left: 190
    right: 0
   

   Confirms it's still dropping unrelated PRs from the merge queue.
6. bestouff commented on Nov 3, 2026

   

   bestouff

   on Nov 3, 2026

   Contributor

   More actions

   Fixed on main by a59c92d.
7. bestouff

   closed this as completedon Nov 3, 2026

Issue actions