GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — [connectors] Judge Kafka test reads against the live high watermark by bestouff · Pull Request #7836 · feldera/feldera

Conversation

@bestouff

@bestouff bestouff commented Nov 4, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Fixes #7765.

Cause

Redpanda answers EndTxn as soon as the commit is decided and writes the commit markers to the data partitions afterwards (do_commit_tm_tx: "release control to the client before committing transaction on data partitions"). The real recovery tests hold that reply and call committed() at once. read_records snapshotted the read_committed watermark before reading, so under CI load the snapshot was the last stable offset of the still open transaction. Once the markers landed, the consumer passed that offset. librdkafka emits PartitionEOF once per end offset, so the position never matched the stale snapshot and the read timed out after 60 s.

The broker side transaction.timeout.ms is not involved. The 60 s REQTMOUT in the logs is librdkafka's socket timeout on the held reply.

Reproduced locally against ghcr.io/feldera/redpanda-ci:v24 by snapshotting the watermark while the transaction is open:

step value
snapshot before commit low 0, high 0
LSO after commit landed 3 (fence batch, record, marker)
consumer EOF position 3, one EOF only

Fix

read_records re-queries the high watermark with a read_uncommitted consumer at each EOF instead of trusting a snapshot. The proxy comment that claimed the transaction is already committed when the broker answers is corrected.

Validation

  • An early-reader scenario that starts committed() while the transaction is open fails with the exact CI panic without the fix and passes with it.
  • transport::kafka::ft::output passed 3 of 3 runs with the fix.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bestouff
bestouff requested a review from mihaibudiu November 4, 2026 14:09
mythical-fred-oss[bot]
mythical-fred-oss Bot reviewed Nov 4, 2026
View reviewed changes
Comment thread crates/connectors/src/transport/kafka/ft/output/tests.rs
.unwrap_or_default(),
}),
Some(Err(KafkaError::PartitionEOF(_))) => {
let (_, high) = watermarks

@mythical-fred-oss mythical-fred-oss Bot Nov 4, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall / medium: A read_committed consumer stops at the LSO, and now it has to reach the uncommitted HWM. Suppose a partition still has a transaction that hasn't been decided yet, for example a producer that died mid-transaction and is waiting for transaction.timeout.ms. In that case this call waits the full 60 s and then panics with "must reach the high watermark", which says nothing about the open transaction. No current caller hits this, because every read happens after EndTxn has been decided. A future test that reads while a transaction is still open would hang, though. Could you document this precondition in the doc comment, or put high and the position in the expect message so the failure is easier to diagnose?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@mythical-fred-oss

mythical-fred-oss Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Review summary. This is a test-only fix for the flaky real Kafka FT recovery read (#7765). The diagnosis is sound: the broker sends the EndTxn reply before it writes the commit markers, so a read_committed snapshot can be stale. Judging each EOF against the live HWM fixes that.

What I ran Result
cargo test --locked -p dbsp_connectors --lib transport::kafka::ft::output -- --skip real (mock cluster, through read_topic → read_records) 12/12 pass
real::* tests not run: no Redpanda broker in this environment

missing / medium: regression test. The description mentions an "early-reader" test that starts committed() while the transaction is still open. It fails without the fix and passes with it, but it isn't in the PR. Without it, nothing guards against this flake coming back (feldera-pr-checks rule 4, flaky tests). Please commit it alongside a_committed_transaction_with_a_delayed_reply_is_not_replayed in real.rs. One possible shape:

// in real.rs: begin reading while EndTxn is still in flight, then release it
proxy.hold_commit(true);
let worker = BatchWorker::new(endpoint);
let reader = std::thread::spawn({ let t = topic.to_owned(); move || committed(&t, 1) });
wait(|| proxy.held_commits() > 0, 30_000).unwrap();
assert_eq!(reader.join().unwrap().len(), 1);
proxy.hold_commit(false);

There are no other gate failures: there are no production code changes, no unsafe blocks and no dependency changes.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bestouff
bestouff force-pushed the kafka-ft-read-records-watermark branch from 88258f2 to c31a690 Compare November 4, 2026 14:56
@bestouff
bestouff marked this pull request as draft November 4, 2026 14:57
@bestouff
bestouff marked this pull request as ready for review November 4, 2026 14:57
bestouff added 2 commits November 4, 2026 15:59
@bestouff
[connectors] Judge Kafka test reads against the live high watermark
897b3e3
A broker answers EndTxn before it writes the commit markers to the data
partitions. The ft output recovery tests hold that reply and read the
topic at once, so the read_committed watermark that read_records took
before reading could be the last stable offset of the still open
transaction. The consumer then passed that offset once the markers landed
and librdkafka emits PartitionEOF only once per end offset, so the read
waited 60 s and panicked with "the consumer must reach the high
watermark".

Query the high watermark with a read_uncommitted consumer at each EOF
instead of trusting a snapshot taken before reading.

Fixes #7765
@bestouff
[connectors] Add a regression test for reads that start before the co…
6a63439
…mmit lands

Also name the topic, partition and the last EOF seen in the watermark
panic, and document that read_records needs every transaction decided.
@bestouff
bestouff force-pushed the kafka-ft-read-records-watermark branch from c31a690 to 6a63439 Compare November 4, 2026 14:59
mythical-fred-oss[bot]
mythical-fred-oss Bot reviewed Nov 4, 2026
View reviewed changes
Comment thread crates/connectors/src/transport/kafka/ft/output/tests/real.rs
let proxy = BrokerProxy::new(default_redpanda_server());
let (mut endpoint, _) = endpoint(topic, &proxy, 60_000, 60_000);
endpoint.batch_start(0, OutputBatchType::Delta).unwrap();
endpoint.push_buffer(b"read before its markers").unwrap();

@mythical-fred-oss mythical-fred-oss Bot Nov 4, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug (medium, flaky test): push_buffer only queues the record in librdkafka. The transactional producer still has to do AddPartitionsToTxn, wait out linger.ms and send the Produce request. Meanwhile the reader thread may reach its first EOF at offset 0, where the read_uncommitted high watermark is still 0. In that case read_records returns [] and assert_eq!(…len(), 1) fails under CI load. This race could also explain part of the "fails without the fix" result. The fix is to wait for delivery before spawning the reader, so the test really checks "open transaction, record on the broker":

Suggested change
endpoint.push_buffer(b"read before its markers").unwrap();
endpoint.push_buffer(b"read before its markers").unwrap();
wait(
|| callbacks.metric("kafka_output_delivered_messages_total") == 1.0,
30_000,
)
.unwrap();

(Bind callbacks instead of _ from endpoint(...).)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@mythical-fred-oss

mythical-fred-oss Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Review summary. This change touches only tests. Re-querying the read_uncommitted high watermark at each EOF fixes the stale-snapshot timeout described in #7765, and I agree with the diagnosis.

What I ran Result
cargo test --locked -p dbsp_connectors --no-run builds
transport::kafka::ft::output::tests (MockCluster, real skipped) 12/12 pass
pre-commit run --files <changed files> pass
real tests, including the new one not run: no Redpanda broker available
  • Flaky test (medium), gate 4: the new a_read_that_starts_before_the_commit_lands_sees_it races the asynchronous produce. If the reader reaches its first EOF before the record lands, the high watermark is 0 and the read returns nothing. See the inline comment for a fix that waits for delivery first. This is the only change I'd ask for before merge.
  • Coverage (medium): with that fix, the new test still only checks that the read finishes. It never checks that the snapshot was taken while the transaction was open. I can't confirm the claim that it fails without the fix, because some of those failures could come from the same race.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mythical-fred-oss

mythical-fred-oss Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

fred-review did not finish; see https://github.com/feldera/feldera/actions/runs/41854460216

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bestouff
[connectors] Order the early-read regression test by the reader's fir…
cf59cfa
…st Fetch

Wait for delivery before the reader starts, assert that the transaction
is open, and route the reader through the proxy so the worker only starts
after the reader has queried the watermark and sent its first Fetch.
@mythical-fred-oss

mythical-fred-oss Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

LGTM. This PR only touches tests. The fix fits the root cause: read_records now re-checks the read_uncommitted high watermark at every EOF instead of trusting a read_committed watermark taken before the commit markers land. The precondition (every transaction decided) is written down, and the panic message now says what went wrong.

What I ran Result
cargo test --locked -p dbsp_connectors --lib transport::kafka::ft::output::tests:: -- --skip real (mock cluster, now with two consumers per read) 12/12 pass
cargo fmt -p dbsp_connectors --check clean
tests::real::*, including the new a_read_that_starts_before_the_commit_lands_sees_it not run: no Redpanda broker in this environment, so I'm relying on CI and the author's 3/3 runs
  • Checked, no problem found (flakiness): the new test orders its steps by events, not sleeps: delivery metric, then an open-transaction assert, then the reader's first Fetch, then the held EndTxn. If the reader panics, BatchWorker::drop cancels shutdown and the proxy drop ends the hold, so a failure can't hang the run.
  • Commit history (medium): commit 3 ("Order the early-read regression test by the reader's first Fetch") reworks the test added in commit 2. Please squash it into commit 2 before merge. The subjects of commits 2 and 3 also go over 72 characters.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

mihaibudiu
mihaibudiu approved these changes Nov 4, 2026
View reviewed changes

@mihaibudiu mihaibudiu left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is somewhat intricate, but I hope it's an improvement in flakiness

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bestouff
bestouff added this pull request to the merge queue Nov 4, 2026
Merged via the queue into main with commit 52346d2 Nov 5, 2026
1 check passed

Uh oh!

There was an error while loading. Please reload this page.

@bestouff
bestouff deleted the kafka-ft-read-records-watermark branch November 5, 2026 00:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mythical-fred-oss mythical-fred-oss[bot] mythical-fred-oss[bot] left review comments
@mihaibudiu mihaibudiu mihaibudiu approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Kafka ft output: restart_skips_a_commit_whose_reply_was_lost times out waiting for high watermark

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@bestouff @mihaibudiu
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.