GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
GeezerVison is showing github.com — [connectors] Judge Kafka test reads against the live high watermark by bestouff · Pull Request #7836 · feldera/feldera Conversation
|
| step | value |
|---|---|
| snapshot before commit | low 0, high 0 |
| LSO after commit landed | 3 (fence batch, record, marker) |
| consumer EOF position | 3, one EOF only |
read_records re-queries the high watermark with a read_uncommitted consumer at each EOF instead of trusting a snapshot. The proxy comment that claimed the transaction is already committed when the broker answers is corrected.
committed() while the transaction is open fails with the exact CI panic without the fix and passes with it.transport::kafka::ft::output passed 3 of 3 runs with the fix.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
| .unwrap_or_default(), | ||
| }), | ||
| Some(Err(KafkaError::PartitionEOF(_))) => { | ||
| let (_, high) = watermarks |
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
pitfall / medium: A read_committed consumer stops at the LSO, and now it has to reach the uncommitted HWM. Suppose a partition still has a transaction that hasn't been decided yet, for example a producer that died mid-transaction and is waiting for transaction.timeout.ms. In that case this call waits the full 60 s and then panics with "must reach the high watermark", which says nothing about the open transaction. No current caller hits this, because every read happens after EndTxn has been decided. A future test that reads while a transaction is still open would hang, though. Could you document this precondition in the doc comment, or put high and the position in the expect message so the failure is easier to diagnose?
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
Review summary. This is a test-only fix for the flaky
missing / medium: regression test. The description mentions an "early-reader" test that starts // in real.rs: begin reading while EndTxn is still in flight, then release it proxy.hold_commit(true); let worker = BatchWorker::new(endpoint); let reader = std::thread::spawn({ let t = topic.to_owned(); move || committed(&t, 1) }); wait(|| proxy.held_commits() > 0, 30_000).unwrap(); assert_eq!(reader.join().unwrap().len(), 1); proxy.hold_commit(false); There are no other gate failures: there are no production code changes, no unsafe blocks and no dependency changes. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
88258f2 to
c31a690
Compare
A broker answers EndTxn before it writes the commit markers to the data partitions. The ft output recovery tests hold that reply and read the topic at once, so the read_committed watermark that read_records took before reading could be the last stable offset of the still open transaction. The consumer then passed that offset once the markers landed and librdkafka emits PartitionEOF only once per end offset, so the read waited 60 s and panicked with "the consumer must reach the high watermark". Query the high watermark with a read_uncommitted consumer at each EOF instead of trusting a snapshot taken before reading. Fixes #7765
…mmit lands Also name the topic, partition and the last EOF seen in the watermark panic, and document that read_records needs every transaction decided.
c31a690 to
6a63439
Compare
| let proxy = BrokerProxy::new(default_redpanda_server()); | ||
| let (mut endpoint, _) = endpoint(topic, &proxy, 60_000, 60_000); | ||
| endpoint.batch_start(0, OutputBatchType::Delta).unwrap(); | ||
| endpoint.push_buffer(b"read before its markers").unwrap(); |
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
bug (medium, flaky test): push_buffer only queues the record in librdkafka. The transactional producer still has to do AddPartitionsToTxn, wait out linger.ms and send the Produce request. Meanwhile the reader thread may reach its first EOF at offset 0, where the read_uncommitted high watermark is still 0. In that case read_records returns [] and assert_eq!(…len(), 1) fails under CI load. This race could also explain part of the "fails without the fix" result. The fix is to wait for delivery before spawning the reader, so the test really checks "open transaction, record on the broker":
| endpoint.push_buffer(b"read before its markers").unwrap(); | |
| endpoint.push_buffer(b"read before its markers").unwrap(); | |
| wait( | |
| || callbacks.metric("kafka_output_delivered_messages_total") == 1.0, | |
| 30_000, | |
| ) | |
| .unwrap(); |
(Bind callbacks instead of _ from endpoint(...).)
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
Review summary. This change touches only tests. Re-querying the read_uncommitted high watermark at each EOF fixes the stale-snapshot timeout described in #7765, and I agree with the diagnosis.
|
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
fred-review did not finish; see https://github.com/feldera/feldera/actions/runs/41854460216 |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
…st Fetch Wait for delivery before the reader starts, assert that the transaction is open, and route the reader through the proxy so the worker only starts after the reader has queried the watermark and sent its first Fetch.
|
LGTM. This PR only touches tests. The fix fits the root cause:
|
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
This is somewhat intricate, but I hope it's an improvement in flakiness
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
Successfully merging this pull request may close these issues.
Kafka ft output: restart_skips_a_commit_whose_reply_was_lost times out waiting for high watermark
There was an error while loading. Please reload this page.