GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — [iceberg] Skip already queued rows on a follow-mode retry by swanandx · Pull Request #7869 · feldera/feldera

Conversation

@swanandx

@swanandx swanandx commented Nov 6, 2026

Copy link
Copy Markdown
Member

Linked Issue

Follow-up to #7590.

In follow mode, a read that failed part way was retried from the start, so its rows were ingested twice. Now the retry skips the rows that earlier attempts already queued, for each data file.

before: attempt 1 queues rows 1-7, fails -> retry queues rows 1-8   (1-7 twice)
after:  attempt 1 queues rows 1-7, fails -> retry queues row 8 only

A restart from a checkpoint taken during a retry still reads the snapshot again (at-least-once), as the docs already say.

Describe Manual Test Plan

New test rest_follow_retry_does_not_repeat_rows: it breaks the last data file of a snapshot, waits for the read to fail, restores the file, and checks that each row is ingested once. It passed 5 of 5 runs. Without the skip, the test fails with rows ingested twice. The full Iceberg suite passes.

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

Breaking Changes?

Mark if you think the answer is yes for any of these components:

  • OpenAPI / REST HTTP API / feldera-types / manager (What is a breaking change?)
  • Feldera SQL (Syntax, Semantics)
  • feldera-sqllib (incl. dependencies fxp, etc.) (What is a breaking change?)
  • Python SDK (What is a breaking change?)
  • fda (CLI arguments)
  • Adapters (including configuration)
  • Storage Format / Checkpoints
  • Others (specify)

Describe Incompatible Changes

None.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@mythical-fred-oss

mythical-fred-oss Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

LGTM.

What I ran

Check Result
cargo test --locked -p feldera-iceberg deletion_vector_tests (incl. new a_retry_skips_the_rows_already_queued) 14/14 pass
cargo check --locked -p dbsp_connectors --tests --features iceberg-tests-follow clean
pre-commit run --files <changed files> pass
rest_follow_retry_does_not_repeat_rows not run (needs REST catalog + S3)

I checked the per-file skip-by-count against the parallel reader (per-file order holds with num_parsers > 1), against selected positions (the filter runs before the skip, so counts stay consistent), and against the new job_queue.flush() on error (no buffer lands after Rollback). I found no correctness issues.

Coverage gap (medium): the integration test breaks a whole file, so on retry each queued file is skipped in full. Two cases only have unit coverage of skip_queued itself: a failure in the middle of a multi-batch file, and a retry of a deletes read (polarity=false, deletion vectors). If the fixture can produce a file that fails after its first row group, a mid-file variant would cover the partial-skip path end to end.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
bestouff
bestouff approved these changes Nov 6, 2026
View reviewed changes

@bestouff bestouff left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some smallish changes, but otherwise LGTM

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread crates/connectors/src/test/iceberg.rs Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread docs.feldera.com/docs/connectors/sources/iceberg.md Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread crates/connectors/src/test/iceberg.rs Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread crates/connectors/src/test/iceberg.rs Resolved

Uh oh!

There was an error while loading. Please reload this page.
@swanandx
[iceberg] Skip already queued rows on a follow-mode retry
bebbf98 
A follow-mode read that failed part way was retried from the start, so
its rows were ingested twice. A retry now skips, in each data file, the
rows that earlier attempts queued.

Signed-off-by: Swanand Mulay <[email protected]>
@swanandx
swanandx force-pushed the iceberg-follow-retry branch from 458e7e9 to bebbf98 Compare November 6, 2026 12:31
@swanandx
swanandx added this pull request to the merge queue Nov 6, 2026
Merged via the queue into main with commit 59e64e8 Nov 6, 2026
1 check passed

Uh oh!

There was an error while loading. Please reload this page.
@swanandx
swanandx deleted the iceberg-follow-retry branch November 6, 2026 13:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bestouff bestouff bestouff approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@swanandx @bestouff
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.