GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — [connectors] Delta merge: get the key ranges of all files at once by swanandx · Pull Request #7948 · feldera/feldera

Conversation

@swanandx

@swanandx swanandx commented Nov 12, 2026

Copy link
Copy Markdown
Member

Linked Issue

None. To skip files, a merge flush uses the key range of each file. The key range is the smallest and the largest key in the file. Now the flush gets the ranges of all files with one call for each key column, not one call for each file.

before: for each file       -> get its key range
after:  for each key column -> get the key ranges of all files   (ROW keys: no change)
Table with 200k files Before After
Time to get the key ranges 1.6 s 25 ms
Time for one flush 2.6 s 0.9 s
Files skipped 199,936 199,936
Memory for each file, BIGINT key (50k files) 431 B 215 B

Describe Manual Test Plan

A new test makes sure that the new key ranges are equal to the old key ranges after DELETE, OPTIMIZE, VACUUM and a reopen. I ran the old code and the new code in turns to get the numbers in the table.

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

Breaking Changes?

None.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
mythical-fred-oss[bot]
mythical-fred-oss Bot reviewed Nov 12, 2026
View reviewed changes
Comment thread crates/connectors/src/integrated/delta_table/merge/probe.rs Resolved

Uh oh!

There was an error while loading. Please reload this page.
@mythical-fred-oss

This comment was marked as resolved.

Sign in to view
@swanandx
swanandx force-pushed the merge-bounds-measure branch from a69aaec to 81e5225 Compare November 12, 2026 12:03
mythical-fred-oss[bot]
mythical-fred-oss Bot reviewed Nov 12, 2026
View reviewed changes
Comment thread crates/connectors/src/integrated/delta_table/merge/probe.rs Resolved

Uh oh!

There was an error while loading. Please reload this page.
@mythical-fred-oss

This comment was marked as resolved.

Sign in to view
@swanandx
[connectors] Delta merge: get the key ranges of all files at once
71f1e92 
To skip files, a merge flush uses the key range of each file. The key
range is the smallest and the largest key in the file. Before this
change, the flush got the ranges one file at a time. Now it gets the
ranges of all files with one call for each key column. ROW keys still
use one call for each file, because delta-rs returns only the first
field of a ROW.

Signed-off-by: Swanand Mulay <[email protected]>
@swanandx
swanandx force-pushed the merge-bounds-measure branch from 81e5225 to 71f1e92 Compare November 12, 2026 13:09
@mythical-fred-oss

mythical-fred-oss Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

LGTM.

What I ran: cargo test --locked -p dbsp_connectors --lib delta_table::merge passed 133 of 133, including the new log_bounds_match_the_per_file_view, row_bounds_stay_aligned_across_chunks and a_partition_key_column_takes_no_log_bound. cargo clippy --tests and rustfmt --check report nothing on the changed files.

What I checked against delta-rs 45a7cd6:

  • LogDataHandler::min_values/max_values and iter() both read self.data in the same order, so a file's row index points to its own bounds. The len == num_files guard protects against a mismatch.
  • round_up_ms gives the same result as ceil_datetime for µs timestamps, including negative values. Mins are not rounded on either path.
  • Binary and decimal columns, and files without stats, give no bound on both paths. Struct bounds still go through LogicalFileView, which applies the nested rounding.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mythical-fred-oss mythical-fred-oss[bot] mythical-fred-oss[bot] left review comments

At least 1 approving review is required to merge this pull request.

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@swanandx
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.