GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — [adapters] Delta: let a filter name a nested column-mapped field by blp · Pull Request #7669 · feldera/feldera

Conversation

blp commented Oct 27, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Member

Linked Issue

Fixes #7657.

When the Delta input connector reads Parquet files itself, it reads them with physical col-<uuid> names at every level. project_physical_to_logical put back only the top-level logical names, so a filter, snapshot_filter, cdc_delete_filter, or cdc_order_by that named a nested field failed with Field ... not found in struct. The nested names were put back only per batch, after the query had run.

Read path	Before
Snapshot, `distributed: false` (delta-rs provider)	works
Snapshot, `distributed: true`	fails (the issue)
Follow, add/remove files	fails
Follow, change data feed	fails
CDC mode	fails

project_physical_to_logical now also puts back nested names, through a scalar function (RelabelNestedUdf) that applies the existing relabel_array to each column that holds column-mapped nested fields. It is zero-copy and keeps struct nulls.

A predicate on a nested field cannot push down through the function, so it gets no row-group pruning. Before this change those predicates failed, so no pruning is lost.

Describe Manual Test Plan

New Rust tests for follow, CDC, and change data feed with nested filters. Each fails without the fix with the error from the issue, and passes with it.
cargo test -p dbsp_adapters --lib delta_table: 430 passed.
cargo clippy -p dbsp_adapters --all-targets: clean.
New platform test test_delta_input_column_mapping_nested_dist_filter for the distributed snapshot case. I have not run it locally; CI will.

Checklist

Unit tests added/updated
Integration tests added/updated
Documentation updated
Changelog updated

Breaking Changes?

None.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

blp added bug

Something isn't working

connectors

Issues related to the adapters/connectors crate

user-reported

Reported by a user or customer

multihost Related to multihost or distributed pipelines labels

Oct 27, 2026

blp self-assigned this

Oct 27, 2026

blp requested review from anandbraman and swanandx

October 27, 2026 23:48

blp commented Oct 27, 2026

Copy link

Copy Markdown

Member Author

@swanandx This is outside my expertise so I wasn't able to effectively review it myself. I think you know more about this.

@anandbraman Does this make sense relative to the bug report?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

swanandx approved these changes

Oct 28, 2026

View reviewed changes

swanandx left a comment

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

we can now remove the nested_map variable and the relabel_nested_columns(&batch, &nested_map) call in execute_df_inner as it shouldn't be required with this fix in place. RelabelNestedUdf already renames nested fields before the query runs, so that step no longer changes anything

also, we should rebase and Sign off the commit before merging

thank you so much for the fix!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

Cargo.toml

		# - "round-robin pre-split unpartitioned scans": reads a many-file table in
		# parallel (#2042). `DELTA_SCAN_ROUND_ROBIN=false` turns it off.
		# Upstream now, do not reapply: the delta-io/delta-rs#4692 backport.
		deltalake = { git = "https://github.com/feldera/delta-rs.git", rev = "9d996253cc60e5a6f35b2be5aef1cce89e08d951" }

swanandx Oct 28, 2026

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why we changing this? rebase should solve this

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

blp Oct 28, 2026

Copy link

Copy Markdown

Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, that was an artifact of an old base. The branch is now rebased onto main as one signed-off commit, and it no longer touches Cargo.toml or Cargo.lock. That commit also drops the per-batch relabel_nested_columns call and nested_map from execute_df_inner, as you suggested. The delta_table tests pass.

(Claude, on behalf of @blp)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

blp force-pushed the delta-nested-filter branch 2 times, most recently from 0939660 to b17f5af Compare

October 28, 2026 16:39


          [connectors] Delta: let a filter name a nested column-mapped field.

e89cf5d

When the connector reads Parquet files itself (distributed snapshot,
follow, change data feed, CDC), it reads them with physical `col-<uuid>`
names at every level. `project_physical_to_logical` put back only the
top-level logical names, so a `filter`, `snapshot_filter`,
`cdc_delete_filter`, or `cdc_order_by` that named a nested field failed
with "Field ... not found in struct". The nested names were put back
only per batch, after the query had run.

`project_physical_to_logical` now also puts back nested names, through a
scalar function that applies `relabel_array` to each column that holds
column-mapped nested fields. This makes the per-batch relabel in
`execute_df_inner` redundant, so it is gone.

Fixes #7657.

Signed-off-by: Ben Pfaff <[email protected]>

blp force-pushed the delta-nested-filter branch from b17f5af to e89cf5d Compare

October 28, 2026 16:44

blp added this pull request to the merge queue

Nov 2, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks

Nov 2, 2026

Uh oh!

There was an error while loading. Please reload this page.

mythical-fred-oss Bot commented Nov 2, 2026

Copy link

Copy Markdown

Contributor

Verdict: PR bug, high confidence. The new test test_delta_input_column_mapping_nested_distributed_filter is 57 chars, 2 chars over the 55-char budget that python/tests/unit/test_pipeline_name_length.py enforces (62-char pipeline name limit minus the 7-char CI tag).

Evidence


Job	Unit Tests / Rust Unit Tests (aarch64)
Failing test	`tests/unit/test_pipeline_name_length.py::test_test_pipeline_names_fit_the_limit`
Log	`logs/125474565199.log:1381-1393`

AssertionError: these test names leave no room for the 7-char CI tag within the 62-char pipeline name limit:
.../test_delta_input_column_mapping_nested.py: 'test_delta_input_column_mapping_nested_distributed_filter' is 57 chars, 2 over

Fix: renamed the test to test_delta_input_column_mapping_nested_dist_filter (50 chars). Pushed on fred/observe-7669-41500486043 as 8f1a8bb, verified with uv run --locked pytest tests/unit/test_pipeline_name_length.py → 3 passed.

Take it with:

git fetch origin fred/observe-7669-41500486043
git cherry-pick 8f1a8bb57

Compare: delta-nested-filter...fred/observe-7669-41500486043

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.


          [python] Shorten nested Delta filter test name to fit the pipeline na…

45d6bb2

…me limit.

Signed-off-by: Ben Pfaff <[email protected]>

blp commented Nov 2, 2026

Copy link

Copy Markdown

Member Author

Fixed in 45d6bb2: renamed the test to test_delta_input_column_mapping_nested_dist_filter (50 chars), the same name you suggested. pytest tests/unit/test_pipeline_name_length.py now passes. The PR description uses the new name.

(Claude, on behalf of @blp)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

blp added this pull request to the merge queue

Nov 2, 2026

Merged via the queue into main with commit 6edc0c3

Nov 3, 2026

1 check passed

Uh oh!

There was an error while loading. Please reload this page.

blp deleted the delta-nested-filter branch

November 3, 2026 01:56

Reviewers

swanandx

swanandx approved these changes

anandbraman

Awaiting requested review from anandbraman

Uh oh!

There was an error while loading. Please reload this page.

Assignees

blp

Labels

bug

Something isn't working

connectors

Issues related to the adapters/connectors crate

multihost

Related to multihost or distributed pipelines

user-reported

Reported by a user or customer

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[connectors] Delta: nested column filter fails when distributed is true

Uh oh!

There was an error while loading. Please reload this page.

2 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

blp commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Linked Issue

Describe Manual Test Plan

Checklist

Breaking Changes?

Uh oh!

blp commented Oct 27, 2026

Uh oh!

swanandx left a comment

Choose a reason for hiding this comment

Uh oh!

swanandx Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

blp Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mythical-fred-oss Bot commented Nov 2, 2026

Uh oh!

blp commented Nov 2, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

blp commented Oct 27, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.