GeezerVison

What I ran

Pitfall (medium): unrelated lockfile downgrades. Besides the iceberg and avro bumps, Cargo.lock re-resolves crates that have nothing to do with this change: hyper-util, quinn and quinn-udp drop from socket2 0.6.5 to 0.5.10, bindgen and prost-derive drop from itertools 0.13/0.14 to 0.10.5, and windows-sys/windows-targets/windows-core also go down. The hyper-util change affects every HTTP client in the product. It probably comes from how the lock was regenerated. Please run cargo update -p iceberg -p iceberg-datafusion … against the main lockfile, so the only changes are the ones the bump needs.

Observation: iceberg brings in new transitive deps (syn 3, wkb, geo-traits, parquet-geospatial, moka, typetag, ouroboros, zmij). All are MIT/Apache-2.0, and none are crypto crates. apache-avro goes from 0.21 to 0.22, but only iceberg uses that copy.

Not covered: no Feldera-side test checks that a scan does not stall when many data files are open; the only coverage is the upstream test. Otherwise LGTM.

What I ran: cargo check --locked -p feldera-iceberg passed and cargo test --locked -p feldera-iceberg passed (24/24). I confirmed 4dd8c946 is identical to apache/iceberg-rust main. The stall fix moved upstream (#3277) with its own tests there. All new transitive crates are MIT, Apache-2.0 or Zlib, and there is no new crypto.

fred is right, fixing this

should be fixed and safe to bump after #7735

What I ran

I could not run python/tests/platform here because it needs a pipeline-manager.

missing (high): no changelog entry for a declared breaking change. The PR body has a good table and says follow pipelines must be restarted from a new snapshot, but none of this is in docs.feldera.com/docs/changelog.md under ## Unreleased. Users only see the changelog. Please move the table and the restart advice there.

missing (medium): no test for the changed behaviour. test_iceberg_snapshot_all_types only writes files that have field ids. Neither case from the breaking-change table is tested:

1. A Parquet file without field ids, added with a name mapping (add_files), that has a ROW column → the struct now reads real values.
2. The same file without a name mapping → the struct now reads NULL instead of matching by position.
3. Follow mode: delete such a file after the upgrade and check that the retraction matches what was ingested.

Cases 1 and 2 fit next to the all-types test in python/tests/platform/test_iceberg_input.py. Write the file with pyarrow and no PARQUET:field_id metadata, then call table.add_files([...]) with and without schema.name-mapping.default set.

Review of the iceberg-rust bump to 4dd8c946

What I ran: cargo test --locked -p feldera-iceberg (debug) builds the renamed iceberg-datafusion and passes 24/24. The pre-commit run timed out here while setting up its hooks, so that check is still open.

Once the changelog entry with migration guidance and the tests for cases (1) and (2) are in, I'm happy to approve.

What I ran

New crates in the lock file (wkb, geo-traits, parquet-geospatial, zmij, darling 0.24): MIT/Apache, no crypto. ✅

Missing (medium): no tests for the new scan behaviour. The bump changes how DataFusion pushes down filters: LIKE prefixes are now unescaped, and Parquet pages are pruned with the page index. Both change which rows a snapshot_filter returns, but the only check is the manual 22-filter run. Please add at least a LIKE filter case. This one passes:

+/// A `LIKE` prefix filter is pushed down to the scan and must keep exactly the matching rows.
+#[engine_test(cfg(feature = "iceberg-tests-fs"))]
+pub fn localfs_input_test_like_filter(engine: &dyn PipelineTestEngine) {
+    iceberg_localfs_input_test(
+        engine,
+        100_000,
+        json!({ "snapshot_filter": "s LIKE 's1%' AND NOT (i IN (1, 10, 100))" }),
+        &|x| {
+            let i = x["i"].as_i64().unwrap();
+            x["s"].as_str().unwrap().starts_with("s1") && ![1, 10, 100].contains(&i)
+        },
+    );
+}

The test data has no literal _ or %, so an escaped prefix such as 's\_%' would need a row like that before it can catch the unescape fix itself.

Missing (medium): the other side of the breaking change has no test. The changelog says that a table with no name mapping now reads nested columns as NULL. Nothing tests that path. A --add-files variant that removes schema.name-mapping.default would pin it down.

Pitfall (low): the commit says apache-avro 0.22 needs serde_json >= 1.0.150, but the workspace still declares 1.0.132. Consider raising the floor.

Skipping the other points:

• LIKE test: without an escaped _ or % it passes before and after the bump, so it catches nothing.
• No-name-mapping test: our test fields are NOT NULL, so the read fails instead of giving NULL. Upstream tests this case (Only pull the full pipeline if refresh_version changed #3437).
• serde_json floor: the lock already has 1.0.154, and apache-avro forces 1.0.150 or later.

What I ran

missing (medium): tests for the new read-path behaviour. The commit says reads now unescape LIKE prefixes before pushdown and prune pages with the page index. No test in the repo covers either one; the PR only mentions a local check of 22 filters. I added a LIKE + NOT snapshot filter locally and it passes:

+#[engine_test(cfg(feature = "iceberg-tests-fs"))]
+pub fn localfs_input_test_like_filter(engine: &dyn PipelineTestEngine) {
+    iceberg_localfs_input_test(
+        engine,
+        100_000,
+        json!({ "snapshot_filter": "s LIKE 's12%' AND NOT (i < 1250)" }),
+        &|x| x["s"].as_str().unwrap().starts_with("s12") && x["i"].as_i64().unwrap() >= 1250,
+    );
+}

That test doesn't catch an unescape bug, though, because the test strings contain no _ or %. A useful regression test needs a --like-strings style flag in create_test_table_s3.py that writes values like a_b/a%b, plus a filter such as s LIKE 'a\_b%'. Please commit the 22-filter check as a test.

Cases not covered by the PR's tests

• add_files table without a name mapping. The changelog says nested columns read NULL; that is untested.
• add_files data in follow mode, including a delete of the added file, which is the scenario the changelog warns about.

What I ran

Missing tests (medium)

1. The commit says LIKE prefixes are now unescaped before pushdown, but s LIKE 's4999%' has nothing to unescape. I added an escaped-prefix case locally and it passes. Please add it:

   -    let filters: [(&str, Keep); 8] = [
   +    let filters: [(&str, Keep); 9] = [
   @@
            ("b AND i < 1000", &|x| i(x) % 2 == 1 && i(x) < 1000),
   +        // Escaped prefix: pushdown must unescape `\9` to `9` before pruning.
   +        ("s LIKE 's4\\9%'", &|x| x["s"].as_str().unwrap().starts_with("s49")),
        ];

2. Nothing tests the other half of the behaviour change: a file without field ids and no name mapping, where nested columns now read as NULL instead of matching by position. One more create_test_table_s3.py flag that removes schema.name-mapping.default after add_files would pin this down.

I didn't check the upstream iceberg-rust diff line by line. I relied on the fork now matching upstream and on the test results above.

What I ran

Gates: tests, docs (changelog) and licences (new transitive crates are MIT/Apache) pass. The apache-avro 0.22 bump comes in only through iceberg; Feldera's Avro format stays on 0.18.

Missing (medium): test cases not covered

1. The other side of the behaviour change: a file without field ids in a table without a name mapping now reads NULL. It is in the changelog but no test pins it. Strip schema.name-mapping.default after add_files and assert NULL nested columns.
2. An ordered read (timestamp_column) of the add_files table. Only unordered snapshot_filter reads are tested.
3. A distributed read (share.rs) of the add_files table. The name mapping travels with each FileScanTask, so it should work, but no test checks it.

Tech-debt (low): share.rs builds its ArrowReader without with_row_selection_enabled, so distributed reads still skip page pruning. The single-host DataSource path now uses it.

Verdict: flaky test, not this PR — merge queue should pass on re-queue

Failed job: Adapter Tests / Test Adapters (amd64)

Already tracked as a known flake: #7820 (filed yesterday from a different PR, same assertion, same line, arm64 runner). This run reproduces it on amd64 with the same preceding burst of client error: nats: IO error reconnects before the 5s WaitForConsumerCount timeout — added a confirming comment there.

Next step: no fix needed here; re-queueing the PR should pass. If it fails again on this same test, that's issue #7820, not a new problem.

Verdict: flaky test, not this PR. Merge queue run failed in Unit Tests / Rust Unit Tests (aarch64), unrelated to the iceberg bump.

This PR's diff only touches crates/iceberg, crates/connectors/src/test/iceberg.rs and crates/adapterlib — nothing in crates/dbsp. The panic is a registration/teardown race in the multihost exchange test harness, already tracked in #7800 (same UnknownExchange(0) message, same harness, a different test in the same suite hit it on a prior PR). Added this run as a repeat occurrence there rather than filing a new issue.

Next step: re-queueing should pass — nothing in the failure touches code this PR changed.

If the fork now matches upstream, why are we still using the fork?

If the fork now matches upstream, why are we still using the fork?

upstream release process is quite slow, and lot of times we want the latest main ( say to use bumped dependecies ) or have some patches until our PRs get accepted upstream. hence sticking to fork