GeezerVison

[connectors] Update iceberg-rust to 4dd8c946 #7721

Merged

mythical-fred-oss Bot commented Oct 30, 2026

Copy link

Copy Markdown

Contributor

LGTM. Here is what I ran:

Check	Result
`cargo test -p feldera-types float_format`, `-p feldera-sqllib --lib variant::test`, `-p static-connectors json_output_keeps` (lock at serde_json 1.0.145)	pass
Same tests after a local `cargo update -p serde_json` to 1.0.154, which pulls in `zmij` (I restored the lockfile afterwards)	pass, so the formatter does override the new float output
`pre-commit run --files <changed>` (clippy, fmt, machete, crypto check)	pass

I looked for other places that write floats with plain serde_json into state or keys and found none outside tests. The JSON encoder's key path re-parses with arbitrary_precision, which keeps the original text. Number::from_f64 only shows up in tests.
The dependency is fine: ryu is Apache-2.0 OR BSL-1.0 (Boost) and was already a transitive dependency.
Missing (medium): no automated checkpoint→resume→delete test covers the failure the PR describes. The unit tests guard the text only, and the delete behaviour rests on the manual test. A TO_JSON(CAST(1e20 AS VARIANT)) row in a pipeline test that is deleted after a resume would lock it down.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

swanandx requested review from blp and mihaibudiu and removed request for blp

October 30, 2026 14:41

blp approved these changes

View reviewed changes

blp left a comment

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like Fred's idea of adding a way to warn if the normal serde_json gets used (I haven't tried anything like that before though).

I hadn't known about serde_json::ser::Formatter. Good to know.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

swanandx added this pull request to the merge queue

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks

Uh oh!

There was an error while loading. Please reload this page.

mythical-fred-oss Bot mentioned this pull request

CI: Runtime Integration Tests fail with k8s ENHANCE_YOUR_CALM (ResourceExhausted) during pipeline ConfigMap creation #7465

Open

mythical-fred-oss Bot commented Oct 30, 2026

Copy link

Copy Markdown

Contributor

Verdict: infrastructure, not this PR. Fairly confident.

The merge queue run's Runtime Integration Tests (gke-runners-amd64) job dropped the PR, but all 9 in-flight failures are unrelated tests (adhoc queries, adaptive joins, clock advance/timezone, checkpoint runtime upgrade, lateness) that failed at the same instant with the same error:

feldera.rest.errors.FelderaAPIError: FelderaError. Error message: upstream connect error or disconnect/reset before headers. reset reason: remote connection failure
Response Status: 503

Job: https://github.com/feldera/feldera/actions/runs/41137819254/job/124345935880
Tests started normally at 22:28:43, then every worker's request hung until 22:34:31 (~5:48 later) and all failed together — an ingress/control-plane connectivity blip on the shared amd64 test cluster, not something any test or the pipeline manager did.
This PR only touches crates/feldera-types, crates/sqllib, crates/static-connectors (stable JSON float formatting) — nothing related to pipeline provisioning or the test infra.

This is the same failure class as #7465 (two prior occurrences); I added this run as a third data point there.

Next step: re-queue — this should pass on a retry.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

mihaibudiu added this pull request to the merge queue

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks

Uh oh!

There was an error while loading. Please reload this page.

mythical-fred-oss Bot mentioned this pull request

CI: gke-runners-amd64 runner shut down mid-job during Rust Unit Tests artifact download #7761

Open

mythical-fred-oss Bot commented Oct 31, 2026

Copy link

Copy Markdown

Contributor

Verdict: CI infrastructure failure, not this PR. Confident.

The only job GitHub reported as failure — Unit Tests / Rust Unit Tests (gke-runners-amd64, x86_64, x86_64-unknown-linux-gnu) — never reached any test. It was mid-download of build artifacts when the runner itself died:

##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.

Job: https://github.com/feldera/feldera/actions/runs/41181351431/job/124483863469 (log line 379)
Every other job in the run shows cancelled — collateral from this one dropping out, not independent failures.
This PR only touches crates/feldera-types, crates/sqllib, crates/static-connectors (stable JSON float formatting) — unrelated to runner infra or artifact download.

Filed #7761 to track the runner eviction.

Next step: re-queue — this should pass on a retry.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

swanandx added this pull request to the merge queue

Merged via the queue into main with commit dbb5357

1 check passed

Uh oh!

There was an error while loading. Please reload this page.

swanandx deleted the json-float-ryu branch

October 31, 2026 15:46

swanandx added a commit that referenced this pull request


          [connectors] Update iceberg-rust to 4dd8c946

7365f03

The fork now matches upstream, which merged our reader stall fix.
The DataFusion integration crate is renamed back to iceberg-datafusion.
Its scans now also prune Parquet pages with the page index.
Nested columns in files without field ids now read as Spark reads them.
apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of
floats does not change because of #7735.

Signed-off-by: Swanand Mulay <[email protected]>

swanandx added a commit that referenced this pull request


          [connectors] Update iceberg-rust to 4dd8c946

7acb094

The fork now matches upstream, which merged our reader stall fix.
The DataFusion integration crate is renamed back to iceberg-datafusion.
Its snapshot reads now run as a DataSource, prune Parquet pages with
the page index, and unescape LIKE prefixes before pushdown. A local
check of 22 snapshot filters, with LIKE, ranges, IN, NOT and OR, gave
the right rows.
Nested columns in files without field ids now read as Spark reads them.
apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of
floats does not change because of #7735.

Signed-off-by: Swanand Mulay <[email protected]>

swanandx added a commit that referenced this pull request


          [connectors] Update iceberg-rust to 4dd8c946

2ec81bd

The fork now matches upstream, which merged our reader stall fix.
The DataFusion integration crate is renamed back to iceberg-datafusion.
Its snapshot reads now run as a DataSource, prune Parquet pages with
the page index, and unescape LIKE prefixes before pushdown. A local
check of 22 snapshot filters, with LIKE, ranges, IN, NOT and OR, gave
the right rows.
Nested columns in files without field ids now read as Spark reads them.
A new test reads a table made with add_files, and the changelog
notes the change.
apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of
floats does not change because of #7735.

Signed-off-by: Swanand Mulay <[email protected]>

swanandx added a commit that referenced this pull request


          [connectors] Update iceberg-rust to 4dd8c946

fb2aac7

The fork now matches upstream, which merged our reader stall fix.
The DataFusion integration crate is renamed back to iceberg-datafusion.
Its snapshot reads now run as a DataSource, prune Parquet pages with
the page index, and unescape LIKE prefixes before pushdown. A local
check of 22 snapshot filters, with LIKE, ranges, IN, NOT and OR, gave
the right rows.
Nested columns in files without field ids now read their real values.
A new test reads a table made with add_files, and the changelog
notes the change.
apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of
floats does not change because of #7735.

Signed-off-by: Swanand Mulay <[email protected]>

swanandx added a commit that referenced this pull request


          [connectors] Update iceberg-rust to 4dd8c946

b1c10c7

The fork now matches upstream, which merged our reader stall fix.
The DataFusion integration crate is renamed back to iceberg-datafusion.
Its snapshot reads now run as a DataSource, prune Parquet pages with
the page index, and unescape LIKE prefixes before pushdown. A local
check of 22 snapshot filters, with LIKE, ranges, IN, NOT and OR, gave
the right rows.
Nested columns in files without field ids now read their real values.
A new test reads a table made with add_files, and the changelog
notes the change.
apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of
floats does not change because of #7735.

Signed-off-by: Swanand Mulay <[email protected]>

swanandx added a commit that referenced this pull request


          [connectors] Update iceberg-rust to 4dd8c946

717ad65

The fork now matches upstream, which merged our reader stall fix.
The DataFusion integration crate is renamed back to iceberg-datafusion.
Its snapshot reads now run as a DataSource, prune Parquet pages with
the page index, and unescape LIKE prefixes before pushdown.
Nested columns in files without field ids now read their real values.
A new test reads such a table, with a page index, under 8 filters.
The changelog notes the change.
apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of
floats does not change because of #7735.

Signed-off-by: Swanand Mulay <[email protected]>

swanandx added a commit that referenced this pull request