GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — fix: Stop the Ray engine from hiding materialization errors by patelchaitany · Pull Request #7026 · feast-dev/feast

Conversation

patelchaitany commented Oct 16, 2026

Copy link

Copy Markdown

Contributor

What this PR does / why we need it:

The Ray compute engine hid errors during materialization. A job could report SUCCEEDED even when nothing was written to the online store. Fixing that showed two more bugs that the hidden errors were covering up. This PR fixes all three, one commit each.

1. A failed online write was reported as success (#7024)
When the online store refused a write, Ray logged the error and still reported SUCCEEDED, so users kept getting old values. Now the error is raised again after it is logged, and the write step is no longer wrapped in safe_batch_processor. A failed write now fails the job, the same as with the Local engine.

2. An aggregated BatchFeatureView could never be written to the online store
The Ray aggregation step dropped the event timestamp and created timestamp columns. The online write needs them, so it always failed (bug 1 hid this). Now the aggregation keeps the latest event and created timestamp for each entity.

3. The UDF ran at a different point in each Ray path
Materialization ran the UDF on raw rows before aggregation, but historical retrieval ran it after aggregation. So a UDF could only work in one path. In the other path its error was hidden by safe_batch_processor, and users got values that were never transformed. Now historical retrieval also runs the UDF before aggregation. This matches the compute engine docs and the Local and Spark engines. The transformation step no longer hides UDF errors.

Behavior changes to note

A materialization that has been failing quietly will now report ERROR.
On Ray, a UDF on an aggregated BatchFeatureView that used aggregated column names (for example sum_conv_rate) must now use the raw column names (for example conv_rate).

safe_batch_processor itself is unchanged; other nodes still use it.

Which issue(s) this PR fixes:

Fixes #7024

Checks

I've made sure the tests are passing.
My commits are signed off (git commit -s)
My PR title follows conventional commits format

Testing Strategy

Unit tests
Integration tests
Manual tests
Testing is not required for this change

Tests in sdk/python/tests/component/ray:

New test_ray_write_node_raises_when_online_write_fails: fails on master, passes with this PR.
test_ray_compute_engine_materialize: now uses the aggregated feature names in the schema, and reads the values back from the SQLite online store. Its UDF was removed, because on master it never actually ran (its error was hidden).
test_ray_compute_engine_get_historical_features: the UDF now uses the raw columns. Expected values are unchanged. Fails on master, passes with this PR.

Ray component tests: 76 passed (non-integration), 14 passed (--integration).

Misc

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

patelchaitany requested a review from a team as a code owner

October 16, 2026 11:13

codecov-commenter commented Oct 16, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 0% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 50.98%. Comparing base (68c5d54) to head (f446423).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
...dk/python/feast/infra/compute_engines/ray/nodes.py	0.00%	11 Missing ⚠️
...feast/infra/compute_engines/ray/feature_builder.py	0.00%	2 Missing ⚠️
...dk/python/feast/infra/compute_engines/ray/utils.py	0.00%	2 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7026      +/-   ##
==========================================
- Coverage   50.99%   50.98%   -0.01%     
==========================================
  Files         455      455              
  Lines       56979    56990      +11     
  Branches     8315     8320       +5     
==========================================
  Hits        29059    29059              
- Misses      25935    25946      +11     
  Partials     1985     1985

Flag	Coverage Δ
go-feature-server	`30.58% <ø> (ø)`
python-unit	`52.45% <0.00%> (-0.02%)`	⬇️

Files with missing lines	Coverage Δ
...feast/infra/compute_engines/ray/feature_builder.py	`0.00% <0.00%> (ø)`
...dk/python/feast/infra/compute_engines/ray/utils.py	`0.00% <0.00%> (ø)`
...dk/python/feast/infra/compute_engines/ray/nodes.py	`0.00% <0.00%> (ø)`

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 68c5d54...f446423. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

haoxu0 reviewed

Oct 16, 2026

View reviewed changes

Comment thread

sdk/python/feast/infra/compute_engines/ray/feature_builder.py

		last_node = self.build_source_node(self.feature_view)

		# 2. Apply filters (TTL, custom filters) BEFORE aggregation
		# 2. Apply transformations to the raw source data, as materialization does

haoxu0 Oct 16, 2026

Copy link

Copy Markdown

Collaborator

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this order change matching other compute engines?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

patelchaitany Oct 17, 2026

Copy link

Copy Markdown

Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Local, Spark and Flink all run the UDF before aggregation (shared FeatureBuilder._build). Only Ray historical retrieval ran it after, and this fixes that.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

patelchaitany force-pushed the fix/ray-materialization-errors branch from e3483a6 to 5feefae Compare

October 17, 2026 14:27

patelchaitany requested a review from haoxu0

October 17, 2026 14:47

ntkathole added the ok-to-test label

Oct 18, 2026

ntkathole reviewed

Oct 18, 2026

View reviewed changes

Comment thread

sdk/python/feast/infra/compute_engines/ray/nodes.py Outdated

		# Not wrapped in safe_batch_processor: a failed write must fail the job.
		def write_batch_with_serialized_artifacts(batch: pd.DataFrame) -> pd.DataFrame:
		"""Write each batch using pre-serialized artifacts."""
		if batch.empty:

ntkathole Oct 18, 2026

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for consistency over other code if _is_empty_batch(batch):

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

patelchaitany reacted with thumbs up emoji

All reactions

👍 1 reaction

ntkathole reviewed

Oct 18, 2026

View reviewed changes

Comment thread

sdk/python/feast/infra/compute_engines/ray/nodes.py


		# Combine aggregation results
		if agg_results:
		# Keep the latest timestamps per group; the online write needs them.

ntkathole Oct 18, 2026

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tiled aggregation doesn’t preserve created_timestamp_col (fix only touched the standard _fallback_pandas_aggregation).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

patelchaitany reacted with thumbs up emoji

All reactions

👍 1 reaction

patelchaitany force-pushed the fix/ray-materialization-errors branch 2 times, most recently from 988a73f to f7e40d2 Compare

October 18, 2026 13:56

patelchaitany requested a review from ntkathole

October 18, 2026 14:19

ntkathole approved these changes

Oct 18, 2026

View reviewed changes

patelchaitany added 3 commits

October 18, 2026 20:01


          fix: Fail Ray materialization when the online store write fails

e0e51f7

When the online store refused a write, the Ray engine only logged the
error and still reported the job as SUCCEEDED. Users kept getting old
feature values and had no sign that anything went wrong.

Now the error is raised again after it is logged, and the write step is
no longer wrapped in safe_batch_processor, which hid every error. A
failed write now fails the job, the same as with the Local engine.

Fixes feast-dev#7024

Signed-off-by: Chaitany Patel <[email protected]>


          fix: Keep timestamps when Ray aggregates a BatchFeatureView

7936ce4

The Ray aggregation step kept only the entity keys and the aggregated
values. It dropped the event timestamp and created timestamp columns.
The online store write needs these columns, so writing an aggregated
BatchFeatureView to the online store always failed. The previous commit
made this failure visible.

Now the aggregation keeps the latest event timestamp and created
timestamp for each entity.

Signed-off-by: Chaitany Patel <[email protected]>


          fix: Run the Ray UDF before aggregation in historical retrieval

f446423

In materialization, the Ray engine ran the UDF on raw rows before
aggregation. In historical retrieval, it ran the UDF after aggregation.
So a UDF could only work in one of the two paths. In the other path it
raised an error, and safe_batch_processor hid that error, so users got
values that were never transformed.

Now historical retrieval also runs the UDF on raw rows before
aggregation. This matches the compute engine docs and the Local and
Spark engines. The transformation step is no longer wrapped in
safe_batch_processor, so a UDF error now fails the job.

A UDF that used aggregated column names (for example sum_conv_rate)
must now use the raw column names (for example conv_rate).

Signed-off-by: Chaitany Patel <[email protected]>

ntkathole force-pushed the fix/ray-materialization-errors branch from f7e40d2 to f446423 Compare

October 18, 2026 14:31

ntkathole merged commit 7903e2e into feast-dev:master

Oct 18, 2026

24 of 25 checks passed

Uh oh!

There was an error while loading. Please reload this page.

ericdelorefice commented Oct 18, 2026

Copy link

Copy Markdown

Contributor

Thanks @patelchaitany for picking this up so quickly, and for fixing the aggregation and UDF problems it was hiding. And thanks @ntkathole for the review and merge.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

franciscojavierarceo pushed a commit that referenced this pull request

Nov 10, 2026


          chore(release): release 0.68.0

fb5e5d5

# [0.68.0](v0.67.0...v0.68.0) (2026-11-10)

### Bug Fixes

* add watch verb on clusterroles and clusterrolebindings for feast-operator ([#7062](#7062)) ([b6bdda4](b6bdda4))
* Apply on demand feature views in SparkRetrievalJob.to_spark_df() ([#6993](#6993)) ([f416309](f416309))
* Break event timestamp ties by created timestamp in Milvus write batches ([#7084](#7084)) ([bbaca76](bbaca76))
* Break event timestamp ties by created timestamp in the Dask offline store ([#6952](#6952)) ([eb93b08](eb93b08))
* Build the Go feature server image in two stages ([1e5e90c](1e5e90c)), closes [#7060](#7060)
* Fall back to the serialized body when UDF source misses globals ([#6816](#6816)) ([f61079f](f61079f))
* Install patched libbrotli1 to address DSA-6582-1 ([#7096](#7096)) ([d458393](d458393))
* Keep description, tags and nested types of ODFV schema fields ([8ef6e75](8ef6e75))
* Keep entity columns out of StreamFeatureView proto features ([#7033](#7033)) ([d520584](d520584))
* Keep the metrics server running when a Gunicorn worker exits ([#7012](#7012)) ([68c5d54](68c5d54)), closes [#7011](#7011)
* Key registry proto caches on the proto, not its id() ([#7085](#7085)) ([147adad](147adad))
* Log ODFV materialization skips instead of printing them ([#7117](#7117)) ([8fdcee7](8fdcee7))
* Pin build dependencies of source builds to the CI lock ([#7104](#7104)) ([dec2f24](dec2f24))
* Populate spec.project in registry REST feature view responses (#… ([#7037](#7037)) ([e304a84](e304a84))
* Preserve colliding source timestamps in Dask retrieval ([#7071](#7071)) ([1396c88](1396c88))
* Preserve Dask historical requests without eligible features ([#7000](#7000)) ([fff09f7](fff09f7))
* Read None elements of UUID and Decimal lists as None ([#7039](#7039)) ([03f30fe](03f30fe))
* Retire the support of Python3.10 ([#7101](#7101)) ([a56766e](a56766e)), closes [#7104](#7104) [#7089](#7089) [#7106](#7106) [#7105](#7105)
* Serialize UUID and TimeUuid entity keys ([#7032](#7032)) ([0c845a1](0c845a1))
* Stop the Ray engine from hiding materialization errors ([#7026](#7026)) ([7903e2e](7903e2e)), closes [#7024](#7024)
* **ui:** stop treating the monitoring baseline as the latest window ([#7056](#7056)) ([0f01802](0f01802)), closes [#7054](#7054)
* Use date-only bounds for partition filters built from string timestamps ([#7116](#7116)) ([b1f2168](b1f2168))
* Validate entity column types in feast dbt import ([#7067](#7067)) ([abca80c](abca80c)), closes [#5876](#5876)
* Warm up the whole DynamoDB connection pool ([#6985](#6985)) ([8ce548c](8ce548c))

### Features

* Add --gc-freeze option to feast serve ([#7089](#7089)) ([aad0c15](aad0c15))
* Add native complex types and scalar index tags to the Milvus online store ([#7120](#7120)) ([f7068d6](f7068d6))
* Add num_partitions and partition_key_isolation to the Milvus online store ([#7083](#7083)) ([68de84e](68de84e)), closes [#6940](#6940)
* Add Pinecone online store integration ([#6585](#6585)) ([7a84f2a](7a84f2a))
* Add PolarisSource and PolarisProvider for Apache Polaris ([#7006](#7006)) ([386759c](386759c))
* Add row deletion to the Milvus online store ([#7082](#7082)) ([d3e2041](d3e2041))
* Add upstream FeatureView lineage support for PushSource ([#6853](#6853)) ([c9b9d72](c9b9d72))
* Optional nameless pre-computed vectors with a schema fingerprint ([#7077](#7077)) ([9420fac](9420fac))

### Performance Improvements

* Scan Lance sources lazily in the DuckDB offline store ([1299e4e](1299e4e))

patelchaitany mentioned this pull request

Nov 11, 2026

chore: Sync upstream feast-dev/feast stable (0.68.0) opendatahub-io/feast#291

Merged

shuchu pushed a commit to shuchu/feast that referenced this pull request

Nov 11, 2026


          chore(release): release 0.68.0

5b94c9e

# [0.68.0](feast-dev/feast@v0.67.0...v0.68.0) (2026-11-10)

### Bug Fixes

* add watch verb on clusterroles and clusterrolebindings for feast-operator ([feast-dev#7062](feast-dev#7062)) ([b6bdda4](feast-dev@b6bdda4))
* Apply on demand feature views in SparkRetrievalJob.to_spark_df() ([feast-dev#6993](feast-dev#6993)) ([f416309](feast-dev@f416309))
* Break event timestamp ties by created timestamp in Milvus write batches ([feast-dev#7084](feast-dev#7084)) ([bbaca76](feast-dev@bbaca76))
* Break event timestamp ties by created timestamp in the Dask offline store ([feast-dev#6952](feast-dev#6952)) ([eb93b08](feast-dev@eb93b08))
* Build the Go feature server image in two stages ([1e5e90c](feast-dev@1e5e90c)), closes [feast-dev#7060](feast-dev#7060)
* Fall back to the serialized body when UDF source misses globals ([feast-dev#6816](feast-dev#6816)) ([f61079f](feast-dev@f61079f))
* Install patched libbrotli1 to address DSA-6582-1 ([feast-dev#7096](feast-dev#7096)) ([d458393](feast-dev@d458393))
* Keep description, tags and nested types of ODFV schema fields ([8ef6e75](feast-dev@8ef6e75))
* Keep entity columns out of StreamFeatureView proto features ([feast-dev#7033](feast-dev#7033)) ([d520584](feast-dev@d520584))
* Keep the metrics server running when a Gunicorn worker exits ([feast-dev#7012](feast-dev#7012)) ([68c5d54](feast-dev@68c5d54)), closes [feast-dev#7011](feast-dev#7011)
* Key registry proto caches on the proto, not its id() ([feast-dev#7085](feast-dev#7085)) ([147adad](feast-dev@147adad))
* Log ODFV materialization skips instead of printing them ([feast-dev#7117](feast-dev#7117)) ([8fdcee7](feast-dev@8fdcee7))
* Pin build dependencies of source builds to the CI lock ([feast-dev#7104](feast-dev#7104)) ([dec2f24](feast-dev@dec2f24))
* Populate spec.project in registry REST feature view responses (#… ([feast-dev#7037](feast-dev#7037)) ([e304a84](feast-dev@e304a84))
* Preserve colliding source timestamps in Dask retrieval ([feast-dev#7071](feast-dev#7071)) ([1396c88](feast-dev@1396c88))
* Preserve Dask historical requests without eligible features ([feast-dev#7000](feast-dev#7000)) ([fff09f7](feast-dev@fff09f7))
* Read None elements of UUID and Decimal lists as None ([feast-dev#7039](feast-dev#7039)) ([03f30fe](feast-dev@03f30fe))
* Retire the support of Python3.10 ([feast-dev#7101](feast-dev#7101)) ([a56766e](feast-dev@a56766e)), closes [feast-dev#7104](feast-dev#7104) [feast-dev#7089](feast-dev#7089) [feast-dev#7106](feast-dev#7106) [feast-dev#7105](feast-dev#7105)
* Serialize UUID and TimeUuid entity keys ([feast-dev#7032](feast-dev#7032)) ([0c845a1](feast-dev@0c845a1))
* Stop the Ray engine from hiding materialization errors ([feast-dev#7026](feast-dev#7026)) ([7903e2e](feast-dev@7903e2e)), closes [feast-dev#7024](feast-dev#7024)
* **ui:** stop treating the monitoring baseline as the latest window ([feast-dev#7056](feast-dev#7056)) ([0f01802](feast-dev@0f01802)), closes [feast-dev#7054](feast-dev#7054)
* Use date-only bounds for partition filters built from string timestamps ([feast-dev#7116](feast-dev#7116)) ([b1f2168](feast-dev@b1f2168))
* Validate entity column types in feast dbt import ([feast-dev#7067](feast-dev#7067)) ([abca80c](feast-dev@abca80c)), closes [feast-dev#5876](feast-dev#5876)
* Warm up the whole DynamoDB connection pool ([feast-dev#6985](feast-dev#6985)) ([8ce548c](feast-dev@8ce548c))

### Features

* Add --gc-freeze option to feast serve ([feast-dev#7089](feast-dev#7089)) ([aad0c15](feast-dev@aad0c15))
* Add native complex types and scalar index tags to the Milvus online store ([feast-dev#7120](feast-dev#7120)) ([f7068d6](feast-dev@f7068d6))
* Add num_partitions and partition_key_isolation to the Milvus online store ([feast-dev#7083](feast-dev#7083)) ([68de84e](feast-dev@68de84e)), closes [feast-dev#6940](feast-dev#6940)
* Add Pinecone online store integration ([feast-dev#6585](feast-dev#6585)) ([7a84f2a](feast-dev@7a84f2a))
* Add PolarisSource and PolarisProvider for Apache Polaris ([feast-dev#7006](feast-dev#7006)) ([386759c](feast-dev@386759c))
* Add row deletion to the Milvus online store ([feast-dev#7082](feast-dev#7082)) ([d3e2041](feast-dev@d3e2041))
* Add upstream FeatureView lineage support for PushSource ([feast-dev#6853](feast-dev#6853)) ([c9b9d72](feast-dev@c9b9d72))
* Optional nameless pre-computed vectors with a schema fingerprint ([feast-dev#7077](feast-dev#7077)) ([9420fac](feast-dev@9420fac))

### Performance Improvements

* Scan Lance sources lazily in the DuckDB offline store ([1299e4e](feast-dev@1299e4e))

Signed-off-by: Shuchu Han <[email protected]>

rhods-devops-app Bot mentioned this pull request

Nov 12, 2026

Automated Sync from main to stable red-hat-data-services/feast#630

Open

Reviewers

ntkathole

ntkathole approved these changes

haoxu0

Awaiting requested review from haoxu0

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

ok-to-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Ray compute engine reports a failed online-store write as a successful materialization

Uh oh!

There was an error while loading. Please reload this page.

5 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

patelchaitany commented Oct 16, 2026

What this PR does / why we need it:

Which issue(s) this PR fixes:

Checks

Testing Strategy

Misc

Uh oh!

codecov-commenter commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

haoxu0 Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

patelchaitany Oct 17, 2026

Choose a reason for hiding this comment

Uh oh!

ntkathole Oct 18, 2026

Choose a reason for hiding this comment

Uh oh!

ntkathole Oct 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ericdelorefice commented Oct 18, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov-commenter commented Oct 16, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.