Expected Behavior
When the online store refuses a write during feast materialize with the Ray compute engine, the materialization fails (job status ERROR and FeatureStore.materialize raises), the way it does with the Local engine on the same setup.
Current Behavior
With batch_engine: ray.engine, the job reports SUCCEEDED, materialize returns normally, and get_online_features keeps serving the previous values. Three handlers absorb the error:
RayWriteNode.execute (sdk/python/feast/infra/compute_engines/ray/nodes.py, 864): the batch writer is decorated with @safe_batch_processor, which logs and returns the batch on any exception (ray/utils.py 210-212). This is the path FeatureStore.materialize takes.
write_to_online_store (ray/utils.py, handler 82-83) and write_to_online_store_from_ray_ds (177-180) log the error and do not raise. These are on the legacy from_offline_store path.
When Feast starts Ray itself, log_to_driver is off unless enable_ray_logging is set (sdk/python/feast/infra/ray_initializer.py 668-682), so the error line does not reach the console either; it only appears in a Ray worker log under /tmp/ray.
RayReadNode in the same file re-raises after logging, and the Local, Spark and Flink engines let the error through.
Something this is hiding
tests/component/ray/test_compute.py::test_ray_compute_engine_materialize passes online_store=MagicMock() and asserts SUCCEEDED. Run as-is with a real SQLite online store in place of the mock, it still reports SUCCEEDED, but the online table holds 0 rows: the aggregated BatchFeatureView's write raises ValueError: Column conv_rate not found in table inside _convert_arrow_to_proto, and the handlers above absorb it. With the aggregate column names in the schema instead, the next error is KeyError: 'Field "event_timestamp" does not exist in schema'. So it looks like the Ray engine cannot currently write an aggregated BatchFeatureView to the online store at all, which matches the TODO at feast/utils.py 577.
Steps to reproduce
On master (386759c), a feature repo with a parquet FileSource, the file offline store, the SQLite online store and batch_engine: {type: ray.engine, max_workers: 1}:
materialize once with the online database writable: values 0.8 / 0.7 are served.
- Make the online database file read-only (
chmod 0444), add newer rows (0.1 / 0.2) to the source, and materialize again. SQLite raises attempt to write a readonly database inside the worker.
- The Ray job status is SUCCEEDED,
materialize returns normally, and get_online_features still returns 0.8 / 0.7.
- The Local engine on the same read-only database reports ERROR and
materialize raises.
The read-only file is a deliberate stand-in for an online store that refuses writes; I have not seen this in a production deployment.
Specifications
- Version: master at 386759c (15 Oct 2026)
- Platform: macOS, Python 3.10.7, Ray 2.58.0
- Subsystem: Ray compute engine (
infra/compute_engines/ray)
Possible Solution
A small change (+9/-2 across ray/utils.py and ray/nodes.py): re-raise after the existing log in the two write functions, and take the write node's batch function out of safe_batch_processor, keeping its empty-batch early return. safe_batch_processor itself is unchanged (its swallow is pinned by test_exception_in_func_returns_original_batch, and other nodes use it). With it, five new tests fail on master and pass, four controls pass both ways, and the existing Ray component tests stay at 75 passed (non-integration) and 13 of 14 (integration).
The one integration test that turns red is test_ray_compute_engine_materialize, for the reason above. That seems like your call, so I have not changed it. A few options:
- assert ERROR there and track aggregated BatchFeatureView online writes on Ray as their own issue,
- drop the aggregations from that test so it tests a plain materialization, or
- fix the aggregated write first.
The intended behaviour change is that a failed online write fails the materialization instead of reporting success, so a pipeline that has been writing nothing will start failing. Happy to open a PR with the change and tests once you have a preference on the test.
This issue was written with AI assistance (Claude Code); the tests above were run before filing.