GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Ray compute engine reports a failed online-store write as a successful materialization · Issue #7024 · feast-dev/feast

Ray compute engine reports a failed online-store write as a successful materialization #7024

New issue
Copy link
New issue
Copy link
Closed
#7026
Closed
Ray compute engine reports a failed online-store write as a successful materialization#7024
#7026
Copy link

Description

@ericdelorefice
ericdelorefice
opened on Oct 15, 2026
Issue body actions

Expected Behavior

When the online store refuses a write during feast materialize with the Ray compute engine, the materialization fails (job status ERROR and FeatureStore.materialize raises), the way it does with the Local engine on the same setup.

Current Behavior

With batch_engine: ray.engine, the job reports SUCCEEDED, materialize returns normally, and get_online_features keeps serving the previous values. Three handlers absorb the error:

  • RayWriteNode.execute (sdk/python/feast/infra/compute_engines/ray/nodes.py, 864): the batch writer is decorated with @safe_batch_processor, which logs and returns the batch on any exception (ray/utils.py 210-212). This is the path FeatureStore.materialize takes.
  • write_to_online_store (ray/utils.py, handler 82-83) and write_to_online_store_from_ray_ds (177-180) log the error and do not raise. These are on the legacy from_offline_store path.

When Feast starts Ray itself, log_to_driver is off unless enable_ray_logging is set (sdk/python/feast/infra/ray_initializer.py 668-682), so the error line does not reach the console either; it only appears in a Ray worker log under /tmp/ray.

RayReadNode in the same file re-raises after logging, and the Local, Spark and Flink engines let the error through.

Something this is hiding

tests/component/ray/test_compute.py::test_ray_compute_engine_materialize passes online_store=MagicMock() and asserts SUCCEEDED. Run as-is with a real SQLite online store in place of the mock, it still reports SUCCEEDED, but the online table holds 0 rows: the aggregated BatchFeatureView's write raises ValueError: Column conv_rate not found in table inside _convert_arrow_to_proto, and the handlers above absorb it. With the aggregate column names in the schema instead, the next error is KeyError: 'Field "event_timestamp" does not exist in schema'. So it looks like the Ray engine cannot currently write an aggregated BatchFeatureView to the online store at all, which matches the TODO at feast/utils.py 577.

Steps to reproduce

On master (386759c), a feature repo with a parquet FileSource, the file offline store, the SQLite online store and batch_engine: {type: ray.engine, max_workers: 1}:

  1. materialize once with the online database writable: values 0.8 / 0.7 are served.
  2. Make the online database file read-only (chmod 0444), add newer rows (0.1 / 0.2) to the source, and materialize again. SQLite raises attempt to write a readonly database inside the worker.
  3. The Ray job status is SUCCEEDED, materialize returns normally, and get_online_features still returns 0.8 / 0.7.
  4. The Local engine on the same read-only database reports ERROR and materialize raises.

The read-only file is a deliberate stand-in for an online store that refuses writes; I have not seen this in a production deployment.

Specifications

  • Version: master at 386759c (15 Oct 2026)
  • Platform: macOS, Python 3.10.7, Ray 2.58.0
  • Subsystem: Ray compute engine (infra/compute_engines/ray)

Possible Solution

A small change (+9/-2 across ray/utils.py and ray/nodes.py): re-raise after the existing log in the two write functions, and take the write node's batch function out of safe_batch_processor, keeping its empty-batch early return. safe_batch_processor itself is unchanged (its swallow is pinned by test_exception_in_func_returns_original_batch, and other nodes use it). With it, five new tests fail on master and pass, four controls pass both ways, and the existing Ray component tests stay at 75 passed (non-integration) and 13 of 14 (integration).

The one integration test that turns red is test_ray_compute_engine_materialize, for the reason above. That seems like your call, so I have not changed it. A few options:

  1. assert ERROR there and track aggregated BatchFeatureView online writes on Ray as their own issue,
  2. drop the aggregations from that test so it tests a plain materialization, or
  3. fix the aggregated write first.

The intended behaviour change is that a failed online write fails the materialization instead of reporting success, so a pipeline that has been writing nothing will start failing. Happy to open a PR with the change and tests once you have a preference on the test.

This issue was written with AI assistance (Claude Code); the tests above were run before filing.

Activity

  1. patelchaitany
    added 2 commits that reference this issue on Oct 17, 2026

    fix: Fail Ray materialization when the online store write fails

    This issue will close once commit ab5b730 is merged into the 'master' branch.
    ab5b730

    fix: Fail Ray materialization when the online store write fails

    This issue will close once commit d492a1d is merged into the 'master' branch.
    d492a1d
  2. ntkathole
    added a commit that references this issue on Oct 18, 2026

    fix: Fail Ray materialization when the online store write fails

    This issue will close once commit e0e51f7 is merged into the 'master' branch.
    e0e51f7
  3. ntkathole
    closed this as completedin #7026on Oct 18, 2026
  4. odh-devops-app
    added a commit that references this issue on Oct 18, 2026

    fix: Stop the Ray engine from hiding materialization errors (feast-de…

    This issue will close once commit 7903e2e is merged into the 'master' branch.
    7903e2e
  5. franciscojavierarceo
    added a commit that references this issue on Nov 10, 2026

    chore(release): release 0.68.0

    fb5e5d5
  6. shuchu
    added a commit that references this issue on Nov 11, 2026

    chore(release): release 0.68.0

    5b94c9e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      You are visitor 01402503 — GeezerVison does not store the pages you look up.
      Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.