GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — feat: Added Ray Compute Engine and Ray Offline Store Support by ntkathole · Pull Request #5526 · feast-dev/feast

Conversation

@ntkathole

@ntkathole ntkathole commented Jul 20, 2025

Copy link
Copy Markdown
Member

What this PR does / why we need it:

This PR introduces Ray as both a compute engine and an offline store in Feast. It enables scalable, distributed feature engineering and data access in Feast using Ray, making it easier to handle large datasets and complex feature pipelines.

Key Features

Ray Compute Engine

  • Distributed DAG execution for feature transformations, aggregations, joins, and materialization.
  • Automatic selection between broadcast and distributed join strategies.
  • Resource-aware partitioning and parallelism.
  • Efficient point-in-time joins for historical feature retrieval.
  • Seamless integration with Feast’s feature view and materialization APIs.

Ray Offline Store

  • High-performance data reading and writing using Ray Datasets.
  • Support for local and distributed Ray clusters.
  • Batch writing and saved dataset persistence.
  • Configurable resource limits for safe local development and testing.

Documentation added for both Ray compute engine and offline store, including configuration, resource management, and usage examples.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
haoxu0, franciscojavierarceo, jyejare, and ebolblga reacted with thumbs up emoji franciscojavierarceo and ebolblga reacted with hooray emoji
All reactions
  • 👍 4 reactions
  • 🎉 2 reactions
@ntkathole ntkathole self-assigned this Jul 20, 2025
@ntkathole
ntkathole requested a review from a team as a code owner July 20, 2025 14:37
@ntkathole
ntkathole force-pushed the ray_compute branch from 3a142ba to 38850cc Compare July 20, 2025 16:47
@ntkathole ntkathole added the ok-to-test label Jul 20, 2025
@ntkathole
ntkathole marked this pull request as draft July 21, 2025 05:18
haoxu0
haoxu0 reviewed Jul 22, 2025
View reviewed changes
Comment thread sdk/python/feast/infra/compute_engines/ray/feature_builder.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@ntkathole
ntkathole force-pushed the ray_compute branch 20 times, most recently from 8549556 to 76af065 Compare July 26, 2025 13:14
@ntkathole
ntkathole marked this pull request as ready for review July 26, 2025 13:16
@ntkathole
ntkathole requested review from franciscojavierarceo and haoxu0 July 26, 2025 14:28
haoxu0
haoxu0 reviewed Jul 26, 2025
View reviewed changes
Comment thread sdk/python/feast/infra/compute_engines/ray/feature_builder.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@ntkathole
ntkathole force-pushed the ray_compute branch from 05d9827 to cf0e01f Compare August 11, 2025 04:52
franciscojavierarceo
franciscojavierarceo reviewed Aug 11, 2025
View reviewed changes
Comment thread sdk/python/feast/infra/compute_engines/ray/compute.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
ntkathole added 9 commits August 11, 2025 21:09
@ntkathole
feat: Added ray to requirements
f1e78e9 
Signed-off-by: ntkathole <[email protected]>
@ntkathole
feat: Added Ray offline store
bd27d23 
Signed-off-by: ntkathole <[email protected]>
@ntkathole
feat: Improved Ray distributed processing
51c3426 
Signed-off-by: ntkathole <[email protected]>
@ntkathole
Fixed tests
5b629b4 
Signed-off-by: ntkathole <[email protected]>
@ntkathole
fix: Added utils for validation
a806ae3 
Signed-off-by: ntkathole <[email protected]>
@ntkathole
feat: Use Ray Dataset for data processing
b31a8fd 
Signed-off-by: ntkathole <[email protected]>
@ntkathole
feat: Added Ray Compute Engine
266f4cd 
Signed-off-by: ntkathole <[email protected]>
@ntkathole
fix: Fixed logic for source/derived feature views
1da9c94 
Signed-off-by: ntkathole <[email protected]>
@ntkathole
fix: Remove redundant source resolution
c3a87ab 
Signed-off-by: ntkathole <[email protected]>
franciscojavierarceo
franciscojavierarceo reviewed Aug 11, 2025
View reviewed changes
Comment thread sdk/python/feast/infra/compute_engines/ray/feature_builder.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
franciscojavierarceo
franciscojavierarceo reviewed Aug 11, 2025
View reviewed changes
Comment thread sdk/python/feast/infra/compute_engines/ray/feature_builder.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
franciscojavierarceo
franciscojavierarceo reviewed Aug 11, 2025
View reviewed changes
Comment thread sdk/python/feast/infra/compute_engines/ray/job.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
franciscojavierarceo
franciscojavierarceo reviewed Aug 11, 2025
View reviewed changes
Comment thread sdk/python/feast/infra/compute_engines/ray/nodes.py
timestamp_col = self.column_info.timestamp_column
requested_feats = getattr(self.column_info, "feature_cols", [])

# Check if the feature dataset contains aggregated features (from aggregation node)

@franciscojavierarceo franciscojavierarceo Aug 11, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
franciscojavierarceo
franciscojavierarceo reviewed Aug 11, 2025
View reviewed changes
Comment thread sdk/python/feast/infra/compute_engines/ray/nodes.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
franciscojavierarceo
franciscojavierarceo reviewed Aug 11, 2025
View reviewed changes
Comment thread sdk/python/feast/infra/offline_stores/contrib/ray_offline_store/ray.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
franciscojavierarceo
franciscojavierarceo reviewed Aug 11, 2025
View reviewed changes
Comment thread sdk/python/feast/infra/offline_stores/contrib/ray_offline_store/ray.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
franciscojavierarceo
franciscojavierarceo reviewed Aug 11, 2025
View reviewed changes

@franciscojavierarceo franciscojavierarceo left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks great some small nits but i think we're almost there. can you make some issues to track the follow up here so we don't forget and I'll merge 👍

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread sdk/python/feast/infra/offline_stores/contrib/ray_offline_store/ray.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread sdk/python/feast/transformation/pandas_transformation.py
cls,
udf: Callable[[Any], Any],
udf_string: str,
udf: Optional[Callable[[Any], Any]] = None,

@franciscojavierarceo franciscojavierarceo Aug 11, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why make this optional?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@ntkathole ntkathole Aug 12, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not optional in real use, we are still maintaining strict validation (lines 72-75 enforce that they can't actually be None). We have to make it optional to ensures that PandasTransformation objects can be properly reconstructed across Ray workers without constructor failures.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
franciscojavierarceo reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@ntkathole
ntkathole force-pushed the ray_compute branch from cf0e01f to af613d4 Compare August 12, 2025 04:08
@ntkathole ntkathole mentioned this pull request Aug 12, 2025
Generalize build materialization_plan from the FeatureBuilder to store intermediate results #5566
Open
@ntkathole
fix: Handle comments and removed use_ray_cluster config
5435859 
Signed-off-by: ntkathole <[email protected]>
@ntkathole
ntkathole force-pushed the ray_compute branch from af613d4 to 5435859 Compare August 12, 2025 05:08
@ntkathole
ntkathole requested a review from franciscojavierarceo August 12, 2025 05:35
franciscojavierarceo
franciscojavierarceo approved these changes Aug 12, 2025
View reviewed changes
@franciscojavierarceo

franciscojavierarceo commented Aug 12, 2025

Copy link
Copy Markdown
Member

Thanks for this @ntkathole !!!
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@franciscojavierarceo
franciscojavierarceo merged commit 72de088 into feast-dev:master Aug 12, 2025
franciscojavierarceo pushed a commit that referenced this pull request Aug 14, 2025
@feast-ci-bot
chore(release): release 0.52.0
acb97fb 
# [0.52.0](v0.51.0...v0.52.0) (2025-08-14)

### Bug Fixes

* Correct entity value type mapping for aliased feature views ([#5492](#5492)) ([bdf20bb](bdf20bb))
* Correct namespace reference in remote Feast project setup for operator upgrade and previous version tests ([df391ec](df391ec))
* dell pydantic v1 ([1189512](1189512))
* Fixed the entity to on-demand feature view relationship ([1c59bba](1c59bba))
* Make transformers optional ([#5544](#5544)) ([a4eef38](a4eef38))
* Push Source inherits the timestamp fields from Data Source ([#5550](#5550)) ([b7ea5cc](b7ea5cc))
* Remove the devcontainer folder. ([a9815c2](a9815c2))

### Features

* Added API for discovering Feature Views by popular tags ([#5558](#5558)) ([2e5f564](2e5f564))
* Added filtering support for featureView and featureServices api ([#5552](#5552)) ([897b3f3](897b3f3))
* Added global search api and necessary unit tests ([#5532](#5532)) ([dd3061f](dd3061f))
* Added Ray Compute Engine and Ray Offline Store Support ([#5526](#5526)) ([72de088](72de088))
* Added recent visit logging api for registry server ([#5545](#5545)) ([2adcf2c](2adcf2c))
* **auth:** support client-credentials & static token for OIDC client auth ([fc44222](fc44222))
* **auth:** support client-credentials & static token for OIDC client auth ([795fc06](795fc06))
* Implement and enhance remote document retrieval functionality ([#5487](#5487)) ([d095b96](d095b96))
* Implemented consistent error handling ([7f10151](7f10151))
* Offline Store historical features retrieval without entity df, but based on datatime range ([#5527](#5527)) ([df942b9](df942b9))
@ntkathole ntkathole mentioned this pull request Aug 16, 2025
to_ray_dataset as a first citizen method for retrieval_job #5568
Closed
@franciscojavierarceo franciscojavierarceo mentioned this pull request Sep 11, 2026
docs: Add Nikhil Kathole as a project maintainer #6830
Merged
3 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jyejare jyejare jyejare left review comments
@haoxu0 haoxu0 haoxu0 approved these changes
Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@franciscojavierarceo franciscojavierarceo franciscojavierarceo approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@ntkathole ntkathole

Labels

ok-to-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@ntkathole @franciscojavierarceo @jyejare @haoxu0
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.