GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — feat: Offline Store historical features retrieval based on datetime range for spark by aniketpalu · Pull Request #5720 · feast-dev/feast

Conversation

aniketpalu commented Nov 13, 2025 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor

What this PR does / why we need it:

Add support for entity_df=None in SparkOfflineStore.get_historical_features with start_date/end_date.
- Builds a derived entity set by UNION DISTINCT of entity columns from each FeatureView within the window (uses date_partition_column when present), casts entities to STRING, and uses a literal as-of timestamp for PIT joins.
- Why: Enables simple time-window backfills without requiring an entity_df.
Type fixes in spark.py: pass KeysView[str] for entity_df_columns.
- Why: Satisfy mypy and keep PIT join builder signature consistent.
spark_source.py: ensure a SparkSession exists in remote threads.
- Why: Prevent remote mode failures due to missing active session.
arrow_error_handler.py: re-raise non-Feast exceptions on the server.
- Why: Ensure Arrow Flight returns actionable errors instead of None.
Test: add unit test for non-entity retrieval (Spark session mocked).

Which issue(s) this PR fixes:

RHOAIENG-38644

Misc

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions


          feat: Offline Store historical features retrieval based on datetime r…

6ae5ada

…ange for spark

Signed-off-by: Aniket Paluskar <[email protected]>

aniketpalu requested a review from a team as a code owner

November 13, 2025 08:57

ntkathole added the ok-to-test label

Nov 13, 2025

jyejare reviewed

Nov 14, 2025

View reviewed changes

jyejare left a comment •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Collaborator

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewewd very high level, looks Good, but could you share the run logs ?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

aniketpalu and others added 4 commits

November 18, 2025 13:10


          Merge branch 'master' into RHOAIENG-38644

759f953


          Merge branch 'master' into RHOAIENG-38644

adef6e6


          Restructured code, extended existing test cases

4fbf122

Signed-off-by: Aniket Paluskar <[email protected]>


          Fixed lint issues

f39ece8

Signed-off-by: Aniket Paluskar <[email protected]>

aniketpalu requested a review from jyejare

November 25, 2025 07:22

aniketpalu commented Nov 25, 2025 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor Author

Reviewewd very high level, looks Good, but could you share the run logs ?

Feature Repo used: https://github.com/aniketpalu/feast_examples/tree/main/feature_repo_spark
Testing script: https://github.com/aniketpalu/feast_examples/blob/2d5cef285996f59818ee4d19c595c0fb2efed4da/test_historical_retrieval.py

Attempted to retrieve historical features using entity_df:

📁 Data in Parquet file: 100 records
📅 Timestamp range: 2025-10-13 00:31:16.754605 to 2025-11-12 00:31:16.754605
👥 User IDs range: 1 to 100

📊 Entity data for feature retrieval:
user_id event_timestamp
0 1 2025-11-11 01:31:16.754605
1 2 2025-10-20 01:31:16.754605
2 3 2025-11-06 01:31:16.754605

🔍 Retrieving features for 3 entities...
/Users/apaluska/Work/feast/sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark_source.py:96: RuntimeWarning: The spark data source API is an experimental feature in alpha development. This API is unstable and it could and most probably will be changed in the future.
warnings.warn(
/Users/apaluska/Work/feast/sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark.py:167: RuntimeWarning: The spark offline store is an experimental feature in alpha development. Some functionality may still be unstable so functionality can change in the future.
warnings.warn(
WARNING: Using incubator modules: jdk.incubator.vector
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
25/11/25 12:57:14 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

✅ Historical features retrieved:
📊 Result shape: (3, 4)

📋 Results:
user_id event_timestamp avg_transaction_amount total_transactions
0 1 2025-11-11 07:01:16.754605 773.62 60
1 2 2025-10-20 07:01:16.754605 84.12 369
2 3 2025-11-06 07:01:16.754605 472.25 382

(Failed) Attempt to retrieve historical features using start_date & end_date before implementing:

📁 Data in Parquet file: 100 records
📅 Data timestamp range: 2025-10-13 00:31:16.754605 to 2025-11-12 00:31:16.754605
📅 Query date range: 2025-10-23 00:31:16.754605 to 2025-11-12 00:31:16.754605

🔍 Retrieving features for date range (no entity_df)...

Users/apaluska/Work/feast/sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark_source.py:96: RuntimeWarning: The spark data source API is an experimental feature in alpha development. This API is unstable and it could and most probably will be changed in the future.
warnings.warn(
❌ Error in date range-based retrieval: SparkOfflineStore.get_historical_features() got an unexpected keyword argument 'start_date'
Traceback (most recent call last):
File "/Users/apaluska/Work/EAP Demo/feast_example_minimal/test_historical_retrieval.py", line 159, in
historical_features = fs.get_historical_features(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/apaluska/Work/feast/sdk/python/feast/feature_store.py", line 1218, in get_historical_features
job = provider.get_historical_features(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/apaluska/Work/feast/sdk/python/feast/infra/passthrough_provider.py", line 469, in get_historical_features
job = self.offline_store.get_historical_features(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: SparkOfflineStore.get_historical_features() got an unexpected keyword argument 'start_date'

(Successful) Attempt to retrieve historical features using start_date & end_date:

📁 Data in Parquet file: 100 records
📅 Data timestamp range: 2025-10-13 00:31:16.754605 to 2025-11-12 00:31:16.754605
📅 Query date range: 2025-10-23 00:31:16.754605 to 2025-11-12 00:31:16.754605

🔍 Retrieving features for date range (no entity_df)...
/Users/apaluska/Work/feast/sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark_source.py:96: RuntimeWarning: The spark data source API is an experimental feature in alpha development. This API is unstable and it could and most probably will be changed in the future.
warnings.warn(
/Users/apaluska/Work/feast/sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark.py:169: RuntimeWarning: The spark offline store is an experimental feature in alpha development. Some functionality may still be unstable so functionality can change in the future.
warnings.warn(

✅ Historical features retrieved:
📊 Result shape: (66, 4)

📋 First 30 rows (showing both user_id entity_ts 0 25 2025-11-12 00:31:16.754605 1 7 2025-11-12 00:31:16.754605 2 77 2025-11-12 00:31:16.754605 3 3 2025-11-12 00:31:16.754605 4 57 2025-11-12 00:31:16.754605 5 45 2025-11-12 00:31:16.754605 6 100 2025-11-12 00:31:16.754605 7 35 2025-11-12 00:31:16.754605 8 96 2025-11-12 00:31:16.754605 9 19 2025-11-12 00:31:16.754605 10 29 2025-11-12 00:31:16.754605 11 22 2025-11-12 00:31:16.754605 12 97 2025-11-12 00:31:16.754605 13 81 2025-11-12 00:31:16.754605 14 31 2025-11-12 00:31:16.754605 15 83 2025-11-12 00:31:16.754605 16 40 2025-11-12 00:31:16.754605 17 82 2025-11-12 00:31:16.754605 18 24 2025-11-12 00:31:16.754605 19 73 2025-11-12 00:31:16.754605 20 74 2025-11-12 00:31:16.754605 21 89 2025-11-12 00:31:16.754605 22 95 2025-11-12 00:31:16.754605 23 48 2025-11-12 00:31:16.754605 24 1 2025-11-12 00:31:16.754605 25 50 2025-11-12 00:31:16.754605 26 59 2025-11-12 00:31:16.754605 27 86 2025-11-12 00:31:16.754605 28 21 2025-11-12 00:31:16.754605 29 27 2025-11-12 00:31:16.754605 event_timestamp and created_timestamp):
avg_transaction_amount total_transactions
NaN NaN
NaN NaN
494.04 127.0
472.25 382.0
NaN NaN
163.35 405.0
NaN NaN
51.20 113.0
NaN NaN
917.33 136.0
NaN NaN
NaN NaN
NaN NaN
NaN NaN
NaN NaN
315.60 313.0
305.82 267.0
NaN NaN
811.43 88.0
NaN NaN
NaN NaN
NaN NaN
65.17 464.0
NaN NaN
773.62 60.0
NaN NaN
921.00 426.0
NaN NaN
376.88 99.0
691.39 447.0

💡 Explanation:

event_timestamp: Point-in-time when features are queried (query timestamp)

created_timestamp_in_source: When the feature was actually created/updated in source data

When using start_date/end_date without entity_df, Feast uses end_date as event_timestamp

This is correct: 'What were the features as of 2025-11-12 00:31:16.754605?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.


          Merge branch 'master' into RHOAIENG-38644

d7ea5db

jyejare commented Dec 2, 2025

Copy link

Copy Markdown

Collaborator

@aniketpalu Did we validate the join works from multiple feature views in a feature service ?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

aniketpalu added 2 commits

December 8, 2025 17:13


          Merge branch 'master' into RHOAIENG-38644

964ac47


          Merge branch 'master' into RHOAIENG-38644

444698f

aniketpalu commented

Dec 9, 2025

View reviewed changes

Comment thread

sdk/python/tests/unit/infra/offline_stores/contrib/spark_offline_store/test_spark.py

		retrieval_job = SparkOfflineStore.get_historical_features(
		config=test_repo_config,
		feature_views=[test_feature_view2, test_feature_view1],
		feature_refs=["test_feature_view2:feature2", "test_feature_view1:feature1"],

aniketpalu Dec 9, 2025

Copy link

Copy Markdown

Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aniketpalu Did we validate the join works from multiple feature views in a feature service ?

@jyejare
We have tested this scenario and even added it in test cases.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

aniketpalu added 2 commits

December 12, 2025 15:29


          Merge branch 'master' into RHOAIENG-38644

dc2cbcf


          Merge branch 'master' into RHOAIENG-38644

9634c23

jyejare approved these changes

Dec 15, 2025

View reviewed changes

ntkathole approved these changes

Dec 15, 2025

View reviewed changes

ntkathole merged commit 27ec8ec into feast-dev:master

Dec 15, 2025

17 checks passed

Uh oh!

There was an error while loading. Please reload this page.

franciscojavierarceo pushed a commit that referenced this pull request

Dec 16, 2025


          chore(release): release 0.58.0

b601ef2

# [0.58.0](v0.57.0...v0.58.0) (2025-12-16)

### Bug Fixes

* Add java proto ([#5719](#5719)) ([fc3ea20](fc3ea20))
* Add possibility to force full features names for materialize ops ([#5728](#5728)) ([55c9c36](55c9c36))
* Fixed file registry cache sync ([09505d4](09505d4))
* Handle hyphon in sqlite project name ([#5575](#5575)) ([#5749](#5749)) ([b8346ff](b8346ff))
* Pinned substrait to fix protobuf issue ([d0ef4da](d0ef4da))
* Set TLS certificate annotation only on gRPC service ([#5715](#5715)) ([75d13db](75d13db))
* SQLite online store deletes tables from other projects in shared registry scenarios ([#5766](#5766)) ([fabce76](fabce76))
* Validate not existing entity join keys for preventing panic ([0b93559](0b93559))

### Features

* Add annotations for pod templates ([534e647](534e647))
* Add Pytorch template ([#5780](#5780)) ([6afd353](6afd353))
* Add support for extra options for stream source ([#5618](#5618)) ([18956c2](18956c2))
* Added matched_tag field search api results with fuzzy search capabilities ([#5769](#5769)) ([4a9ffae](4a9ffae))
* Added support for enabling metrics in Feast Operator ([#5317](#5317)) ([#5748](#5748)) ([a8498c2](a8498c2))
* Configure CacheTTLSecondscache,CacheMode for file-based registry in Feast Operator([#5708](#5708)) ([#5744](#5744)) ([f25f83b](f25f83b))
* Implemented Tiling Support for Time-Windowed Aggregations ([#5724](#5724)) ([7a99166](7a99166))
* Offline Store historical features retrieval based on datetime range for spark ([#5720](#5720)) ([27ec8ec](27ec8ec))
* Offline Store historical features retrieval based on datetime range in dask ([#5717](#5717)) ([a16582a](a16582a))
* Production ready feast operator with v1 apiversion ([#5771](#5771)) ([49359c6](49359c6))
* Support for Map value data type ([#5768](#5768)) ([#5772](#5772)) ([b99a8a9](b99a8a9))

antznette1 pushed a commit to antznette1/feast that referenced this pull request

Jan 3, 2026


          chore(release): release 0.58.0

a78a5de

# [0.58.0](feast-dev/feast@v0.57.0...v0.58.0) (2025-12-16)

### Bug Fixes

* Add java proto ([feast-dev#5719](feast-dev#5719)) ([fc3ea20](feast-dev@fc3ea20))
* Add possibility to force full features names for materialize ops ([feast-dev#5728](feast-dev#5728)) ([55c9c36](feast-dev@55c9c36))
* Fixed file registry cache sync ([09505d4](feast-dev@09505d4))
* Handle hyphon in sqlite project name ([feast-dev#5575](feast-dev#5575)) ([feast-dev#5749](feast-dev#5749)) ([b8346ff](feast-dev@b8346ff))
* Pinned substrait to fix protobuf issue ([d0ef4da](feast-dev@d0ef4da))
* Set TLS certificate annotation only on gRPC service ([feast-dev#5715](feast-dev#5715)) ([75d13db](feast-dev@75d13db))
* SQLite online store deletes tables from other projects in shared registry scenarios ([feast-dev#5766](feast-dev#5766)) ([fabce76](feast-dev@fabce76))
* Validate not existing entity join keys for preventing panic ([0b93559](feast-dev@0b93559))

### Features

* Add annotations for pod templates ([534e647](feast-dev@534e647))
* Add Pytorch template ([feast-dev#5780](feast-dev#5780)) ([6afd353](feast-dev@6afd353))
* Add support for extra options for stream source ([feast-dev#5618](feast-dev#5618)) ([18956c2](feast-dev@18956c2))
* Added matched_tag field search api results with fuzzy search capabilities ([feast-dev#5769](feast-dev#5769)) ([4a9ffae](feast-dev@4a9ffae))
* Added support for enabling metrics in Feast Operator ([feast-dev#5317](feast-dev#5317)) ([feast-dev#5748](feast-dev#5748)) ([a8498c2](feast-dev@a8498c2))
* Configure CacheTTLSecondscache,CacheMode for file-based registry in Feast Operator([feast-dev#5708](feast-dev#5708)) ([feast-dev#5744](feast-dev#5744)) ([f25f83b](feast-dev@f25f83b))
* Implemented Tiling Support for Time-Windowed Aggregations ([feast-dev#5724](feast-dev#5724)) ([7a99166](feast-dev@7a99166))
* Offline Store historical features retrieval based on datetime range for spark ([feast-dev#5720](feast-dev#5720)) ([27ec8ec](feast-dev@27ec8ec))
* Offline Store historical features retrieval based on datetime range in dask ([feast-dev#5717](feast-dev#5717)) ([a16582a](feast-dev@a16582a))
* Production ready feast operator with v1 apiversion ([feast-dev#5771](feast-dev#5771)) ([49359c6](feast-dev@49359c6))
* Support for Map value data type ([feast-dev#5768](feast-dev#5768)) ([feast-dev#5772](feast-dev#5772)) ([b99a8a9](feast-dev@b99a8a9))

Signed-off-by: Anthonette Adanyin <[email protected]>

franciscojavierarceo pushed a commit that referenced this pull request

Jan 5, 2026


          feat: Offline Store historical features retrieval based on datetime r…

50ab160

…ange for spark (#5720)

* feat: Offline Store historical features retrieval based on datetime range for spark

Signed-off-by: Aniket Paluskar <[email protected]>

* Restructured code, extended existing test cases

Signed-off-by: Aniket Paluskar <[email protected]>

* Fixed lint issues

Signed-off-by: Aniket Paluskar <[email protected]>

---------

Signed-off-by: Aniket Paluskar <[email protected]>
Signed-off-by: Francisco Javier Arceo <[email protected]>

franciscojavierarceo pushed a commit that referenced this pull request

Jan 5, 2026


          chore(release): release 0.58.0

807e80e

# [0.58.0](v0.57.0...v0.58.0) (2025-12-16)

### Bug Fixes

* Add java proto ([#5719](#5719)) ([fc3ea20](fc3ea20))
* Add possibility to force full features names for materialize ops ([#5728](#5728)) ([55c9c36](55c9c36))
* Fixed file registry cache sync ([09505d4](09505d4))
* Handle hyphon in sqlite project name ([#5575](#5575)) ([#5749](#5749)) ([b8346ff](b8346ff))
* Pinned substrait to fix protobuf issue ([d0ef4da](d0ef4da))
* Set TLS certificate annotation only on gRPC service ([#5715](#5715)) ([75d13db](75d13db))
* SQLite online store deletes tables from other projects in shared registry scenarios ([#5766](#5766)) ([fabce76](fabce76))
* Validate not existing entity join keys for preventing panic ([0b93559](0b93559))

### Features

* Add annotations for pod templates ([534e647](534e647))
* Add Pytorch template ([#5780](#5780)) ([6afd353](6afd353))
* Add support for extra options for stream source ([#5618](#5618)) ([18956c2](18956c2))
* Added matched_tag field search api results with fuzzy search capabilities ([#5769](#5769)) ([4a9ffae](4a9ffae))
* Added support for enabling metrics in Feast Operator ([#5317](#5317)) ([#5748](#5748)) ([a8498c2](a8498c2))
* Configure CacheTTLSecondscache,CacheMode for file-based registry in Feast Operator([#5708](#5708)) ([#5744](#5744)) ([f25f83b](f25f83b))
* Implemented Tiling Support for Time-Windowed Aggregations ([#5724](#5724)) ([7a99166](7a99166))
* Offline Store historical features retrieval based on datetime range for spark ([#5720](#5720)) ([27ec8ec](27ec8ec))
* Offline Store historical features retrieval based on datetime range in dask ([#5717](#5717)) ([a16582a](a16582a))
* Production ready feast operator with v1 apiversion ([#5771](#5771)) ([49359c6](49359c6))
* Support for Map value data type ([#5768](#5768)) ([#5772](#5772)) ([b99a8a9](b99a8a9))

Signed-off-by: Francisco Javier Arceo <[email protected]>

jyejare mentioned this pull request

Jan 13, 2026

Spark - historical retrieval without entity dataframe #5839

Closed

Reviewers

jyejare

jyejare approved these changes

ntkathole

ntkathole approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

ok-to-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

aniketpalu commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Misc

Uh oh!

jyejare left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aniketpalu commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jyejare commented Dec 2, 2025

Uh oh!

aniketpalu Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aniketpalu commented Nov 13, 2025 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

jyejare left a comment •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

aniketpalu commented Nov 25, 2025 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.