GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — fix: Apply on demand feature views in SparkRetrievalJob.to_spark_df() by shanalyb · Pull Request #6993 · feast-dev/feast

Conversation

shanalyb commented Oct 10, 2026

Copy link

Copy Markdown

Contributor

What this PR does / why we need it:

SparkRetrievalJob.to_spark_df() returns the raw query result, while on demand feature views are applied only in RetrievalJob.to_arrow(). The Spark DataFrame therefore has no ODFV columns, and neither do persist() and to_feast_df(), which build on it.

to_spark_df() now computes the requested ODFV features on the executors with mapInArrow.

SparkRetrievalJob subclasses now override _to_spark_df_internal() instead of to_spark_df().

Added test_to_spark_df_applies_odfvs.

Which issue(s) this PR fixes:

Related to #5475 (no new spark modes) and #6359.

Checks

I've made sure the tests are passing.
My commits are signed off (git commit -s)
My PR title follows conventional commits format

Testing Strategy

Unit tests
Integration tests
Manual tests
Testing is not required for this change

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions


          fix: Apply on demand feature views in SparkRetrievalJob.to_spark_df()

3b5655a

SparkRetrievalJob.to_spark_df() returned the raw query result, so on
demand feature views were applied only by to_df() and to_arrow(), and
persist() and to_feast_df() left them out as well.

Compute the requested ODFV features on the executors with mapInArrow,
selecting them as RetrievalJob.to_arrow does. ODFVs are shipped as
protos together with their projection, and outputs are cast to the
declared feature types that have an Arrow mapping.
SparkDAGRetrievalJob now overrides _to_spark_df_internal, so to_df()
keeps reading the plan result.

Signed-off-by: Denis Ivanov <[email protected]>

shanalyb requested a review from a team as a code owner

October 10, 2026 23:27


          fix: Map PdfBytes, ImageBytes, Uuid, TimeUuid and ZonedTimestamp to S…

b6e2cbf

…park types

from_feast_to_spark_type had no entries for PdfBytes, ImageBytes, Uuid,
TimeUuid and ZonedTimestamp, so an on demand feature view declaring one
of them failed in to_spark_df() before it ran. Map them to the Spark
type of the Arrow values Feast produces for them: BinaryType, StringType
and TimestampType.

Signed-off-by: Denis Ivanov <[email protected]>

codecov-commenter commented Oct 11, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 14.92537% with 57 lines in your changes missing coverage. Please review.
✅ Project coverage is 50.57%. Comparing base (4fc9726) to head (b6e2cbf).

Files with missing lines	Patch %	Lines
...ffline_stores/contrib/spark_offline_store/spark.py	15.15%	55 Missing and 1 partial ⚠️
...dk/python/feast/infra/compute_engines/spark/job.py	0.00%	1 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6993      +/-   ##
==========================================
- Coverage   50.61%   50.57%   -0.05%     
==========================================
  Files         451      451              
  Lines       56457    56521      +64     
  Branches     8229     8248      +19     
==========================================
+ Hits        28575    28584       +9     
- Misses      25920    25973      +53     
- Partials     1962     1964       +2

Flag	Coverage Δ
go-feature-server	`30.58% <ø> (ø)`
python-unit	`52.01% <14.92%> (-0.05%)`	⬇️

Files with missing lines	Coverage Δ
.../python/feast/infra/compute_engines/spark/nodes.py	`26.13% <ø> (ø)`
...dk/python/feast/infra/compute_engines/spark/job.py	`0.00% <0.00%> (ø)`
...ffline_stores/contrib/spark_offline_store/spark.py	`34.31% <15.15%> (-1.68%)`	⬇️

... and 1 file with indirect coverage changes

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4fc9726...b6e2cbf. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

haoxu0 reviewed

Oct 13, 2026

View reviewed changes

haoxu0 left a comment

Copy link

Copy Markdown

Collaborator

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do it in a OOD. So that we can optimize the on demand feature transformation in future

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark.py Outdated

		return self._metadata


		def _apply_odfv_transformations(

haoxu0 Oct 13, 2026

Copy link

Copy Markdown

Collaborator

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of doing this function driven. I think it should be a class.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

shanalyb Oct 13, 2026

Copy link

Copy Markdown

Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved it into a SparkOnDemandFeatureViewProcessor class in c02d9ab.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

shanalyb added 2 commits

October 13, 2026 17:31


          refactor: Move Spark on demand feature view application into a class

c02d9ab

Signed-off-by: Denis Ivanov <[email protected]>


          refactor: Share the on demand feature selection between to_arrow and …

848ecd0

…to_spark_df

Move the requested ODFV feature lookup from RetrievalJob.to_arrow into
RetrievalJob._get_odfv_feature_refs, so SparkOnDemandFeatureViewProcessor
uses it instead of a copy.

Signed-off-by: Denis Ivanov <[email protected]>

shanalyb requested a review from a team as a code owner

October 15, 2026 20:34

shanalyb requested review from ejscribner, robhowley and tokoko and removed request for a team

October 15, 2026 20:34


          Merge branch 'master' into fix/spark-odfv-to-spark-df

1f1bb16

shuchu approved these changes

Nov 10, 2026

View reviewed changes

shuchu left a comment

Copy link

Copy Markdown

Collaborator

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

shuchu merged commit f416309 into feast-dev:master

Nov 10, 2026

2 checks passed

Uh oh!

There was an error while loading. Please reload this page.

patelchaitany mentioned this pull request

Nov 10, 2026

chore: Sync upstream feast-dev/feast master opendatahub-io/feast#288

Merged

franciscojavierarceo pushed a commit that referenced this pull request

Nov 10, 2026


          chore(release): release 0.68.0

fb5e5d5

# [0.68.0](v0.67.0...v0.68.0) (2026-11-10)

### Bug Fixes

* add watch verb on clusterroles and clusterrolebindings for feast-operator ([#7062](#7062)) ([b6bdda4](b6bdda4))
* Apply on demand feature views in SparkRetrievalJob.to_spark_df() ([#6993](#6993)) ([f416309](f416309))
* Break event timestamp ties by created timestamp in Milvus write batches ([#7084](#7084)) ([bbaca76](bbaca76))
* Break event timestamp ties by created timestamp in the Dask offline store ([#6952](#6952)) ([eb93b08](eb93b08))
* Build the Go feature server image in two stages ([1e5e90c](1e5e90c)), closes [#7060](#7060)
* Fall back to the serialized body when UDF source misses globals ([#6816](#6816)) ([f61079f](f61079f))
* Install patched libbrotli1 to address DSA-6582-1 ([#7096](#7096)) ([d458393](d458393))
* Keep description, tags and nested types of ODFV schema fields ([8ef6e75](8ef6e75))
* Keep entity columns out of StreamFeatureView proto features ([#7033](#7033)) ([d520584](d520584))
* Keep the metrics server running when a Gunicorn worker exits ([#7012](#7012)) ([68c5d54](68c5d54)), closes [#7011](#7011)
* Key registry proto caches on the proto, not its id() ([#7085](#7085)) ([147adad](147adad))
* Log ODFV materialization skips instead of printing them ([#7117](#7117)) ([8fdcee7](8fdcee7))
* Pin build dependencies of source builds to the CI lock ([#7104](#7104)) ([dec2f24](dec2f24))
* Populate spec.project in registry REST feature view responses (#… ([#7037](#7037)) ([e304a84](e304a84))
* Preserve colliding source timestamps in Dask retrieval ([#7071](#7071)) ([1396c88](1396c88))
* Preserve Dask historical requests without eligible features ([#7000](#7000)) ([fff09f7](fff09f7))
* Read None elements of UUID and Decimal lists as None ([#7039](#7039)) ([03f30fe](03f30fe))
* Retire the support of Python3.10 ([#7101](#7101)) ([a56766e](a56766e)), closes [#7104](#7104) [#7089](#7089) [#7106](#7106) [#7105](#7105)
* Serialize UUID and TimeUuid entity keys ([#7032](#7032)) ([0c845a1](0c845a1))
* Stop the Ray engine from hiding materialization errors ([#7026](#7026)) ([7903e2e](7903e2e)), closes [#7024](#7024)
* **ui:** stop treating the monitoring baseline as the latest window ([#7056](#7056)) ([0f01802](0f01802)), closes [#7054](#7054)
* Use date-only bounds for partition filters built from string timestamps ([#7116](#7116)) ([b1f2168](b1f2168))
* Validate entity column types in feast dbt import ([#7067](#7067)) ([abca80c](abca80c)), closes [#5876](#5876)
* Warm up the whole DynamoDB connection pool ([#6985](#6985)) ([8ce548c](8ce548c))

### Features

* Add --gc-freeze option to feast serve ([#7089](#7089)) ([aad0c15](aad0c15))
* Add native complex types and scalar index tags to the Milvus online store ([#7120](#7120)) ([f7068d6](f7068d6))
* Add num_partitions and partition_key_isolation to the Milvus online store ([#7083](#7083)) ([68de84e](68de84e)), closes [#6940](#6940)
* Add Pinecone online store integration ([#6585](#6585)) ([7a84f2a](7a84f2a))
* Add PolarisSource and PolarisProvider for Apache Polaris ([#7006](#7006)) ([386759c](386759c))
* Add row deletion to the Milvus online store ([#7082](#7082)) ([d3e2041](d3e2041))
* Add upstream FeatureView lineage support for PushSource ([#6853](#6853)) ([c9b9d72](c9b9d72))
* Optional nameless pre-computed vectors with a schema fingerprint ([#7077](#7077)) ([9420fac](9420fac))

### Performance Improvements

* Scan Lance sources lazily in the DuckDB offline store ([1299e4e](1299e4e))

patelchaitany mentioned this pull request

Nov 11, 2026

chore: Sync upstream feast-dev/feast stable (0.68.0) opendatahub-io/feast#291

Merged

shuchu added a commit to shuchu/feast that referenced this pull request

Nov 11, 2026


          fix: Apply on demand feature views in SparkRetrievalJob.to_spark_df() (…

4d9e868

…feast-dev#6993)

* fix: Apply on demand feature views in SparkRetrievalJob.to_spark_df()

SparkRetrievalJob.to_spark_df() returned the raw query result, so on
demand feature views were applied only by to_df() and to_arrow(), and
persist() and to_feast_df() left them out as well.

Compute the requested ODFV features on the executors with mapInArrow,
selecting them as RetrievalJob.to_arrow does. ODFVs are shipped as
protos together with their projection, and outputs are cast to the
declared feature types that have an Arrow mapping.
SparkDAGRetrievalJob now overrides _to_spark_df_internal, so to_df()
keeps reading the plan result.

Signed-off-by: Denis Ivanov <[email protected]>

* fix: Map PdfBytes, ImageBytes, Uuid, TimeUuid and ZonedTimestamp to Spark types

from_feast_to_spark_type had no entries for PdfBytes, ImageBytes, Uuid,
TimeUuid and ZonedTimestamp, so an on demand feature view declaring one
of them failed in to_spark_df() before it ran. Map them to the Spark
type of the Arrow values Feast produces for them: BinaryType, StringType
and TimestampType.

Signed-off-by: Denis Ivanov <[email protected]>

* refactor: Move Spark on demand feature view application into a class

Signed-off-by: Denis Ivanov <[email protected]>

* refactor: Share the on demand feature selection between to_arrow and to_spark_df

Move the requested ODFV feature lookup from RetrievalJob.to_arrow into
RetrievalJob._get_odfv_feature_refs, so SparkOnDemandFeatureViewProcessor
uses it instead of a copy.

Signed-off-by: Denis Ivanov <[email protected]>

---------

Signed-off-by: Denis Ivanov <[email protected]>
Co-authored-by: Shuchu Han <[email protected]>
Signed-off-by: Shuchu Han <[email protected]>

shuchu pushed a commit to shuchu/feast that referenced this pull request

Nov 11, 2026


          chore(release): release 0.68.0

5b94c9e

# [0.68.0](feast-dev/feast@v0.67.0...v0.68.0) (2026-11-10)

### Bug Fixes

* add watch verb on clusterroles and clusterrolebindings for feast-operator ([feast-dev#7062](feast-dev#7062)) ([b6bdda4](feast-dev@b6bdda4))
* Apply on demand feature views in SparkRetrievalJob.to_spark_df() ([feast-dev#6993](feast-dev#6993)) ([f416309](feast-dev@f416309))
* Break event timestamp ties by created timestamp in Milvus write batches ([feast-dev#7084](feast-dev#7084)) ([bbaca76](feast-dev@bbaca76))
* Break event timestamp ties by created timestamp in the Dask offline store ([feast-dev#6952](feast-dev#6952)) ([eb93b08](feast-dev@eb93b08))
* Build the Go feature server image in two stages ([1e5e90c](feast-dev@1e5e90c)), closes [feast-dev#7060](feast-dev#7060)
* Fall back to the serialized body when UDF source misses globals ([feast-dev#6816](feast-dev#6816)) ([f61079f](feast-dev@f61079f))
* Install patched libbrotli1 to address DSA-6582-1 ([feast-dev#7096](feast-dev#7096)) ([d458393](feast-dev@d458393))
* Keep description, tags and nested types of ODFV schema fields ([8ef6e75](feast-dev@8ef6e75))
* Keep entity columns out of StreamFeatureView proto features ([feast-dev#7033](feast-dev#7033)) ([d520584](feast-dev@d520584))
* Keep the metrics server running when a Gunicorn worker exits ([feast-dev#7012](feast-dev#7012)) ([68c5d54](feast-dev@68c5d54)), closes [feast-dev#7011](feast-dev#7011)
* Key registry proto caches on the proto, not its id() ([feast-dev#7085](feast-dev#7085)) ([147adad](feast-dev@147adad))
* Log ODFV materialization skips instead of printing them ([feast-dev#7117](feast-dev#7117)) ([8fdcee7](feast-dev@8fdcee7))
* Pin build dependencies of source builds to the CI lock ([feast-dev#7104](feast-dev#7104)) ([dec2f24](feast-dev@dec2f24))
* Populate spec.project in registry REST feature view responses (#… ([feast-dev#7037](feast-dev#7037)) ([e304a84](feast-dev@e304a84))
* Preserve colliding source timestamps in Dask retrieval ([feast-dev#7071](feast-dev#7071)) ([1396c88](feast-dev@1396c88))
* Preserve Dask historical requests without eligible features ([feast-dev#7000](feast-dev#7000)) ([fff09f7](feast-dev@fff09f7))
* Read None elements of UUID and Decimal lists as None ([feast-dev#7039](feast-dev#7039)) ([03f30fe](feast-dev@03f30fe))
* Retire the support of Python3.10 ([feast-dev#7101](feast-dev#7101)) ([a56766e](feast-dev@a56766e)), closes [feast-dev#7104](feast-dev#7104) [feast-dev#7089](feast-dev#7089) [feast-dev#7106](feast-dev#7106) [feast-dev#7105](feast-dev#7105)
* Serialize UUID and TimeUuid entity keys ([feast-dev#7032](feast-dev#7032)) ([0c845a1](feast-dev@0c845a1))
* Stop the Ray engine from hiding materialization errors ([feast-dev#7026](feast-dev#7026)) ([7903e2e](feast-dev@7903e2e)), closes [feast-dev#7024](feast-dev#7024)
* **ui:** stop treating the monitoring baseline as the latest window ([feast-dev#7056](feast-dev#7056)) ([0f01802](feast-dev@0f01802)), closes [feast-dev#7054](feast-dev#7054)
* Use date-only bounds for partition filters built from string timestamps ([feast-dev#7116](feast-dev#7116)) ([b1f2168](feast-dev@b1f2168))
* Validate entity column types in feast dbt import ([feast-dev#7067](feast-dev#7067)) ([abca80c](feast-dev@abca80c)), closes [feast-dev#5876](feast-dev#5876)
* Warm up the whole DynamoDB connection pool ([feast-dev#6985](feast-dev#6985)) ([8ce548c](feast-dev@8ce548c))

### Features

* Add --gc-freeze option to feast serve ([feast-dev#7089](feast-dev#7089)) ([aad0c15](feast-dev@aad0c15))
* Add native complex types and scalar index tags to the Milvus online store ([feast-dev#7120](feast-dev#7120)) ([f7068d6](feast-dev@f7068d6))
* Add num_partitions and partition_key_isolation to the Milvus online store ([feast-dev#7083](feast-dev#7083)) ([68de84e](feast-dev@68de84e)), closes [feast-dev#6940](feast-dev#6940)
* Add Pinecone online store integration ([feast-dev#6585](feast-dev#6585)) ([7a84f2a](feast-dev@7a84f2a))
* Add PolarisSource and PolarisProvider for Apache Polaris ([feast-dev#7006](feast-dev#7006)) ([386759c](feast-dev@386759c))
* Add row deletion to the Milvus online store ([feast-dev#7082](feast-dev#7082)) ([d3e2041](feast-dev@d3e2041))
* Add upstream FeatureView lineage support for PushSource ([feast-dev#6853](feast-dev#6853)) ([c9b9d72](feast-dev@c9b9d72))
* Optional nameless pre-computed vectors with a schema fingerprint ([feast-dev#7077](feast-dev#7077)) ([9420fac](feast-dev@9420fac))

### Performance Improvements

* Scan Lance sources lazily in the DuckDB offline store ([1299e4e](feast-dev@1299e4e))

Signed-off-by: Shuchu Han <[email protected]>

rhods-devops-app Bot mentioned this pull request

Nov 12, 2026

Automated Sync from main to stable red-hat-data-services/feast#630

Open

Reviewers

haoxu0

haoxu0 left review comments

shuchu

shuchu approved these changes

ejscribner

Awaiting requested review from ejscribner

ejscribner is a code owner automatically assigned from feast-dev/maintainers

robhowley

Awaiting requested review from robhowley

robhowley is a code owner automatically assigned from feast-dev/maintainers

tokoko

Awaiting requested review from tokoko

tokoko is a code owner automatically assigned from feast-dev/maintainers

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

shanalyb commented Oct 10, 2026

What this PR does / why we need it:

Which issue(s) this PR fixes:

Checks

Testing Strategy

Uh oh!

codecov-commenter commented Oct 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

haoxu0 left a comment

Choose a reason for hiding this comment

Uh oh!

haoxu0 Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

shanalyb Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

shuchu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov-commenter commented Oct 11, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.