GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — fix: Invalid column names in get_historical_features when there are field mappings on join keys by aloysius-lim · Pull Request #4886 · feast-dev/feast

Conversation

@aloysius-lim

@aloysius-lim aloysius-lim commented Jan 3, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

When there are field mappings on an entity's join keys, get_historical_features() fails because it does not use the original column name in the data source.

This is needed when the source data cannot be modified, and a field mapping is used to map the source data's column to the join key.

Example:

from feast import Entity, Field, FeatureStore, FeatureView
from feast.types import Float32, String
from feast.infra.offline_stores.contrib.spark_offline_store.spark_source import SparkSource

# Initialize Feature Store.
store = FeatureStore(...)

# "driver_id" is used in the Feature Store.
driver = Entity(name="driver", join_keys=["driver_id"])

# Using SparkSource as an example, but this applies to other sources.
# Source data contains a primary key called "id". This is mapped to the join key "driver_id".
driver_stats_src = SparkSource(
    name="driver_stats",
    field_mapping={"id": "driver_id"},
    path=...,
    file_format=...,
)
driver_stats_fv = FeatureView(
    name="driver_stats",
    source= driver_stats_src,
    entities=[driver],
    schema=[
        # join key must be specified in the schema, else it is not included in driver_stats_fv.entity_columns
        Field(name="driver_id", dtype=String),
        Field(name="stat1", dtype=Float32),
        Field(name="stat2", dtype=Float32),
    ]
)

# Get historical features
store.get_historical_features(
    entity_df=...,
    features=[
        "driver_stats:stat1",
        "driver_stats:stat2",
    ]

Without this fix (Spark example):

pyspark.errors.exceptions.captured.AnalysisException: [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column or function parameter with name `driver_id` cannot be resolved. Did you mean one of the following? [`id`, `stat1`, `stat2`]

Underlying Spark query:

driver_stats__subquery AS (
    SELECT
        event_timestamp as event_timestamp,
        created as created_timestamp,

        -- Here is the problem.
        driver_id AS driver_id,

        stat1 as stat1, stat2 as stat2
    FROM `feast_entity_df_677a1a6fd13443c6b0e8ccc059b25f01`
    WHERE event_timestamp <= '2025-01-05T14:00:00'
)

With this fix, the field mapping is respected and the query becomes:

driver_stats__subquery AS (
    SELECT
        event_timestamp as event_timestamp,
        created as created_timestamp,

        id AS driver_id,
        
        stat1 as stat1, stat2 as stat2
    FROM `feast_entity_df_677a1a6fd13443c6b0e8ccc059b25f01`
    WHERE event_timestamp <= '2025-01-05T14:00:00'
)

Which issue(s) this PR fixes:

#4889

Misc

This only works if the entity join key is specified in the schema of the FeatureView (see above).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@aloysius-lim
aloysius-lim requested review from a team as code owners January 3, 2025 07:24
@aloysius-lim
aloysius-lim requested review from franciscojavierarceo, haoxu0 and tokoko January 3, 2025 07:24
@aloysius-lim
aloysius-lim force-pushed the fix-get-historical-features-join-key-field-mapping branch from 9c6ca6e to b0a70ac Compare January 3, 2025 07:26
@dmartinol

dmartinol commented Jan 3, 2025

Copy link
Copy Markdown
Contributor

Thanks for your contribution!
I’m wondering why we don’t have an issue to track this error: if you can’t find an existing issue, could you please open one before?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@franciscojavierarceo franciscojavierarceo added the ok-to-test label Jan 3, 2025
@aloysius-lim aloysius-lim mentioned this pull request Jan 3, 2025
get_historical_features does not work on join keys with field mappings #4889
Closed
@aloysius-lim

aloysius-lim commented Jan 3, 2025

Copy link
Copy Markdown
Contributor Author

Thanks for your contribution! I’m wondering why we don’t have an issue to track this error: if you can’t find an existing issue, could you please open one before?

I added issue #4889 for this.

dmartinol reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@franciscojavierarceo

franciscojavierarceo commented Jan 3, 2025

Copy link
Copy Markdown
Member

Do you mind rebasing? I fixed an issue so after you rebase it should go away.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@aloysius-lim
aloysius-lim force-pushed the fix-get-historical-features-join-key-field-mapping branch from b0a70ac to 84c355f Compare January 3, 2025 23:16
@aloysius-lim

aloysius-lim commented Jan 4, 2025

Copy link
Copy Markdown
Contributor Author

I've rebased, but the tests are still failing due to other errors with Keycloak.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@franciscojavierarceo

franciscojavierarceo commented Jan 6, 2025

Copy link
Copy Markdown
Member

Can you rebase again? Sorry I reverted my most recent change. 😅

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

aloysius-lim added 3 commits January 10, 2025 22:55
@aloysius-lim
Map join key to original column name in field mapping.
764dcf5
Signed-off-by: Aloysius Lim <[email protected]>
@aloysius-lim
Add test.
23a7dc5
Signed-off-by: Aloysius Lim <[email protected]>
@aloysius-lim
Format.
9b60d43
Signed-off-by: Aloysius Lim <[email protected]>
@aloysius-lim
aloysius-lim force-pushed the fix-get-historical-features-join-key-field-mapping branch from 84c355f to 9b60d43 Compare January 10, 2025 14:56
franciscojavierarceo
franciscojavierarceo approved these changes Feb 3, 2025
View reviewed changes

@franciscojavierarceo franciscojavierarceo left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@franciscojavierarceo
franciscojavierarceo merged commit c9aca2d into feast-dev:master Feb 3, 2025
franciscojavierarceo pushed a commit that referenced this pull request Feb 4, 2025
@feast-ci-bot
chore(release): release 0.44.0
aa7224a
# [0.44.0](v0.43.0...v0.44.0) (2025-02-04)

### Bug Fixes

* Adding periodic check to fix the sporadic failures of the operator e2e tests.  ([#4952](#4952)) ([1d086be](1d086be))
* Adding the feast-operator/bin to the .gitignore directory. Somehow it… ([#5005](#5005)) ([1a027ee](1a027ee))
* Changed Env Vars for e2e tests ([#4975](#4975)) ([fa0084f](fa0084f))
* Fix GitHub Actions to pass authentication ([#4963](#4963)) ([22b9138](22b9138)), closes [#4937](#4937) [#4939](#4939) [#4941](#4941) [#4940](#4940) [#4943](#4943) [#4944](#4944) [#4945](#4945) [#4946](#4946) [#4947](#4947) [#4948](#4948) [#4951](#4951) [#4954](#4954) [#4957](#4957) [#4958](#4958) [#4959](#4959) [#4960](#4960) [#4962](#4962)
* Fix showing selected navigation item in UI sidebar ([#4969](#4969)) ([8ac6a85](8ac6a85))
* Invalid column names in get_historical_features when there are field mappings on join keys ([#4886](#4886)) ([c9aca2d](c9aca2d))
* Read project data from the 'projects' key while loading the registry state in the Feast UI ([#4772](#4772)) ([cb81939](cb81939))
* Remove grpcurl dependency from Operator ([#4972](#4972)) ([439e0b9](439e0b9))
* Removed the dry-run flag to test and we will add it back later. ([#5007](#5007)) ([d112b52](d112b52))
* Render UI navigation items as links instead of buttons ([#4970](#4970)) ([1267703](1267703))
* Resolve Operator CRD bloat due to long field descriptions ([#4985](#4985)) ([7593bb3](7593bb3))
* Update manifest to add feature server image for odh ([#4973](#4973)) ([6a1c102](6a1c102))
* Updating release workflows to refer to yml instead of yaml ([#4935](#4935)) ([02b0a68](02b0a68))
* Use locally built feast-ui package in dev feature-server image ([#4998](#4998)) ([0145e55](0145e55))

### Features

* Added OWNERS file for OpenshiftCI ([#4991](#4991)) ([86a2ee8](86a2ee8))
* Adding Milvus demo to examples ([#4910](#4910)) ([2daf852](2daf852))
* Adding retrieve_online_documents endpoint ([#5002](#5002)) ([6607d3d](6607d3d))
* Adding support to return additional features from vector retrieval for Milvus db ([#4971](#4971)) ([6ce08d3](6ce08d3))
* Creating/updating the stable branch after the release. ([#5003](#5003)) ([e9b53cc](e9b53cc))
* Implementing online_read for MilvusOnlineStore ([#4996](#4996)) ([92dde13](92dde13))
* Improve exception message for unsupported Snowflake data types ([#4779](#4779)) ([5992364](5992364))
* Operator add feast ui deployment ([#4930](#4930)) ([b026d0c](b026d0c))
* Updating documents to highlight v2 api for Vector Similarity Se… ([#5000](#5000)) ([32b82a4](32b82a4))
@1fanwang 1fanwang mentioned this pull request May 12, 2026
fix(compute-engine/local): Honor field_mapping on join keys in dedup + join nodes #6395
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@franciscojavierarceo franciscojavierarceo franciscojavierarceo approved these changes
@tokoko tokoko Awaiting requested review from tokoko tokoko is a code owner automatically assigned from feast-dev/maintainers
@haoxu0 haoxu0 Awaiting requested review from haoxu0 haoxu0 is a code owner automatically assigned from feast-dev/maintainers

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

ok-to-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@aloysius-lim @dmartinol @franciscojavierarceo
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.