GeezerVison is showing github.com — Go feature server build wrong redis key that be made from `INT32` type entity · Issue #5357 · feast-dev/feast
Go feature server build wrong redis key that be made from INT32 type entity #5357Descriptioncc. @franciscojavierarceo
These days, I try to contributing Go feature server in this repository.(e.g recent PR(#5352) that I made)
At present, I use Go feature server(alpha) and apply it on my production environment. My feast environment is below.
But when using Go feature server with redis, I encountered the bug in building redis hash key.
At first, I define my entities, feature views and register them. At this time, one of my entities has int32 type. This int32 type is supported according to feast docs.
After that, I finish materializing features from offline store to online store and run Go feature server. Finally, client application request to go feature server.
But, I check that a specific feature is materialized well but from go feature server, the feature value is null. So instead of go feature server, I tried to fetch the feature with feast python sdk. When using feast python sdk, the feature value is not null. So I confused at this circumstance.
So, I deeply dive about this bug. I found that the features from other entity that has string type is not null from go feature server. So I paid attention to the entity that has int32 type.
I have two feast environments(dev, prod). At present, the prod environment is in the problem. So in dev environment, I intentionally cast the type of entity from int32 to string and re-register, materialize it.
After that, when I fetch the feature from go feature server in dev, I surprised that the feature value is not null!
So, I guess that the cause of this issue is maybe in this source code in go feature server. the probelm is in making hashkey for fetching(HMGet) to redis. Because, in the dev environment, hashkey by made from go feature server source code can be found in the redis. I check it by redis cli or redis insight GUI tool. But in the prod environment, the hashkey from go feature server was not in redis.
I build and run go feature server by debug mode and evaluate the redisKeys variable. I copy the data from redisKeys variable, and try to find the key in redis but there is no hashkey in redis.
So, I want to fix this bug.. Can you advise to me? I mention @shuchu based on the commit history about integrating redis as online store in go feature server.
Expected Behavior
In the same feature, the fetched feature from feast-sdk and go feature server is the same.
Current Behavior
In the same feature, the fetched feature from feast-sdk is normal but feature from go feature server is null.
Steps to reproduce
Specifications
- Version: locally development but the problematic source code is the same as in the master branch.
- Platform: MacOS 64bit arm64 OS platform
- Subsystem:
Possible Solution
What is the potential cause of this bug? Activitychanged the title [-]Go feature server can't build redis key that be made from integer entity[/-] [+]Go feature server build wrong redis key that be made from integer entity[/+] on May 15, 2025franciscojavierarceo commented on May 16, 2025 franciscojavierarceo commented on May 16, 2025 can you provide some minimal example feature view and mock data? iamcodingcat commented on May 16, 2025 ContributorAuthor More actions @franciscojavierarceo cc. @jyejare @ntkathole
For you, I ready script for register and mock data. The mock data can be download from below link.
And I wrote reigster.py script based on this mock data.
# register.py
from feast.types import Int64, Float64
from feast import (
FileSource,
Entity,
ValueType,
FeatureView,
Field
)
user = Entity(name="user", join_keys=["entity_id"], value_type=ValueType.INT64)
source = FileSource(
name="dummy_data_source",
path="/your/path/to/mock-data/feast_issue_#5357_entity_int32.csv",
timestamp_field="timestamp"
)
fv = FeatureView(
name="user_fv",
entities=[user],
source=source,
schema=[
Field(name="price", dtype=Int64),
Field(name="discount", dtype=Float64)
]
)Finally, I gave you feature_store.yaml. I use s3 storage but you can use local stroage as registry store. But you ensure that using redis as online store for testing my issue.
project: my_project
provider: aws
registry:
path: s3://my-bucket/registry.db
cache_ttl_seconds: 600 # default
entity_key_serialization_version: 2
online_store:
type: redis
redis_type: redis_cluster
connection_string: localhost:6379 If any issues occur in the testing environment, let me know! In my production, I run Go feature server. So I try to fix it ASAP.. 😢 . If you have no time to fix it, I am willing to try to do it. Reacted by Jitendra Yejare jyejare commented on May 16, 2025 @iamcodingcat Could you rebuild the feast environment with the serialization version 3 and retry? iamcodingcat commented on May 16, 2025 ContributorAuthor More actions @jyejare I will try to your guide. But while testing, I asked for your rebuild meaning. You mean that I must specify serialization version 3 including register and materialize features? iamcodingcat commented on May 16, 2025 ContributorAuthor More actions @jyejare At first, I register and materialize feature using serialization version 2 but when Go feature server uses serialization version 3, the hashkey doesn't still exist in redis.
I will com back after trying all serialization version 3 from register, materialize and Go feature server. iamcodingcat commented on May 16, 2025 ContributorAuthor More actions @jyejare After I test as version 3 from register, materilalize and go feature server, the error is still.
I store all data that use version 2 and 3. So in the redis GUI tools, the data by version is below attachment
the hashkey format between version 2 and 3 seems to be different. So specifying to version 3 is affected but it is not likely to resolve this problem. the hashkey version 3 is not in redis. Can you test it on this issue? shuchu commented on May 16, 2025 @iamcodingcat thank you for finding this issue. I'm check on it now. by any chance, can you tell me which Redis version you are using locally for debugging? or you use the AWS elasticache valkey? shuchu commented on May 16, 2025 iamcodingcat commented on May 17, 2025 ContributorAuthor More actions iamcodingcat commented on May 17, 2025 ContributorAuthor More actions @shuchu @jyejare cc. @franciscojavierarceo
Hi, I investigate it more and I am likely to find the root cause of this issue. But I ask you to check and review my found cause again.
First of all, Although register and materialize using entity type as INT32, go feature server is unconditionally regard as INT64. I check this by debug mode in below attachment.
Private opinion) This another issue causes on potential feature inconsistency between feast python sdk and go feature server like this issue. I saw the keys that will request to redis are different each other.(the difference is more including just only \x00(null) data when INT64) Let’s set this issue aside for now.
So to avoid type confusion, I cast from INT32 to INT64 from my source data(I use SparkSource and parquet files in s3) and I simulate the following test
The environment I tested under is as follows:
- When registering and materializing, the entity ID type was set to INT64.
- For registration, materialization, and the Go feature server, the entity_key_serialization_version was fixed to 2.
Currently, when retrieving features using the Feast Python SDK with that entity ID, the lookup works correctly. Even using the Go feature server, the lookup does work well. To investigate more, I ran both the Python SDK and the Go feature server in debug mode, and compared the two Redis keys(e.g python sdk[1] and Go feature server[2]) each used for lookup.
- entity join key name:
sku_id
- entity key value:
60922975
At first, I check the hex binary and string data in below variables in the feast python sdk(normal case)
- variable name:
redis_key_bin
- data:
\x02\x00\x00\x00sku_id\x04\x00\x00\x00\x08\x00\x00\x00_\x9c\xa1\x03\x00\x00\x00\x00recommendations (For easy distinguishment, I split a few pieces)
x02\x00\x00\x00sku_id\x04\x00\x00\x00\x08\x00\x00\x00_\x9c\xa1\\x03\x00\x00\x00\x00recommendations
- variable name:
hset_keys
- data:
[b'\x8d\x06\xaa\xbe', '_ts:sku_sparse_fv']
\x8d\x06\xaa\xbe_ts:sku_sparse_fv
After that, I compare above binary or string data with the data from Go feature server. Below, I build go feature server in debug mode. And I got the binary data and convert hex binary data to string based on ASCII code. The value in parenthesis is converted from ASCII code table.
- variable name:
redisKey
- data:
02000000736b755f696404000000080000005f9ca103000000007265636f6d6d656e646174696f6e73(I split a few pieces, too)
02 00 00 00 => is same with the value in redis_key_bin variable73(s) 6b(k) 75(u) 5f(_) 69(i) 64(d) => this is sku_id. So the same, too04 00 00 00 08 00 00 00 => same, too5f(_) 9c a1 => same, too03 00 00 00 00 => same, too72(r) 65(e) 63(c) 6f(o) 6d(m) 6d(m) 65(e) 6e(n) 64(d) 61(a) 74(t) 69(i) 6f(o) 6e(n) 73(s) => this is recommendations. So the same, too
- variable name:
hsetKeys
- data:
[8d06aabe 5f74733a736b755f7370617273655f6676]
8d 06 aa be => is same with \x8d\x06\xaa\xbe of hset_keys variable in feast python sdk5f(_) 74(t) 73(s) 3a(:) 73(s) 6b(k) 75(u) 5f(_) 73(s) 70(p) 61(a) 72(r) 73(s) 65(e) 5f(_) 66(f) 76(v) => this is _ts:sku_sparse_fv. So the same, too
Finally, Whether feast python sdk or Go feature server, the keys to redis seems to be same. So, due to above simulation, I know the reason whay Go feature server response is not null. I ask you to doubely check it again, please!!
More, I test it below another environment, Go feature server lookup well.
- Source data:
INT32
- register:
INT64
In this test case, materialization is done well, and fetching feature from go feature server is done, too. changed the title [-]Go feature server build wrong redis key that be made from integer entity[/-] [+]Go feature server build wrong redis key that be made from `INT32` type entity[/+] on May 17, 2025jyejare commented on May 17, 2025 @iamcodingcat Perfect Debug. I think you are on the right path here. Reacted by Shuchu Han and Younghun iamcodingcat commented on May 18, 2025 ContributorAuthor More actions @jyejare thanks for review. Then I’ll go ahead and close this issue.However, I do think that at some point the Go feature server should be improved to handle INT32 entity types properly as well. If possible in the future, I’ll try to work on improving that part too! shuchu commented on May 18, 2025 Reacted by Younghun Reacted by Francisco Javier Arceo Reacted by Francisco Javier Arceo
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|
