GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — feat: Add Milvus Vector Database Implementation by franciscojavierarceo · Pull Request #4751 · feast-dev/feast

Conversation

franciscojavierarceo commented Nov 11, 2024 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Member

What this PR does / why we need it:

This PR introduces the Milvus Vector Database into Feast. This integration enhances the capability of Feast to handle and store vector embeddings, along with traditional features. More specifically, this allows for vector retrieval configurable at the feature-level.

.github/workflows/pr_local_integration_tests.yml

Updated test cache key

Makefile

Added milvus makefile target

docs/how-to-guides/customizing-feast/adding-support-for-a-new-online-store.md

Updated to remove path

sdk/python/docs/source/feast.infra.online_stores.milvus_online_store.rst

Updated sphinx docs

sdk/python/feast/infra/online_stores/milvus_online_store/milvus.py

Added Milvus online store implementation

sdk/python/feast/repo_config.py

Added milvus to config

sdk/python/feast/type_map.py

Updated _proto_value_to_value_type function

sdk/python/tests/foo_provider.py

Updated requested_features as Optional

sdk/python/tests/integration/online_store/test_universal_online.py

Added specific Milvus test...will likely remove later

Which issue(s) this PR fixes:

#4364

Misc

Will add docs in subsequent PR

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

franciscojavierarceo force-pushed the milvus-vector-db branch 2 times, most recently from a714f85 to 458d434 Compare

November 20, 2024 18:05

franciscojavierarceo force-pushed the milvus-vector-db branch 3 times, most recently from c8a989f to c8f81a5 Compare

November 27, 2024 13:42

franciscojavierarceo force-pushed the milvus-vector-db branch from 9f883cb to 01c9a5f Compare

December 9, 2024 03:36

tchughesiv commented Dec 9, 2024

Copy link

Copy Markdown

Contributor

@franciscojavierarceo as we add support for additional data store types, should we encourage devs to update the operator db store types accordingly as well? or should we treat that as a separate chore since the dev language is different?

feast/infra/feast-operator/api/v1alpha1/featurestore_types.go

Line 161 in 19424bc

// +kubebuilder:validation:Enum=snowflake.online;redis;ikv;datastore;dynamodb;bigtable;postgres;cassandra;mysql;hazelcast;singlestore

feast/infra/feast-operator/api/v1alpha1/featurestore_types.go

Lines 169 to 181 in 19424bc

	var ValidOnlineStoreDBStorePersistenceTypes = []string{
	"snowflake.online",
	"redis",
	"ikv",
	"datastore",
	"dynamodb",
	"bigtable",
	"postgres",
	"cassandra",
	"mysql",
	"hazelcast",
	"singlestore",
	}

looking at the python code, it appears we need to add several more supported types to the operator.

franciscojavierarceo reacted with thumbs up emoji

All reactions

👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

franciscojavierarceo force-pushed the milvus-vector-db branch 2 times, most recently from 83db9bc to a026729 Compare

December 17, 2024 21:28

franciscojavierarceo commented Dec 18, 2024

Copy link

Copy Markdown

Member Author

@franciscojavierarceo as we add support for additional data store types, should we encourage devs to update the operator db store types accordingly as well? or should we treat that as a separate chore since the dev language is different?

feast/infra/feast-operator/api/v1alpha1/featurestore_types.go

Line 161 in 19424bc

// +kubebuilder:validation:Enum=snowflake.online;redis;ikv;datastore;dynamodb;bigtable;postgres;cassandra;mysql;hazelcast;singlestore

feast/infra/feast-operator/api/v1alpha1/featurestore_types.go

Lines 169 to 181 in 19424bc

var ValidOnlineStoreDBStorePersistenceTypes = []string{

"snowflake.online",

"redis",

"ikv",

"datastore",

"dynamodb",

"bigtable",

"postgres",

"cassandra",

"mysql",

"hazelcast",

"singlestore",

}

looking at the python code, it appears we need to add several more supported types to the operator.

Definitely. We can add an explicit test for it too and update the docs. I can do that in a separate PR after I get this thing wrangled.

I'm still only partially through this and I still have quite a bit of work to do.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

franciscojavierarceo force-pushed the milvus-vector-db branch from e27258d to d05d601 Compare

December 18, 2024 22:07

franciscojavierarceo added the ok-to-test label

Dec 20, 2024

franciscojavierarceo marked this pull request as ready for review

December 20, 2024 21:24

franciscojavierarceo requested review from a team as code owners

December 20, 2024 21:24

franciscojavierarceo requested review from haoxu0, shuchu and tokoko

December 20, 2024 21:24

tchughesiv mentioned this pull request

Dec 21, 2024

feat: Add milvus package to release image & option to Operator #4870

Merged

franciscojavierarceo force-pushed the milvus-vector-db branch from e6a3f1f to 218f58b Compare

December 22, 2024 00:59

franciscojavierarceo mentioned this pull request

Dec 22, 2024

feat: Adding features field to retrieve_online_features to return mor… #4869

Merged

franciscojavierarceo added 9 commits

December 21, 2024 21:19


          merged rebase

ab7d4fb

Signed-off-by: Francisco Javier Arceo <[email protected]>


          adding configuration

a6c9005

Signed-off-by: Francisco Javier Arceo <[email protected]>


          updated

b61b965

Signed-off-by: Francisco Javier Arceo <[email protected]>


          changed things and linted

88bca02

Signed-off-by: Francisco Javier Arceo <[email protected]>


          adding updated builds

81bd84a

Signed-off-by: Francisco Javier Arceo <[email protected]>


          adding repo config

083fe67

Signed-off-by: Francisco Javier Arceo <[email protected]>


          updated

fcf32d9

Signed-off-by: Francisco Javier Arceo <[email protected]>


          renaming test and adding milvus to integration test

662119a

Signed-off-by: Francisco Javier Arceo <[email protected]>


          not quite working but have milvus talking

4a7edd8

Signed-off-by: Francisco Javier Arceo <[email protected]>

franciscojavierarceo added 19 commits

December 21, 2024 21:19


          updated setup

2ee74c5

Signed-off-by: Francisco Javier Arceo <[email protected]>


          checking in progresss...getting there

1859e76

Signed-off-by: Francisco Javier Arceo <[email protected]>


          making some progress

35557ce

Signed-off-by: Francisco Javier Arceo <[email protected]>


          partially running

c54309a

Signed-off-by: Francisco Javier Arceo <[email protected]>


          checking in progress...finding issues still

0094e27

Signed-off-by: Francisco Javier Arceo <[email protected]>


          have the apply working

25a065e

Signed-off-by: Francisco Javier Arceo <[email protected]>


          adjusting some type issues

f1a92e0

Signed-off-by: Francisco Javier Arceo <[email protected]>


          updated and removed test

3282a05

Signed-off-by: Francisco Javier Arceo <[email protected]>


          have things behaving with the enviornment arg

2dee462

Signed-off-by: Francisco Javier Arceo <[email protected]>


          updated milvus

be8c5bc

Signed-off-by: Francisco Javier Arceo <[email protected]>


          almost have retrieval working, having to make a lot of changes to onl…

dbc11e2

…ine retrieval. long term this can all go in the FeatureView class and in get_online_features

Signed-off-by: Francisco Javier Arceo <[email protected]>


          almost have deserialization from the search results done

37d93f1

Signed-off-by: Francisco Javier Arceo <[email protected]>


          got the retrieval working now too :D

67ad688

Signed-off-by: Francisco Javier Arceo <[email protected]>


          updates to fix linter and new signature for all implementations

d1c15cf

Signed-off-by: Francisco Javier Arceo <[email protected]>


          linter

6d94d7a

Signed-off-by: Francisco Javier Arceo <[email protected]>


          more linting

fe518ed

Signed-off-by: Francisco Javier Arceo <[email protected]>


          Removing some unnecessary code

a639013

Signed-off-by: Francisco Javier Arceo <[email protected]>


          removing change to setup

c91681f

Signed-off-by: Francisco Javier Arceo <[email protected]>


          adding sphinx docs

bd5ff48

Signed-off-by: Francisco Javier Arceo <[email protected]>

franciscojavierarceo force-pushed the milvus-vector-db branch from 218f58b to bd5ff48 Compare

December 22, 2024 02:19

franciscojavierarceo added 3 commits

December 22, 2024 22:19


          adjusting workflow

32c9a9f

Signed-off-by: Francisco Javier Arceo <[email protected]>


          adding logging to debug

22bac8b

Signed-off-by: Francisco Javier Arceo <[email protected]>


          changing to vectordb environment

134b908

Signed-off-by: Francisco Javier Arceo <[email protected]>

franciscojavierarceo requested review from lokeshrangineni and tchughesiv

December 23, 2024 14:54

shuchu approved these changes

Dec 24, 2024

View reviewed changes

shuchu left a comment

Copy link

Copy Markdown

Collaborator

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

franciscojavierarceo reacted with hooray emoji

All reactions

🎉 1 reaction

franciscojavierarceo merged commit 22c7b58 into master

Dec 24, 2024

dharmisha pushed a commit to nishantgaurav-dev/feast that referenced this pull request

Jan 15, 2025


          feat: Add Milvus Vector Database Implementation (feast-dev#4751)

206c3fc

franciscojavierarceo pushed a commit that referenced this pull request

Jan 20, 2025


          chore(release): release 0.43.0

5a7e6f8

# [0.43.0](v0.42.0...v0.43.0) (2025-01-20)

### Bug Fixes

* Add k8s module to feature-server image ([#4839](#4839)) ([f565565](f565565))
* Adding input to workflow ([e3e8c97](e3e8c97))
* Change image push to use --all-tags option ([#4926](#4926)) ([02458fd](02458fd))
* Fix integration build/push for images ([#4923](#4923)) ([695e49b](695e49b))
* Fix integration operator push ([#4924](#4924)) ([13c7267](13c7267))
* Fix release.yml ([#4845](#4845)) ([b4768a8](b4768a8))
* Fixing some of the warnings with the github actions ([#4763](#4763)) ([1119439](1119439))
* Improve status.applied updates & add offline pvc unit test ([#4871](#4871)) ([3f49517](3f49517))
* Made fixes to Go Operator DB persistence ([#4830](#4830)) ([cdc0753](cdc0753))
* Make transformation_service_endpoint configuration optional ([#4880](#4880)) ([c62377b](c62377b))
* Move pre-release image builds to quay.io, retire gcr.io pushes ([#4922](#4922)) ([40b975b](40b975b))
* Performance regression in /get-online-features ([#4892](#4892)) ([0db56a2](0db56a2))
* Refactor Operator to deploy all feast services to the same Deployment/Pod ([#4863](#4863)) ([88854dd](88854dd))
* Remove unnecessary google cloud steps & upgrade docker action versions ([#4925](#4925)) ([32aaf9a](32aaf9a))
* Remove verifyClient TLS offlineStore option from the Operator ([#4847](#4847)) ([79fa247](79fa247))
* Resolving syntax error while querying a feature view with column name starting with a number and BigQuery as data source ([#4908](#4908)) ([d3495a0](d3495a0))
* Updated python-helm-demo example to use MinIO instead of GS ([#4691](#4691)) ([31afd99](31afd99))

### Features

* Add date field support to spark ([#4913](#4913)) ([a8aeb79](a8aeb79))
* Add date support when converting from python to feast types ([#4918](#4918)) ([bd9f071](bd9f071))
* Add duckdb extra to multicloud release image ([#4862](#4862)) ([b539eba](b539eba))
* Add milvus package to release image & option to Operator ([#4870](#4870)) ([ef724b6](ef724b6))
* Add Milvus Vector Database Implementation ([#4751](#4751)) ([22c7b58](22c7b58))
* Add online/offline replica support ([#4812](#4812)) ([b97da6c](b97da6c))
* Added pvc accessModes support ([#4851](#4851)) ([a73514c](a73514c))
* Adding EnvFrom support for the OptionalConfigs type to the Go Operator ([#4909](#4909)) ([e01e510](e01e510))
* Adding Feature Server to components docs ([#4868](#4868)) ([f95e54b](f95e54b))
* Adding features field to retrieve_online_features to return mor… ([#4869](#4869)) ([7df287e](7df287e))
* Adding packages for Milvus Online Store ([#4854](#4854)) ([49171bd](49171bd))
* Adding vector_search parameter to fields ([#4855](#4855)) ([739eaa7](739eaa7))
* Feast Operator support log level configuration for services ([#4808](#4808)) ([19424bc](19424bc))
* Go Operator - Parsing the output to go structs ([#4832](#4832)) ([732865f](732865f))
* Implement `date_partition_column` for `SparkSource` ([#4844](#4844)) ([c5ffa03](c5ffa03))
* Loading the CA trusted store certificate into Feast to verify the public certificate. ([#4852](#4852)) ([132ce2a](132ce2a))
* Operator E2E test to validate FeatureStore custom resource using remote registry ([#4822](#4822)) ([d558ef7](d558ef7))
* Operator improvements ([#4928](#4928)) ([7a1f4dd](7a1f4dd))
* Removing the tls_verify_client flag from feast cli for offline server. ([#4842](#4842)) ([8320e23](8320e23))
* Separating the RBAC and Remote related integration tests. ([#4905](#4905)) ([76e1e21](76e1e21))
* Snyk vulnerability issues fix. ([#4867](#4867)) ([dbc9207](dbc9207)), closes [#6](#6) [#3](#3) [#4](#4)
* Use ASOF JOIN in Snowflake offline store query ([#4850](#4850)) ([8f591a2](8f591a2))

### Reverts

* Revert "chore: Add Milvus to pr_integration_tests.yml" ([#4900](#4900)) ([07958f7](07958f7)), closes [#4891](#4891)

tmihalac pushed a commit to opendatahub-io/feast that referenced this pull request

Jan 21, 2025


          chore(release): release 0.43.0

520cf53

# [0.43.0](feast-dev/feast@v0.42.0...v0.43.0) (2025-01-20)

### Bug Fixes

* Add k8s module to feature-server image ([feast-dev#4839](feast-dev#4839)) ([f565565](feast-dev@f565565))
* Adding input to workflow ([e3e8c97](feast-dev@e3e8c97))
* Change image push to use --all-tags option ([feast-dev#4926](feast-dev#4926)) ([02458fd](feast-dev@02458fd))
* Fix integration build/push for images ([feast-dev#4923](feast-dev#4923)) ([695e49b](feast-dev@695e49b))
* Fix integration operator push ([feast-dev#4924](feast-dev#4924)) ([13c7267](feast-dev@13c7267))
* Fix release.yml ([feast-dev#4845](feast-dev#4845)) ([b4768a8](feast-dev@b4768a8))
* Fixing some of the warnings with the github actions ([feast-dev#4763](feast-dev#4763)) ([1119439](feast-dev@1119439))
* Improve status.applied updates & add offline pvc unit test ([feast-dev#4871](feast-dev#4871)) ([3f49517](feast-dev@3f49517))
* Made fixes to Go Operator DB persistence ([feast-dev#4830](feast-dev#4830)) ([cdc0753](feast-dev@cdc0753))
* Make transformation_service_endpoint configuration optional ([feast-dev#4880](feast-dev#4880)) ([c62377b](feast-dev@c62377b))
* Move pre-release image builds to quay.io, retire gcr.io pushes ([feast-dev#4922](feast-dev#4922)) ([40b975b](feast-dev@40b975b))
* Performance regression in /get-online-features ([feast-dev#4892](feast-dev#4892)) ([0db56a2](feast-dev@0db56a2))
* Refactor Operator to deploy all feast services to the same Deployment/Pod ([feast-dev#4863](feast-dev#4863)) ([88854dd](feast-dev@88854dd))
* Remove unnecessary google cloud steps & upgrade docker action versions ([feast-dev#4925](feast-dev#4925)) ([32aaf9a](feast-dev@32aaf9a))
* Remove verifyClient TLS offlineStore option from the Operator ([feast-dev#4847](feast-dev#4847)) ([79fa247](feast-dev@79fa247))
* Resolving syntax error while querying a feature view with column name starting with a number and BigQuery as data source ([feast-dev#4908](feast-dev#4908)) ([d3495a0](feast-dev@d3495a0))
* Updated python-helm-demo example to use MinIO instead of GS ([feast-dev#4691](feast-dev#4691)) ([31afd99](feast-dev@31afd99))

### Features

* Add date field support to spark ([feast-dev#4913](feast-dev#4913)) ([a8aeb79](feast-dev@a8aeb79))
* Add date support when converting from python to feast types ([feast-dev#4918](feast-dev#4918)) ([bd9f071](feast-dev@bd9f071))
* Add duckdb extra to multicloud release image ([feast-dev#4862](feast-dev#4862)) ([b539eba](feast-dev@b539eba))
* Add milvus package to release image & option to Operator ([feast-dev#4870](feast-dev#4870)) ([ef724b6](feast-dev@ef724b6))
* Add Milvus Vector Database Implementation ([feast-dev#4751](feast-dev#4751)) ([22c7b58](feast-dev@22c7b58))
* Add online/offline replica support ([feast-dev#4812](feast-dev#4812)) ([b97da6c](feast-dev@b97da6c))
* Added pvc accessModes support ([feast-dev#4851](feast-dev#4851)) ([a73514c](feast-dev@a73514c))
* Adding EnvFrom support for the OptionalConfigs type to the Go Operator ([feast-dev#4909](feast-dev#4909)) ([e01e510](feast-dev@e01e510))
* Adding Feature Server to components docs ([feast-dev#4868](feast-dev#4868)) ([f95e54b](feast-dev@f95e54b))
* Adding features field to retrieve_online_features to return mor… ([feast-dev#4869](feast-dev#4869)) ([7df287e](feast-dev@7df287e))
* Adding packages for Milvus Online Store ([feast-dev#4854](feast-dev#4854)) ([49171bd](feast-dev@49171bd))
* Adding vector_search parameter to fields ([feast-dev#4855](feast-dev#4855)) ([739eaa7](feast-dev@739eaa7))
* Feast Operator support log level configuration for services ([feast-dev#4808](feast-dev#4808)) ([19424bc](feast-dev@19424bc))
* Go Operator - Parsing the output to go structs ([feast-dev#4832](feast-dev#4832)) ([732865f](feast-dev@732865f))
* Implement `date_partition_column` for `SparkSource` ([feast-dev#4844](feast-dev#4844)) ([c5ffa03](feast-dev@c5ffa03))
* Loading the CA trusted store certificate into Feast to verify the public certificate. ([feast-dev#4852](feast-dev#4852)) ([132ce2a](feast-dev@132ce2a))
* Operator E2E test to validate FeatureStore custom resource using remote registry ([feast-dev#4822](feast-dev#4822)) ([d558ef7](feast-dev@d558ef7))
* Operator improvements ([feast-dev#4928](feast-dev#4928)) ([7a1f4dd](feast-dev@7a1f4dd))
* Removing the tls_verify_client flag from feast cli for offline server. ([feast-dev#4842](feast-dev#4842)) ([8320e23](feast-dev@8320e23))
* Separating the RBAC and Remote related integration tests. ([feast-dev#4905](feast-dev#4905)) ([76e1e21](feast-dev@76e1e21))
* Snyk vulnerability issues fix. ([feast-dev#4867](feast-dev#4867)) ([dbc9207](feast-dev@dbc9207)), closes [#6](feast-dev#6) [#3](feast-dev#3) [#4](feast-dev#4)
* Use ASOF JOIN in Snowflake offline store query ([feast-dev#4850](feast-dev#4850)) ([8f591a2](feast-dev@8f591a2))

### Reverts

* Revert "chore: Add Milvus to pr_integration_tests.yml" ([feast-dev#4900](feast-dev#4900)) ([07958f7](feast-dev@07958f7)), closes [feast-dev#4891](feast-dev#4891)

Reviewers

shuchu

shuchu approved these changes

tokoko

Awaiting requested review from tokoko

tokoko was automatically assigned from feast-dev/maintainers

haoxu0

Awaiting requested review from haoxu0

haoxu0 was automatically assigned from feast-dev/maintainers

tchughesiv

Awaiting requested review from tchughesiv

lokeshrangineni

Awaiting requested review from lokeshrangineni

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

ok-to-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

franciscojavierarceo commented Nov 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Misc

Uh oh!

tchughesiv commented Dec 9, 2024

Uh oh!

franciscojavierarceo commented Dec 18, 2024

Uh oh!

shuchu left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

franciscojavierarceo commented Nov 11, 2024 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.