GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — feat: Updating retrieve online documents v2 to work for other fields for sq… by franciscojavierarceo · Pull Request #5082 · feast-dev/feast

Conversation

@franciscojavierarceo

@franciscojavierarceo franciscojavierarceo commented Feb 22, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

What this PR does / why we need it:

This PR enables full text search for the retrieve_online_documents/ endpoint for SQLite Vec. It also establishes a new parameter in the SDK method called query_string that can be passed to use key word search. There are a number of limitations with this approach as the top_k parameter can be misleading (as evident by the example). This offers a good start for keyword search that leverages the existing vector retrieval endpoint. As a next step, enabling hybrid search would be beneficial.

It makes keyword search as simple as:

results = store.retrieve_online_documents_v2(
    features=[
        "document_embeddings:Embeddings",
        "document_embeddings:content",
        "document_embeddings:title",
    ],
    query=query_embedding,
    query_string="(content: 5) OR (title: 1) OR (title: 3)",
    top_k=3,
).to_dict()
print(results)
  • feature_server.py:
    • Added optional query_string parameter to the GetOnlineFeaturesRequest class.
    • Updated retrieve_online_documents to support the query_string parameter.
  • feature_store.py:
    • Added optional query_string parameter to retrieve_online_documents_v2.
    • Updated related methods to handle query_string.
  • feature_view.py:
    • Added an assertion to ensure only one vector feature per feature view.
  • milvus.py:
    • Added optional query_string parameter to retrieve_online_documents_v2.
  • online_store.py:
    • Added optional query_string parameter to retrieve_online_documents_v2.
  • sqlite.py:
    • Extensive changes to support text search with BM25, including adding text_search_enabled configuration and handling query_string.
    • Updated SQL operations to support the new functionalities.
  • passthrough_provider.py and provider.py:
    • Updated retrieve_online_documents_v2 to support the query_string.
  • types.py:
    • Added FEAST_VECTOR_TYPES list for handling vector types.
  • example_feature_repo_1.py:
    • Added content and title fields to an example feature view.

Which issue(s) this PR fixes:

#5081
#5073

Misc

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@franciscojavierarceo franciscojavierarceo changed the title Updating retrieve online documents v2 to work for other fields for sq… feat: Updating retrieve online documents v2 to work for other fields for sq… Feb 22, 2025
@franciscojavierarceo

franciscojavierarceo commented Feb 25, 2025

Copy link
Copy Markdown
Member Author

@HaoXuAI

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@franciscojavierarceo franciscojavierarceo added the ok-to-test label Feb 26, 2025
franciscojavierarceo added 4 commits February 26, 2025 13:35
@franciscojavierarceo
Updating retrieve online documents v2 to work for other fields for sq…
65cd347
…lite

Signed-off-by: Francisco Javier Arceo <[email protected]>
@franciscojavierarceo
updating tests...not working entirely yet but close
198cb20
Signed-off-by: Francisco Javier Arceo <[email protected]>
@franciscojavierarceo
bug fix for addition of new features
3ea7138
Signed-off-by: Francisco Javier Arceo <[email protected]>
@franciscojavierarceo
updated to implement full text search
0dff5f1
Signed-off-by: Francisco Javier Arceo <[email protected]>
@franciscojavierarceo
franciscojavierarceo force-pushed the sqlite-full-text-search branch from 2623df8 to 0dff5f1 Compare February 26, 2025 18:35
haoxu0
haoxu0 reviewed Feb 26, 2025
View reviewed changes
Comment thread sdk/python/feast/types.py
UnixTimestamp: pyarrow.timestamp("us", tz=_utc_now().tzname()),
}

FEAST_VECTOR_TYPES: List[Union[ValueType, PrimitiveFeastType, ComplexFeastType]] = [

@haoxu0 haoxu0 Feb 26, 2025

Copy link
Copy Markdown
Collaborator

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wonder if this is used somewhere? :)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

franciscojavierarceo reacted with laugh emoji
All reactions
  • 😄 1 reaction

@franciscojavierarceo franciscojavierarceo Feb 26, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/feast-dev/feast/pull/5082/files#r1972213724

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
franciscojavierarceo
franciscojavierarceo commented Feb 26, 2025
View reviewed changes
Comment thread sdk/python/feast/infra/online_stores/sqlite.py
vector_bin = serialize_f32(
val.float_list_val.val, config.online_store.vector_len
) # type: ignore
if feature_type_dict[feature_name] in FEAST_VECTOR_TYPES:

@franciscojavierarceo franciscojavierarceo Feb 26, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@HaoXuAI see here!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@franciscojavierarceo
franciscojavierarceo marked this pull request as ready for review February 26, 2025 19:05
@franciscojavierarceo
franciscojavierarceo requested review from a team as code owners February 26, 2025 19:05
@franciscojavierarceo
franciscojavierarceo requested review from ejscribner, haoxu0 and tokoko February 26, 2025 19:05
shuchu
shuchu approved these changes Feb 26, 2025
View reviewed changes
ejscribner
ejscribner approved these changes Feb 26, 2025
View reviewed changes
@franciscojavierarceo
franciscojavierarceo merged commit fc121c3 into master Feb 26, 2025
@franciscojavierarceo

franciscojavierarceo commented Feb 26, 2025

Copy link
Copy Markdown
Member Author

Technically there's a flaw here I need to resolve because you have to pass in a query embedding for just pure text search, which is silly.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@franciscojavierarceo franciscojavierarceo mentioned this pull request Feb 26, 2025
Introduce Hybrid Search API using SQLite FTS5 + Vector search ogx-ai/ogx#1158
Closed
This was referenced Mar 6, 2025
feat(sqlite-vec): enable keyword search for sqlite-vec ogx-ai/ogx#1439
Merged
Add Keyword Search Support to Milvus #5123
Closed
franciscojavierarceo pushed a commit that referenced this pull request Mar 10, 2025
@feast-ci-bot
chore(release): release 0.47.0
3e3afa9
# [0.47.0](v0.46.0...v0.47.0) (2025-03-10)

* feat!: Include PUBLIC_URL in defaultProjectListPromise URL in /ui ([2f0f7b3](2f0f7b3))

### Bug Fixes

* Add transformation_service_endpoit to support Go feature server. ([#5071](#5071)) ([5627d7c](5627d7c))
* Adding extra space on the VM to kind cluster to see if this solves the issue with memory not available with operator e2e tests. ([#5102](#5102)) ([e6e928c](e6e928c))
* Allow unencrypted Snowflake key ([#5097](#5097)) ([87a7c23](87a7c23))
* Cant add different type of list types ([#5118](#5118)) ([bebd7be](bebd7be))
* Fixing transformations on writes ([#5127](#5127)) ([95ac34a](95ac34a))
* Identify s3/remote uri path correctly ([#5076](#5076)) ([93becff](93becff))
* Increase available action VM storage and reduce dev feature-server image size ([#5112](#5112)) ([75f5a90](75f5a90))
* Move Feast to pyproject.toml instead of setup.py ([#5067](#5067)) ([4231274](4231274))
* Skip refresh if already in progress or if lock is already held ([#5068](#5068)) ([f3a24de](f3a24de))

### Features

* Add an OOTB Chat uI to the Feature Server to support RAG demo ([#5106](#5106)) ([40ea7a9](40ea7a9))
* Add Couchbase Columnar as an Offline Store ([#5025](#5025)) ([4373cbf](4373cbf))
* Add Feast Operator RBAC example with Kubernetes Authentication … ([#5077](#5077)) ([2179fbe](2179fbe))
* Added docling and pytorch as add on ([#5089](#5089)) ([135342b](135342b))
* Feast Operator example with Postgres in TLS mode. ([#5028](#5028)) ([2c46f6a](2c46f6a))
* Operator - Add feastProjectDir section to CR with git & init options ([#5079](#5079)) ([d64f01e](d64f01e))
* Override the udf name when provided as input to an on demand transformation ([#5094](#5094)) ([8a714bb](8a714bb))
* Set value_type of entity directly in from_proto ([#5092](#5092)) ([90e7498](90e7498))
* Updating retrieve online documents v2 to work for other fields for sq… ([#5082](#5082)) ([fc121c3](fc121c3))

### BREAKING CHANGES

* The PUBLIC_URL environment variable is now taken into account by default
when fetching the projects list. This is a breaking change only if all
these points apply:

1. You're using Feast UI as a module

2. You're serving the UI files from a non-root path via the PUBLIC_URL
   environment variable

3. You're serving the project list from the root path

4. You're not passing the `feastUIConfigs.projectListPromise` prop to
   the FeastUI component

In this case, you need to explicitly fetch the project list from the
root path via the `feastUIConfigs.projectListPromise` prop:

```diff
 const root = createRoot(document.getElementById("root")!);
 root.render(
   <React.StrictMode>
-    <FeastUI />
+    <FeastUI
+      feastUIConfigs={{
+        projectListPromise: fetch("/projects-list.json", {
+            headers: {
+              "Content-Type": "application/json",
+            },
+          }).then((res) => res.json())
+      }}
+    />
   </React.StrictMode>
 );
```

Signed-off-by: Harri Lehtola <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@shuchu shuchu shuchu approved these changes
@ejscribner ejscribner ejscribner approved these changes
@tokoko tokoko Awaiting requested review from tokoko tokoko is a code owner automatically assigned from feast-dev/maintainers
@haoxu0 haoxu0 Awaiting requested review from haoxu0 haoxu0 is a code owner automatically assigned from feast-dev/maintainers

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

ok-to-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@franciscojavierarceo @shuchu @ejscribner @haoxu0
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.