GeezerVison is showing github.com — feat: Add BM25 field tags to the Milvus online store by jiwidi · Pull Request #7144 · feast-dev/feast
Conversation
What this PR does / why we need it:
Builds on #7143, which adds text analysis tags. A String feature can now declare a BM25
index, which Feast creates with the collection:
Field(
name="title",
dtype=String,
tags={
"milvus.bm25": "true",
"milvus.bm25_index_params": '{"inverted_index_algo": "TAAT_NAIVE", "bm25_b": 0.3}',
},
),
milvus.bm25: "true" adds a SPARSE_FLOAT_VECTOR field <field>_bm25, a BM25 function
<field>_bm25_fn and a SPARSE_INVERTED_INDEX bm25_index_<field>. Names are derived from the
field, as for scalar_index_<field> and vector_index_<field>.
milvus.bm25_index_params passes a JSON object of build parameters to that index.
- Writes leave
<field>_bm25 to Milvus: PyMilvus rejects rows that carry a function output.
- A view whose only vector is a BM25 output gets no
_placeholder_vector.
- A feature named like a BM25 output is rejected (PyMilvus accepts duplicate field names).
- Retrieval change:
retrieve_online_documents_v2 now names the only dense vector field even
when it is not a requested feature. Before, Feast passed no anns_field and Milvus inferred
it, which fails with "multiple anns_fields exist" once the collection also has a BM25 output.
No behaviour change for collections without one.
- Feast does not search the BM25 output yet; clients that query Milvus directly can. A
follow-up can add BM25 and hybrid search to retrieve_online_documents_v2.
- An existing collection without the output is reported by the existing schema check, and one
without the index by the missing-index warning.
Which issue(s) this PR fixes:
N/A
Checks
- I've made sure the tests are passing.
- My commits are signed off (
git commit -s)
- My PR title follows conventional commits format
Testing Strategy
-
Unit tests
-
Integration tests
-
Manual tests
-
Testing is not required for this change
-
Unit: the created output, function, index and build params; no placeholder; validation;
writes and re-writes through a fresh store on Milvus Lite (fails without the write change);
the named anns_field (fails without the retrieval change).
-
Integration (test_milvus_remote.py, against Milvus 2.5.27, 2.6.27 and 3.0.2): the server stores the
index params, writes succeed, and a BM25 search on <field>_bm25 finds the row.
-
Milvus Lite does not reproduce the anns_field ambiguity, so the unit test checks the argument
Feast passes.
Misc
Stacked on #7143, whose commit is included here until it merges; review the last commit.
Written with @jonasfagerlund, who is a co-author of the commit.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
The milvus.analyzer_params and milvus.enable_match field tags set the
analyzer and TEXT_MATCH on a String feature when Feast creates the
collection. Feast does not query the tokens; clients that query Milvus
directly can.
Co-authored-by: jonasfagerlund <[email protected]>
Signed-off-by: jonasfagerlund <[email protected]>
Signed-off-by: jiwidi <[email protected]>
milvus.bm25 on a String feature makes Feast add a sparse output, a BM25
function and its index when it creates the collection, and
milvus.bm25_index_params sets that index's build parameters. Writes leave
the output to Milvus, and searches name the dense vector field, which
Milvus can no longer infer once the collection has a BM25 output.
Co-authored-by: jonasfagerlund <[email protected]>
Signed-off-by: jonasfagerlund <[email protected]>
Signed-off-by: jiwidi <[email protected]>
|
⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.
❌ Patch coverage is 98.03922% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 52.32%. Comparing base (e81e69d) to head (32fdb65).
Additional details and impacted files

@@ Coverage Diff @@
## master #7144 +/- ##
==========================================
+ Coverage 52.29% 52.32% +0.02%
==========================================
Files 459 459
Lines 58090 58126 +36
Branches 8560 8568 +8
==========================================
+ Hits 30381 30416 +35
Misses 25629 25629
- Partials 2080 2081 +1
Continue to review full report in Codecov by Harness.
Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e81e69d...32fdb65. Read the comment docs.
🚀 New features to boost your workflow:
- ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
- 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
This branch has not been deployed
No deployments
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|