GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
GeezerVison is showing github.com — feat: Add native complex types and scalar index tags to the Milvus online store by jiwidi · Pull Request #7120 · feast-dev/feast Conversation
|
| Feast type | With enable_openai_compatible_store: true |
|---|---|
Array of String, Bool, Int32, Int64, Float32, Float64 or UnixTimestamp, without vector_index |
ARRAY of the element type, with at most milvus.max_capacity elements (default and maximum 4096) |
Json and Array(Json) |
JSON |
Both columns are nullable unless the field has milvus.nullable: "false", so a missing value stays
null. A Json value must be an object or an array: PyMilvus cannot store a top-level string,
number or boolean in a JSON column, and JSON null becomes a null column value. Map and Struct
stay serialized, because JSON cannot hold their nested bytes, timestamps or empty typed lists.
The column types of an existing collection decide how a row is written and read, as #7025 does
for numeric columns, so collections created before this change keep working. Filter keys can
address a JSON path or an array element with Milvus syntax, so retrieve_online_documents_v2 and
online_delete can select rows by them:
ComparisonFilter(type="gt", key='attributes["price"]', value=100) ComparisonFilter(type="eq", key="tags[0]", value="sale")
A filter on a scalar column scans the collection unless the column has an index. Field tags now
declare scalar indexes, which Feast creates with the collection, in the same way the
milvus.partition_key tag declares the partition key:
Field(name="brand_id", dtype=String, tags={"milvus.index": "INVERTED"}), Field(name="price", dtype=Float64, tags={"milvus.index": "STL_SORT"}), Field( name="attributes", dtype=Json, tags={"milvus.json_indexes": '[{"json_path": "attributes[\\"price\\"]", "json_cast_type": "double"}]'}, ),
milvus.index names any scalar index type the server supports, milvus.index_params passes its
parameters and milvus.json_indexes lists JSON path indexes of a JSON column. For an existing
collection without them, Feast logs a warning, as it does for a missing partition key. The
Elasticsearch online store indexes every feature value by default, Qdrant creates payload indexes
on the fields its filters use, and Postgres indexes its filter columns; Milvus filters hit the
feature columns, and those have no index unless one is declared.
The docs describe the migration: the flag only applies when Feast creates a collection, a
feast teardown drops the collections of the whole project, and a staged cutover avoids it.
Benchmark. One million rows on a local single-node Milvus 3.0.2 in Docker, through PyMilvus:
a 1000-value brand text column, a price number, a tags array of two to four values and a
doc JSON object, plus an 8-dimensional HNSW vector. Three collections hold the same rows: text
columns as before this PR, native columns, and native columns with an INVERTED index on brand
and price and a JSON path index on doc["a"]. Each cell is the p50 / p95 latency of 30 runs
after 5 warm-ups, for a plain query with the filter and for a top_k=10 vector search with the
filter, measured after every index reported all rows indexed.
| text columns | native columns | native columns with indexes | |
|---|---|---|---|
| insert one million rows | 14.5 s | 14.0 s | 14.1 s |
brand == 'b_7', query |
3.7 / 6.1 ms | 3.5 / 4.4 ms | 1.4 / 2.1 ms |
brand == 'b_7', search |
4.0 / 5.2 ms | 3.8 / 5.5 ms | 2.5 / 2.7 ms |
price > 990, query |
rejected by Milvus | 1.4 / 1.7 ms | 1.4 / 2.2 ms |
price > 990, search |
rejected by Milvus | 2.0 / 2.8 ms | 2.9 / 3.8 ms |
ARRAY_CONTAINS(tags, 't_7'), query |
rejected by Milvus | 11.8 / 12.8 ms | 10.9 / 11.9 ms |
ARRAY_CONTAINS(tags, 't_7'), search |
rejected by Milvus | 12.4 / 13.1 ms | 14.4 / 15.3 ms |
doc["a"] > 990, query |
rejected by Milvus | 62.8 / 64.0 ms | 3.6 / 4.2 ms |
doc["a"] > 990, search |
rejected by Milvus | 5.1 / 6.4 ms | 4.9 / 6.1 ms |
Native columns cost nothing at write time and make three kinds of filter possible that text
columns reject. The inverted index cuts the equality filter from 3.5 to 1.4 ms, and the JSON path
index cuts the JSON range query from 63 to 3.6 ms. The numeric index changes nothing at this size,
and a filtered vector search gains less than a plain query because the vector search dominates it.
Which issue(s) this PR fixes:
N/A
Checks
- I've made sure the tests are passing.
- My commits are signed off (
git commit -s) - My PR title follows conventional commits format
Testing Strategy
-
Unit tests
-
Integration tests
-
Manual tests
-
Testing is not required for this change
-
Unit (mocked client):
Jsonand arrays get nullableJSONandARRAYcolumns with the
element type,milvus.max_capacityandmax_length,MapstaysVARCHAR;milvus.indexandmilvus.json_indexestags are
passed to Milvus as indexes; the tags are validated; a missing index on an existing collection is
logged. -
Unit (Milvus Lite): twelve complex values, empty arrays included, round-trip through
online_write_batchandonline_read, and a missing value reads as null without any tag; JSON
nullreads as a missing value and a top-level JSON string or number is rejected; a JSON column
tagged not nullable rejects a missing value; a JSON path filter selects search hits and an
array element filter selects rows to delete; the filter key syntax accepts paths and rejects
anything else. -
Server: the same round-trip, the JSON path filter, the array element delete, and the scalar and
JSON path indexes against a real server.
Unit tests run on Milvus Lite 3.2.1 (pymilvus 3.0.2). The server test in
sdk/python/tests/integration/online_store/test_milvus_remote.py passes against a local Milvus
3.0.2 server. Milvus Lite does not support JSON path indexes, so that part runs on the server only.
The existing Milvus unit and server tests pass unchanged.
Misc
Stacked on #7119, whose commit is included here until it merges; review the last
commit. Follows #7082, #7083 and #7084 and Simon Hearne's series #6882, #6895, #6917, #6921 and
#6940. When #7025 lands, native_numeric_types will control these columns too, as it is an alias
of the same flag.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
With the default configuration the Milvus online store could not return most non-numeric Feast types as written. A Bool failed to read, a UnixTimestamp got no column so every write of it failed, an Array(String) failed to read, a numeric array without vector_index was declared as a vector of embedding_dim and could not be written, Bytes came back as base64 text and Json as the base64 of its protobuf. A missing value was stored as the column's default, which failed to read for a numeric text column and came back as 0 or false for a native one. Search hits decoded only primitive types, and entity columns were written as the text of their Python value. The write path now stores every scalar natively or as text, bytes as base64 and every other value as the base64 of its ValueProto, for features and entity columns alike, and one helper rebuilds the ValueProto for reads and searches from every declared field type. UnixTimestamp gets a column, only a field with vector_index=True becomes a vector, with the dimension from Field.vector_length before embedding_dim, and an empty text in a numeric or timestamp column reads as null. Array(Bool) with vector_index is rejected, because Milvus binary vectors were written unpacked and indexed with a float metric and could not work. The new milvus.nullable field tag makes a column nullable, so a missing value is stored and read as null. Signed-off-by: jiwidi <[email protected]>
…line store With enable_openai_compatible_store the Milvus online store stored numeric features as native columns, but arrays and Json values were still base64 text, so a filter could not reach them and Milvus could not validate them. With the flag, an array of scalars without vector_index is now an ARRAY column of the element type, with milvus.max_capacity elements at most, and a Json value, or an array of them, is a JSON column. Both columns are nullable unless the field says otherwise, so a missing value stays null. A Json value must be an object or an array, because PyMilvus cannot store a top-level string, number or bool in a JSON column, and JSON null is stored as a null column value. Map and Struct values stay serialized, because JSON cannot hold their nested bytes, timestamps or empty typed lists. The column types of an existing collection decide how a row is written and read, so collections created before this change keep working. Filter keys can address a JSON path or an array element, for example attributes["price"] or tags[0]. Field tags declare scalar indexes, which Feast creates with the collection: milvus.index names the index type of a scalar column, milvus.index_params passes its parameters and milvus.json_indexes lists JSON path indexes of a JSON column. For an existing collection without them Feast logs a warning, as it does for a missing partition key. Signed-off-by: jiwidi <[email protected]>
codecov-commenter
commented
Nov 9, 2026
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
edited
Uh oh!
There was an error while loading. Please reload this page.
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@ ## master #7120 +/- ## ========================================== + Coverage 51.32% 51.48% +0.15% ========================================== Files 458 458 Lines 57441 57534 +93 Branches 8421 8442 +21 ========================================== + Hits 29482 29620 +138 + Misses 25950 25917 -33 + Partials 2009 1997 -12
... and 1 file with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
All reactions
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
shuchu
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
Uh oh!
There was an error while loading. Please reload this page.
# [0.68.0](v0.67.0...v0.68.0) (2026-11-10) ### Bug Fixes * add watch verb on clusterroles and clusterrolebindings for feast-operator ([#7062](#7062)) ([b6bdda4](b6bdda4)) * Apply on demand feature views in SparkRetrievalJob.to_spark_df() ([#6993](#6993)) ([f416309](f416309)) * Break event timestamp ties by created timestamp in Milvus write batches ([#7084](#7084)) ([bbaca76](bbaca76)) * Break event timestamp ties by created timestamp in the Dask offline store ([#6952](#6952)) ([eb93b08](eb93b08)) * Build the Go feature server image in two stages ([1e5e90c](1e5e90c)), closes [#7060](#7060) * Fall back to the serialized body when UDF source misses globals ([#6816](#6816)) ([f61079f](f61079f)) * Install patched libbrotli1 to address DSA-6582-1 ([#7096](#7096)) ([d458393](d458393)) * Keep description, tags and nested types of ODFV schema fields ([8ef6e75](8ef6e75)) * Keep entity columns out of StreamFeatureView proto features ([#7033](#7033)) ([d520584](d520584)) * Keep the metrics server running when a Gunicorn worker exits ([#7012](#7012)) ([68c5d54](68c5d54)), closes [#7011](#7011) * Key registry proto caches on the proto, not its id() ([#7085](#7085)) ([147adad](147adad)) * Log ODFV materialization skips instead of printing them ([#7117](#7117)) ([8fdcee7](8fdcee7)) * Pin build dependencies of source builds to the CI lock ([#7104](#7104)) ([dec2f24](dec2f24)) * Populate spec.project in registry REST feature view responses (#… ([#7037](#7037)) ([e304a84](e304a84)) * Preserve colliding source timestamps in Dask retrieval ([#7071](#7071)) ([1396c88](1396c88)) * Preserve Dask historical requests without eligible features ([#7000](#7000)) ([fff09f7](fff09f7)) * Read None elements of UUID and Decimal lists as None ([#7039](#7039)) ([03f30fe](03f30fe)) * Retire the support of Python3.10 ([#7101](#7101)) ([a56766e](a56766e)), closes [#7104](#7104) [#7089](#7089) [#7106](#7106) [#7105](#7105) * Serialize UUID and TimeUuid entity keys ([#7032](#7032)) ([0c845a1](0c845a1)) * Stop the Ray engine from hiding materialization errors ([#7026](#7026)) ([7903e2e](7903e2e)), closes [#7024](#7024) * **ui:** stop treating the monitoring baseline as the latest window ([#7056](#7056)) ([0f01802](0f01802)), closes [#7054](#7054) * Use date-only bounds for partition filters built from string timestamps ([#7116](#7116)) ([b1f2168](b1f2168)) * Validate entity column types in feast dbt import ([#7067](#7067)) ([abca80c](abca80c)), closes [#5876](#5876) * Warm up the whole DynamoDB connection pool ([#6985](#6985)) ([8ce548c](8ce548c)) ### Features * Add --gc-freeze option to feast serve ([#7089](#7089)) ([aad0c15](aad0c15)) * Add native complex types and scalar index tags to the Milvus online store ([#7120](#7120)) ([f7068d6](f7068d6)) * Add num_partitions and partition_key_isolation to the Milvus online store ([#7083](#7083)) ([68de84e](68de84e)), closes [#6940](#6940) * Add Pinecone online store integration ([#6585](#6585)) ([7a84f2a](7a84f2a)) * Add PolarisSource and PolarisProvider for Apache Polaris ([#7006](#7006)) ([386759c](386759c)) * Add row deletion to the Milvus online store ([#7082](#7082)) ([d3e2041](d3e2041)) * Add upstream FeatureView lineage support for PushSource ([#6853](#6853)) ([c9b9d72](c9b9d72)) * Optional nameless pre-computed vectors with a schema fingerprint ([#7077](#7077)) ([9420fac](9420fac)) ### Performance Improvements * Scan Lance sources lazily in the DuckDB offline store ([1299e4e](1299e4e))
…line store (feast-dev#7120) * fix: Round-trip every Feast type and null in the Milvus online store With the default configuration the Milvus online store could not return most non-numeric Feast types as written. A Bool failed to read, a UnixTimestamp got no column so every write of it failed, an Array(String) failed to read, a numeric array without vector_index was declared as a vector of embedding_dim and could not be written, Bytes came back as base64 text and Json as the base64 of its protobuf. A missing value was stored as the column's default, which failed to read for a numeric text column and came back as 0 or false for a native one. Search hits decoded only primitive types, and entity columns were written as the text of their Python value. The write path now stores every scalar natively or as text, bytes as base64 and every other value as the base64 of its ValueProto, for features and entity columns alike, and one helper rebuilds the ValueProto for reads and searches from every declared field type. UnixTimestamp gets a column, only a field with vector_index=True becomes a vector, with the dimension from Field.vector_length before embedding_dim, and an empty text in a numeric or timestamp column reads as null. Array(Bool) with vector_index is rejected, because Milvus binary vectors were written unpacked and indexed with a float metric and could not work. The new milvus.nullable field tag makes a column nullable, so a missing value is stored and read as null. Signed-off-by: jiwidi <[email protected]> * feat: Add native complex types and scalar index tags to the Milvus online store With enable_openai_compatible_store the Milvus online store stored numeric features as native columns, but arrays and Json values were still base64 text, so a filter could not reach them and Milvus could not validate them. With the flag, an array of scalars without vector_index is now an ARRAY column of the element type, with milvus.max_capacity elements at most, and a Json value, or an array of them, is a JSON column. Both columns are nullable unless the field says otherwise, so a missing value stays null. A Json value must be an object or an array, because PyMilvus cannot store a top-level string, number or bool in a JSON column, and JSON null is stored as a null column value. Map and Struct values stay serialized, because JSON cannot hold their nested bytes, timestamps or empty typed lists. The column types of an existing collection decide how a row is written and read, so collections created before this change keep working. Filter keys can address a JSON path or an array element, for example attributes["price"] or tags[0]. Field tags declare scalar indexes, which Feast creates with the collection: milvus.index names the index type of a scalar column, milvus.index_params passes its parameters and milvus.json_indexes lists JSON path indexes of a JSON column. For an existing collection without them Feast logs a warning, as it does for a missing partition key. Signed-off-by: jiwidi <[email protected]> --------- Signed-off-by: jiwidi <[email protected]> Signed-off-by: Shuchu Han <[email protected]>
# [0.68.0](feast-dev/feast@v0.67.0...v0.68.0) (2026-11-10) ### Bug Fixes * add watch verb on clusterroles and clusterrolebindings for feast-operator ([feast-dev#7062](feast-dev#7062)) ([b6bdda4](feast-dev@b6bdda4)) * Apply on demand feature views in SparkRetrievalJob.to_spark_df() ([feast-dev#6993](feast-dev#6993)) ([f416309](feast-dev@f416309)) * Break event timestamp ties by created timestamp in Milvus write batches ([feast-dev#7084](feast-dev#7084)) ([bbaca76](feast-dev@bbaca76)) * Break event timestamp ties by created timestamp in the Dask offline store ([feast-dev#6952](feast-dev#6952)) ([eb93b08](feast-dev@eb93b08)) * Build the Go feature server image in two stages ([1e5e90c](feast-dev@1e5e90c)), closes [feast-dev#7060](feast-dev#7060) * Fall back to the serialized body when UDF source misses globals ([feast-dev#6816](feast-dev#6816)) ([f61079f](feast-dev@f61079f)) * Install patched libbrotli1 to address DSA-6582-1 ([feast-dev#7096](feast-dev#7096)) ([d458393](feast-dev@d458393)) * Keep description, tags and nested types of ODFV schema fields ([8ef6e75](feast-dev@8ef6e75)) * Keep entity columns out of StreamFeatureView proto features ([feast-dev#7033](feast-dev#7033)) ([d520584](feast-dev@d520584)) * Keep the metrics server running when a Gunicorn worker exits ([feast-dev#7012](feast-dev#7012)) ([68c5d54](feast-dev@68c5d54)), closes [feast-dev#7011](feast-dev#7011) * Key registry proto caches on the proto, not its id() ([feast-dev#7085](feast-dev#7085)) ([147adad](feast-dev@147adad)) * Log ODFV materialization skips instead of printing them ([feast-dev#7117](feast-dev#7117)) ([8fdcee7](feast-dev@8fdcee7)) * Pin build dependencies of source builds to the CI lock ([feast-dev#7104](feast-dev#7104)) ([dec2f24](feast-dev@dec2f24)) * Populate spec.project in registry REST feature view responses (#… ([feast-dev#7037](feast-dev#7037)) ([e304a84](feast-dev@e304a84)) * Preserve colliding source timestamps in Dask retrieval ([feast-dev#7071](feast-dev#7071)) ([1396c88](feast-dev@1396c88)) * Preserve Dask historical requests without eligible features ([feast-dev#7000](feast-dev#7000)) ([fff09f7](feast-dev@fff09f7)) * Read None elements of UUID and Decimal lists as None ([feast-dev#7039](feast-dev#7039)) ([03f30fe](feast-dev@03f30fe)) * Retire the support of Python3.10 ([feast-dev#7101](feast-dev#7101)) ([a56766e](feast-dev@a56766e)), closes [feast-dev#7104](feast-dev#7104) [feast-dev#7089](feast-dev#7089) [feast-dev#7106](feast-dev#7106) [feast-dev#7105](feast-dev#7105) * Serialize UUID and TimeUuid entity keys ([feast-dev#7032](feast-dev#7032)) ([0c845a1](feast-dev@0c845a1)) * Stop the Ray engine from hiding materialization errors ([feast-dev#7026](feast-dev#7026)) ([7903e2e](feast-dev@7903e2e)), closes [feast-dev#7024](feast-dev#7024) * **ui:** stop treating the monitoring baseline as the latest window ([feast-dev#7056](feast-dev#7056)) ([0f01802](feast-dev@0f01802)), closes [feast-dev#7054](feast-dev#7054) * Use date-only bounds for partition filters built from string timestamps ([feast-dev#7116](feast-dev#7116)) ([b1f2168](feast-dev@b1f2168)) * Validate entity column types in feast dbt import ([feast-dev#7067](feast-dev#7067)) ([abca80c](feast-dev@abca80c)), closes [feast-dev#5876](feast-dev#5876) * Warm up the whole DynamoDB connection pool ([feast-dev#6985](feast-dev#6985)) ([8ce548c](feast-dev@8ce548c)) ### Features * Add --gc-freeze option to feast serve ([feast-dev#7089](feast-dev#7089)) ([aad0c15](feast-dev@aad0c15)) * Add native complex types and scalar index tags to the Milvus online store ([feast-dev#7120](feast-dev#7120)) ([f7068d6](feast-dev@f7068d6)) * Add num_partitions and partition_key_isolation to the Milvus online store ([feast-dev#7083](feast-dev#7083)) ([68de84e](feast-dev@68de84e)), closes [feast-dev#6940](feast-dev#6940) * Add Pinecone online store integration ([feast-dev#6585](feast-dev#6585)) ([7a84f2a](feast-dev@7a84f2a)) * Add PolarisSource and PolarisProvider for Apache Polaris ([feast-dev#7006](feast-dev#7006)) ([386759c](feast-dev@386759c)) * Add row deletion to the Milvus online store ([feast-dev#7082](feast-dev#7082)) ([d3e2041](feast-dev@d3e2041)) * Add upstream FeatureView lineage support for PushSource ([feast-dev#6853](feast-dev#6853)) ([c9b9d72](feast-dev@c9b9d72)) * Optional nameless pre-computed vectors with a schema fingerprint ([feast-dev#7077](feast-dev#7077)) ([9420fac](feast-dev@9420fac)) ### Performance Improvements * Scan Lance sources lazily in the DuckDB offline store ([1299e4e](feast-dev@1299e4e)) Signed-off-by: Shuchu Han <[email protected]>
Reviewers
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assignedLabels
Projects
Milestone
No milestoneDevelopment
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
3 participants
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.