GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — fix: Round-trip every Feast type and null in the Milvus online store by jiwidi · Pull Request #7119 · feast-dev/feast

Conversation

@jiwidi

@jiwidi jiwidi commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Every other online store returns a feature value as it was written, whatever its type, because
Redis, Postgres, SQLite, DynamoDB, Elasticsearch, Qdrant and the rest store the serialized
ValueProto. The Milvus online store keeps one column per feature, and with the default
configuration it could not return most non-numeric types. On upstream master with Milvus Lite:

Feast type Before this PR
Bool the write works, the read raises TypeError
UnixTimestamp no column is created, so every write of the feature view fails
Array(String) the write works, the read raises AttributeError
an array of numbers without vector_index declared as a vector of embedding_dim, so every write fails
Bytes comes back as its base64 text, not the bytes
Json comes back as the base64 of its protobuf, not the JSON text
a missing value stored as "", 0 or false; the read of a numeric text column raises ValueError
any Array, Map or Struct in a search hit comes back as text, because the search path decoded primitive types only
a Bytes or UnixTimestamp entity column written as the text of its Python value, b'...' or a datetime

The write path now stores every scalar natively or as text, bytes as base64, and every other
value as the base64 of its ValueProto, for features and entity columns alike, and one helper
rebuilds the ValueProto for reads and searches from every declared field type. UnixTimestamp gets a column. Only a field with vector_index=True becomes a vector,
with the dimension from Field.vector_length before embedding_dim; the Elasticsearch, Postgres,
Qdrant and MongoDB stores already pick the vector field this way. An empty text in a non-string
column reads as null. Array(Bool) with vector_index=True now raises at collection creation:
it was mapped to a binary vector but written unpacked and indexed with a float metric, so it could
not work.

A missing value is still written as the column's default when the column cannot hold NULL. The
new milvus.nullable: "true" field tag makes a column nullable, so a missing value is stored and
read as null, as every other store does:

Field(name="rating", dtype=Float32, tags={"milvus.nullable": "true"})

Nothing here needs a flag. Every change fixes a call that failed or returned a wrong value, the
read fixes apply to existing collections at once, and the new columns and the tag only apply when
Feast creates a collection, like the partition key.

Which issue(s) this PR fixes:

N/A

Checks

  • I've made sure the tests are passing.
  • My commits are signed off (git commit -s)
  • My PR title follows conventional commits format

Testing Strategy

  • Unit tests

  • Integration tests

  • Manual tests

  • Testing is not required for this change

  • Unit (Milvus Lite): thirteen Feast values, empty bytes included, round-trip through
    online_write_batch and online_read, with and without native numeric types; a missing value
    in a text column reads as null; a milvus.nullable column stores null and a plain column keeps
    its default; search hits decode every type the same way as reads; Bytes and UnixTimestamp
    entity columns round-trip.

  • Unit (mocked client): a numeric array without vector_index is a VARCHAR column, a
    vector_index field takes its vector_length, UnixTimestamp gets a column; the tag is
    rejected on entity, vector and partition key fields and for an invalid value; Array(Bool) with
    vector_index is rejected.

  • Server: the same round-trip and the nullable column against a real server, with and without
    native numeric types.

Unit tests run on Milvus Lite 3.2.1 (pymilvus 3.0.2). The server tests in
sdk/python/tests/integration/online_store/test_milvus_remote.py pass against a local Milvus
3.0.2 server. The existing Milvus unit and server tests pass unchanged.

Misc

Follows #7082, #7083 and #7084 and Simon Hearne's series #6882, #6895, #6917, #6921 and #6940.
The next PR builds on this one: native ARRAY and JSON columns and scalar index tags behind
enable_openai_compatible_store.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@jiwidi
fix: Round-trip every Feast type and null in the Milvus online store
f9be11e
With the default configuration the Milvus online store could not return
most non-numeric Feast types as written. A Bool failed to read, a
UnixTimestamp got no column so every write of it failed, an Array(String)
failed to read, a numeric array without vector_index was declared as a
vector of embedding_dim and could not be written, Bytes came back as
base64 text and Json as the base64 of its protobuf. A missing value was
stored as the column's default, which failed to read for a numeric text
column and came back as 0 or false for a native one. Search hits decoded
only primitive types, and entity columns were written as the text of
their Python value.

The write path now stores every scalar natively or as text, bytes as
base64 and every other value as the base64 of its ValueProto, for
features and entity columns alike, and one helper rebuilds the ValueProto
for reads and searches from every declared field type. UnixTimestamp
gets a column, only a field with vector_index=True becomes a vector, with
the dimension from Field.vector_length before embedding_dim, and an
empty text in a numeric or timestamp column reads as null. Array(Bool)
with vector_index is rejected, because Milvus binary vectors were written
unpacked and indexed with a float metric and could not work. The new
milvus.nullable field tag makes a column nullable, so a missing value is
stored and read as null.

Signed-off-by: jiwidi <[email protected]>
@jiwidi
jiwidi requested a review from a team as a code owner November 9, 2026 13:14
@jiwidi jiwidi mentioned this pull request Nov 9, 2026
feat: Add native complex types and scalar index tags to the Milvus online store #7120
Merged
5 of 7 tasks
@codecov-commenter

codecov-commenter commented Nov 9, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 78.65169% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 51.38%. Comparing base (bbaca76) to head (f9be11e).

Files with missing lines Patch % Lines
.../infra/online_stores/milvus_online_store/milvus.py 78.65% 16 Missing and 3 partials ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #7119      +/-   ##
==========================================
+ Coverage   51.32%   51.38%   +0.06%     
==========================================
  Files         458      458              
  Lines       57441    57434       -7     
  Branches     8421     8414       -7     
==========================================
+ Hits        29482    29513      +31     
+ Misses      25950    25924      -26     
+ Partials     2009     1997      -12     
Flag Coverage Δ
go-feature-server 30.58% <ø> (ø)
python-unit 52.86% <78.65%> (+0.06%) ⬆️
Files with missing lines Coverage Δ
.../infra/online_stores/milvus_online_store/milvus.py 84.50% <78.65%> (+6.90%) ⬆️

... and 1 file with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bbaca76...f9be11e. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jiwidi

jiwidi commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

Closing. This landed with #7120, which included this commit.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jiwidi jiwidi closed this Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@jiwidi @codecov-commenter
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.