GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — feat: Add --gc-freeze option to feast serve by aborgatin · Pull Request #7089 · feast-dev/feast

Conversation

aborgatin commented Oct 29, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor

What this PR does / why we need it:

Adds an opt-in feast serve --gc-freeze flag, also available as feature_server.gc_freeze in feature_store.yaml.

Full GC collections in a feature server worker walk the registry, feature definitions and protobuf classes that live for the whole process. Each one blocks the event loop for 50–75 ms, and the requests waiting at that moment look like slow online store reads. With the flag, the server runs gc.collect() and gc.freeze() in the gunicorn master after the app is built and before the workers fork. Workers inherit the frozen state, including the ones restarted by --max-requests. On Windows, where the server runs on uvicorn without gunicorn, it freezes after get_app. Off by default, so nothing changes for current users.

Also adds a short "Garbage collection pauses" section to the performance tuning guide and the flag to the Python feature server reference.

Which issue(s) this PR fixes:

Fixes #7088

Checks

I've made sure the tests are passing.
My commits are signed off (git commit -s)
My PR title follows conventional commits format

Testing Strategy

Unit tests
Integration tests
Manual tests
Testing is not required for this change

Unit tests check the call order (collect, freeze, then the gunicorn run) for the flag and the config, and that nothing is frozen by default.

Manual test: one worker, 15 req/s for 10 min. Requests over 50 ms went from 16 of 8,991 to 0 of 8,933, p99.9 from 61 to 29 ms. Also tested on a Kubernetes deployment with 0.65.0 and --max-requests, with the same result. Details in #7088.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

aborgatin requested a review from a team as a code owner

October 29, 2026 09:47

codecov-commenter commented Oct 29, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 86.66667% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 51.30%. Comparing base (dec2f24) to head (cf86e1c).

Files with missing lines	Patch %	Lines
sdk/python/feast/feature_server.py	83.33%	2 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7089      +/-   ##
==========================================
+ Coverage   51.28%   51.30%   +0.02%     
==========================================
  Files         458      458              
  Lines       57409    57423      +14     
  Branches     8415     8417       +2     
==========================================
+ Hits        29444    29463      +19     
+ Misses      25957    25950       -7     
- Partials     2008     2010       +2

Flag	Coverage Δ
go-feature-server	`30.58% <ø> (ø)`
python-unit	`52.78% <86.66%> (+0.02%)`	⬆️

Files with missing lines	Coverage Δ
sdk/python/feast/cli/serve.py	`69.38% <100.00%> (+0.31%)`	⬆️
sdk/python/feast/feature_store.py	`46.88% <ø> (ø)`
.../python/feast/infra/feature_servers/base_config.py	`100.00% <100.00%> (ø)`
sdk/python/feast/feature_server.py	`65.63% <83.33%> (+1.19%)`	⬆️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dec2f24...cf86e1c. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

shuchu approved these changes

Nov 4, 2026

View reviewed changes

shuchu left a comment

Copy link

Copy Markdown

Collaborator

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

shuchu commented Nov 4, 2026

Copy link

Copy Markdown

Collaborator

@aborgatin the PR is out-of-date, please rebase the master branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.


          feat: Add --gc-freeze option to feast serve

bf59320

Full garbage collections walk the registry, feature definitions and
protobuf classes that the feature server keeps for its whole life. In a
worker this blocks the event loop for tens of milliseconds and shows up
as slow online store reads. The new option runs gc.collect() and
gc.freeze() after the app is built and before gunicorn forks the
workers. Off by default; also available as feature_server.gc_freeze.

Signed-off-by: Alexander Borgatin <[email protected]>

aborgatin force-pushed the feat/serve-gc-freeze branch from 8ce78b0 to bf59320 Compare

November 4, 2026 07:14

aborgatin commented Nov 4, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor Author

@aborgatin the PR is out-of-date, please rebase the master branch.

Hi @shuchu, thanks for the reminder! I've rebased onto master, with no conflicts. The one failing check (unit-test-python (3.10)) seems unrelated: sqlglotc fails to build on Python 3.10, and it fails on master too. Could you please take another look when you have a moment? Thanks!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

aborgatin added 2 commits

November 5, 2026 11:51


          Merge branch 'master' into feat/serve-gc-freeze


          Merge branch 'master' into feat/serve-gc-freeze

cf86e1c

shuchu merged commit aad0c15 into feast-dev:master

Nov 5, 2026

28 of 30 checks passed

Uh oh!

There was an error while loading. Please reload this page.

shuchu pushed a commit to shuchu/feast that referenced this pull request

Nov 6, 2026


          feat: Add --gc-freeze option to feast serve (feast-dev#7089)

a3d6b5c

Signed-off-by: SHUCHU HAN <[email protected]>

shuchu added a commit that referenced this pull request

Nov 6, 2026


          fix: Retire the support of Python3.10 (#7101)

a56766e

* fix: Retire the support of Python3.10.

Signed-off-by: SHUCHU HAN <[email protected]>
Signed-off-by: SHUCHU HAN <[email protected]>

* fix: Update the pyproject.toml file.

Signed-off-by: Shuchu Han <[email protected]>
Signed-off-by: SHUCHU HAN <[email protected]>

* fix: Update the pixi.lock.

Signed-off-by: Shuchu Han <[email protected]>
Signed-off-by: SHUCHU HAN <[email protected]>

* fix: Update the pixi.lock with python3.11

Signed-off-by: Shuchu Han <[email protected]>
Signed-off-by: SHUCHU HAN <[email protected]>

* fix: Fix snowflake_udf unit test error.

Signed-off-by: Shuchu Han <[email protected]>
Signed-off-by: SHUCHU HAN <[email protected]>

* fix: Pin build dependencies of source builds to the CI lock (#7104)

sqlglotc has no Python 3.10 wheels on PyPI, so on 3.10 uv builds
sqlglotc==30.21.0 from source. Its build pulls the newest unpinned
sqlglot, whose Schema class is no longer a mypyc trait, and mypyc fails
with "Multiple inheritance is not supported (except for traits)".

Pass the CI lock as --build-constraints so source builds use the locked
versions (sqlglot==30.21.0) instead of the newest release.

Signed-off-by: Chaitany Patel <[email protected]>
Signed-off-by: SHUCHU HAN <[email protected]>

* feat: Add --gc-freeze option to feast serve (#7089)

Signed-off-by: SHUCHU HAN <[email protected]>

* ci: Pin Java 17 for HDFS registration tests (#7106)

Signed-off-by: Chaitany Patel <[email protected]>
Co-authored-by: Shuchu Han <[email protected]>
Signed-off-by: SHUCHU HAN <[email protected]>

* ci: Align Pixi version with shared lockfile (#7105)

* Initial plan

* ci: Align Pixi version with shared lockfile

Co-authored-by: shuchu <[email protected]>

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: shuchu <[email protected]>
Co-authored-by: Shuchu Han <[email protected]>
Signed-off-by: SHUCHU HAN <[email protected]>

* Update dependencies in pyproject.toml for ray

Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Signed-off-by: SHUCHU HAN <[email protected]>

* fix: Update the pixi.lock.

Signed-off-by: Shuchu Han <[email protected]>
Signed-off-by: SHUCHU HAN <[email protected]>

---------

Signed-off-by: SHUCHU HAN <[email protected]>
Signed-off-by: SHUCHU HAN <[email protected]>
Signed-off-by: Shuchu Han <[email protected]>
Signed-off-by: Chaitany Patel <[email protected]>
Co-authored-by: SHUCHU HAN <[email protected]>
Co-authored-by: Chaitany Patel <[email protected]>
Co-authored-by: Alexandr Borgatin <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: shuchu <[email protected]>
Co-authored-by: Copilot Autofix powered by AI <[email protected]>

franciscojavierarceo pushed a commit that referenced this pull request

Nov 10, 2026


          chore(release): release 0.68.0

fb5e5d5

# [0.68.0](v0.67.0...v0.68.0) (2026-11-10)

### Bug Fixes

* add watch verb on clusterroles and clusterrolebindings for feast-operator ([#7062](#7062)) ([b6bdda4](b6bdda4))
* Apply on demand feature views in SparkRetrievalJob.to_spark_df() ([#6993](#6993)) ([f416309](f416309))
* Break event timestamp ties by created timestamp in Milvus write batches ([#7084](#7084)) ([bbaca76](bbaca76))
* Break event timestamp ties by created timestamp in the Dask offline store ([#6952](#6952)) ([eb93b08](eb93b08))
* Build the Go feature server image in two stages ([1e5e90c](1e5e90c)), closes [#7060](#7060)
* Fall back to the serialized body when UDF source misses globals ([#6816](#6816)) ([f61079f](f61079f))
* Install patched libbrotli1 to address DSA-6582-1 ([#7096](#7096)) ([d458393](d458393))
* Keep description, tags and nested types of ODFV schema fields ([8ef6e75](8ef6e75))
* Keep entity columns out of StreamFeatureView proto features ([#7033](#7033)) ([d520584](d520584))
* Keep the metrics server running when a Gunicorn worker exits ([#7012](#7012)) ([68c5d54](68c5d54)), closes [#7011](#7011)
* Key registry proto caches on the proto, not its id() ([#7085](#7085)) ([147adad](147adad))
* Log ODFV materialization skips instead of printing them ([#7117](#7117)) ([8fdcee7](8fdcee7))
* Pin build dependencies of source builds to the CI lock ([#7104](#7104)) ([dec2f24](dec2f24))
* Populate spec.project in registry REST feature view responses (#… ([#7037](#7037)) ([e304a84](e304a84))
* Preserve colliding source timestamps in Dask retrieval ([#7071](#7071)) ([1396c88](1396c88))
* Preserve Dask historical requests without eligible features ([#7000](#7000)) ([fff09f7](fff09f7))
* Read None elements of UUID and Decimal lists as None ([#7039](#7039)) ([03f30fe](03f30fe))
* Retire the support of Python3.10 ([#7101](#7101)) ([a56766e](a56766e)), closes [#7104](#7104) [#7089](#7089) [#7106](#7106) [#7105](#7105)
* Serialize UUID and TimeUuid entity keys ([#7032](#7032)) ([0c845a1](0c845a1))
* Stop the Ray engine from hiding materialization errors ([#7026](#7026)) ([7903e2e](7903e2e)), closes [#7024](#7024)
* **ui:** stop treating the monitoring baseline as the latest window ([#7056](#7056)) ([0f01802](0f01802)), closes [#7054](#7054)
* Use date-only bounds for partition filters built from string timestamps ([#7116](#7116)) ([b1f2168](b1f2168))
* Validate entity column types in feast dbt import ([#7067](#7067)) ([abca80c](abca80c)), closes [#5876](#5876)
* Warm up the whole DynamoDB connection pool ([#6985](#6985)) ([8ce548c](8ce548c))

### Features

* Add --gc-freeze option to feast serve ([#7089](#7089)) ([aad0c15](aad0c15))
* Add native complex types and scalar index tags to the Milvus online store ([#7120](#7120)) ([f7068d6](f7068d6))
* Add num_partitions and partition_key_isolation to the Milvus online store ([#7083](#7083)) ([68de84e](68de84e)), closes [#6940](#6940)
* Add Pinecone online store integration ([#6585](#6585)) ([7a84f2a](7a84f2a))
* Add PolarisSource and PolarisProvider for Apache Polaris ([#7006](#7006)) ([386759c](386759c))
* Add row deletion to the Milvus online store ([#7082](#7082)) ([d3e2041](d3e2041))
* Add upstream FeatureView lineage support for PushSource ([#6853](#6853)) ([c9b9d72](c9b9d72))
* Optional nameless pre-computed vectors with a schema fingerprint ([#7077](#7077)) ([9420fac](9420fac))

### Performance Improvements

* Scan Lance sources lazily in the DuckDB offline store ([1299e4e](1299e4e))

patelchaitany mentioned this pull request

Nov 11, 2026

chore: Sync upstream feast-dev/feast stable (0.68.0) opendatahub-io/feast#291

Merged

shuchu pushed a commit to shuchu/feast that referenced this pull request

Nov 11, 2026


          chore(release): release 0.68.0

5b94c9e

# [0.68.0](feast-dev/feast@v0.67.0...v0.68.0) (2026-11-10)

### Bug Fixes

* add watch verb on clusterroles and clusterrolebindings for feast-operator ([feast-dev#7062](feast-dev#7062)) ([b6bdda4](feast-dev@b6bdda4))
* Apply on demand feature views in SparkRetrievalJob.to_spark_df() ([feast-dev#6993](feast-dev#6993)) ([f416309](feast-dev@f416309))
* Break event timestamp ties by created timestamp in Milvus write batches ([feast-dev#7084](feast-dev#7084)) ([bbaca76](feast-dev@bbaca76))
* Break event timestamp ties by created timestamp in the Dask offline store ([feast-dev#6952](feast-dev#6952)) ([eb93b08](feast-dev@eb93b08))
* Build the Go feature server image in two stages ([1e5e90c](feast-dev@1e5e90c)), closes [feast-dev#7060](feast-dev#7060)
* Fall back to the serialized body when UDF source misses globals ([feast-dev#6816](feast-dev#6816)) ([f61079f](feast-dev@f61079f))
* Install patched libbrotli1 to address DSA-6582-1 ([feast-dev#7096](feast-dev#7096)) ([d458393](feast-dev@d458393))
* Keep description, tags and nested types of ODFV schema fields ([8ef6e75](feast-dev@8ef6e75))
* Keep entity columns out of StreamFeatureView proto features ([feast-dev#7033](feast-dev#7033)) ([d520584](feast-dev@d520584))
* Keep the metrics server running when a Gunicorn worker exits ([feast-dev#7012](feast-dev#7012)) ([68c5d54](feast-dev@68c5d54)), closes [feast-dev#7011](feast-dev#7011)
* Key registry proto caches on the proto, not its id() ([feast-dev#7085](feast-dev#7085)) ([147adad](feast-dev@147adad))
* Log ODFV materialization skips instead of printing them ([feast-dev#7117](feast-dev#7117)) ([8fdcee7](feast-dev@8fdcee7))
* Pin build dependencies of source builds to the CI lock ([feast-dev#7104](feast-dev#7104)) ([dec2f24](feast-dev@dec2f24))
* Populate spec.project in registry REST feature view responses (#… ([feast-dev#7037](feast-dev#7037)) ([e304a84](feast-dev@e304a84))
* Preserve colliding source timestamps in Dask retrieval ([feast-dev#7071](feast-dev#7071)) ([1396c88](feast-dev@1396c88))
* Preserve Dask historical requests without eligible features ([feast-dev#7000](feast-dev#7000)) ([fff09f7](feast-dev@fff09f7))
* Read None elements of UUID and Decimal lists as None ([feast-dev#7039](feast-dev#7039)) ([03f30fe](feast-dev@03f30fe))
* Retire the support of Python3.10 ([feast-dev#7101](feast-dev#7101)) ([a56766e](feast-dev@a56766e)), closes [feast-dev#7104](feast-dev#7104) [feast-dev#7089](feast-dev#7089) [feast-dev#7106](feast-dev#7106) [feast-dev#7105](feast-dev#7105)
* Serialize UUID and TimeUuid entity keys ([feast-dev#7032](feast-dev#7032)) ([0c845a1](feast-dev@0c845a1))
* Stop the Ray engine from hiding materialization errors ([feast-dev#7026](feast-dev#7026)) ([7903e2e](feast-dev@7903e2e)), closes [feast-dev#7024](feast-dev#7024)
* **ui:** stop treating the monitoring baseline as the latest window ([feast-dev#7056](feast-dev#7056)) ([0f01802](feast-dev@0f01802)), closes [feast-dev#7054](feast-dev#7054)
* Use date-only bounds for partition filters built from string timestamps ([feast-dev#7116](feast-dev#7116)) ([b1f2168](feast-dev@b1f2168))
* Validate entity column types in feast dbt import ([feast-dev#7067](feast-dev#7067)) ([abca80c](feast-dev@abca80c)), closes [feast-dev#5876](feast-dev#5876)
* Warm up the whole DynamoDB connection pool ([feast-dev#6985](feast-dev#6985)) ([8ce548c](feast-dev@8ce548c))

### Features

* Add --gc-freeze option to feast serve ([feast-dev#7089](feast-dev#7089)) ([aad0c15](feast-dev@aad0c15))
* Add native complex types and scalar index tags to the Milvus online store ([feast-dev#7120](feast-dev#7120)) ([f7068d6](feast-dev@f7068d6))
* Add num_partitions and partition_key_isolation to the Milvus online store ([feast-dev#7083](feast-dev#7083)) ([68de84e](feast-dev@68de84e)), closes [feast-dev#6940](feast-dev#6940)
* Add Pinecone online store integration ([feast-dev#6585](feast-dev#6585)) ([7a84f2a](feast-dev@7a84f2a))
* Add PolarisSource and PolarisProvider for Apache Polaris ([feast-dev#7006](feast-dev#7006)) ([386759c](feast-dev@386759c))
* Add row deletion to the Milvus online store ([feast-dev#7082](feast-dev#7082)) ([d3e2041](feast-dev@d3e2041))
* Add upstream FeatureView lineage support for PushSource ([feast-dev#6853](feast-dev#6853)) ([c9b9d72](feast-dev@c9b9d72))
* Optional nameless pre-computed vectors with a schema fingerprint ([feast-dev#7077](feast-dev#7077)) ([9420fac](feast-dev@9420fac))

### Performance Improvements

* Scan Lance sources lazily in the DuckDB offline store ([1299e4e](feast-dev@1299e4e))

Signed-off-by: Shuchu Han <[email protected]>

rhods-devops-app Bot mentioned this pull request

Nov 12, 2026

Automated Sync from main to stable red-hat-data-services/feast#630

Open

Reviewers

shuchu

shuchu approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Feature server: long GC pauses cause latency spikes on online reads

Uh oh!

There was an error while loading. Please reload this page.

3 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

aborgatin commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Checks

Testing Strategy

Uh oh!

codecov-commenter commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

shuchu left a comment

Choose a reason for hiding this comment

Uh oh!

shuchu commented Nov 4, 2026

Uh oh!

aborgatin commented Nov 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aborgatin commented Oct 29, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

codecov-commenter commented Oct 29, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

aborgatin commented Nov 4, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.