GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — fix: Keep the metrics server running when a Gunicorn worker exits by aborgatin · Pull Request #7012 · feast-dev/feast

Conversation

aborgatin commented Oct 14, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor

What this PR does / why we need it:

Since #6855 the metrics server runs in a multiprocessing.Process(daemon=True) started in the Gunicorn master. Every forked worker inherits multiprocessing's registry of child processes, and at worker exit multiprocessing.util._exit_function terminates every daemon process in it. So the first worker restart (--max-requests, worker timeout, reload) sends SIGTERM to the master's metrics server, and /metrics stops responding until feast serve is restarted. Every worker exit also logs AssertionError: can only join a child process. Details and steps to reproduce in #7011.

When the metrics process is started, this registers an os.register_at_fork(after_in_child=...) hook that removes it from the registry in every child forked from that process. An at-fork hook rather than a Gunicorn hook, so it also covers workers that exit before post_worker_init (e.g. a worker that fails to boot). Workers no longer terminate the metrics server; the master still terminates it on its own exit.

Tested with the steps from #7011 (--max-requests 3):

macOS, Python 3.12: /metrics returns 200 after the worker restarts (000 before the change), no AssertionError.
Linux (python:3.12-slim, 0.67.0 with this metrics.py): 0.67.0 as released: /metrics 000 after the first worker exit. With the change, 2 workers: 19 worker exits, /metrics answered all 300 checks, no AssertionError; after stopping feast serve no metrics process is left.

Which issue(s) this PR fixes:

Fixes #7011

Checks

I've made sure the tests are passing. (pytest sdk/python/tests/unit/test_metrics.py: 127 passed; ruff check and ruff format --check pass; mypy feast reports no errors in metrics.py)
My commits are signed off (git commit -s)
My PR title follows conventional commits format

Testing Strategy

Unit tests – TestMetricsProcessSurvivesForkedChildExit: start_metrics_server registers a daemon stand-in process (fork context, so it starts instantly on every platform), a forked child runs multiprocessing.util._exit_function() as a worker does at exit, then the test checks the process is still alive (before the change it is terminated with -SIGTERM). The /metrics HTTP checks are the end-to-end runs above.
Integration tests

🤖 Generated with Claude Code

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

aborgatin requested a review from a team as a code owner

October 14, 2026 13:06

codecov-commenter commented Oct 14, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 88.88889% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 50.99%. Comparing base (1299e4e) to head (125f4e8).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
sdk/python/feast/metrics.py	88.88%	0 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #7012   +/-   ##
=======================================
  Coverage   50.99%   50.99%           
=======================================
  Files         455      455           
  Lines       56970    56979    +9     
  Branches     8313     8315    +2     
=======================================
+ Hits        29051    29059    +8     
  Misses      25935    25935           
- Partials     1984     1985    +1

Flag	Coverage Δ
go-feature-server	`30.58% <ø> (ø)`
python-unit	`52.46% <88.88%> (+<0.01%)`	⬆️

Files with missing lines	Coverage Δ
sdk/python/feast/metrics.py	`86.19% <88.88%> (+0.10%)`	⬆️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1299e4e...125f4e8. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

patelchaitany reviewed

Oct 15, 2026

View reviewed changes

Comment thread

sdk/python/tests/unit/test_metrics.py Outdated

		pid = os.fork()
		if pid == 0:
		# What a worker runs at interpreter exit
		multiprocessing.util._exit_function()

patelchaitany Oct 15, 2026

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I reproduced the worker exit killing the metrics process and confirmed this fix keeps it alive. One small thing: the test runs the full _exit_function() inside pytest, so checking proc not in active_children() in the forked child would be safer.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

aborgatin Oct 16, 2026

Copy link

Copy Markdown

Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for reviewing! Just applied this fix, could you please check

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

patelchaitany approved these changes

Oct 16, 2026

View reviewed changes

patelchaitany left a comment

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update! LGTM.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

patelchaitany approved these changes

Oct 16, 2026

View reviewed changes

haoxu0 added the ok-to-test label

Oct 17, 2026

— with

ChatGPT Codex Connector

ntkathole approved these changes

Oct 18, 2026

View reviewed changes

aborgatin and others added 4 commits

October 18, 2026 18:57


          fix: Keep the metrics server running when a Gunicorn worker exits

1f0df5e

Since the metrics server runs in a multiprocessing.Process started in
the Gunicorn master, every forked worker inherits multiprocessing's
registry of child processes. At worker exit,
multiprocessing.util._exit_function terminates every daemon process in
that registry, so the first worker restart (--max-requests, timeout,
reload) sends SIGTERM to the master's metrics server and /metrics stops
responding. It also logs "AssertionError: can only join a child
process" on every worker exit.

Register an at-fork hook when the metrics process is started that
removes it from the registry in every forked child. Workers no longer
terminate it; the master still does on its own exit.

Fixes feast-dev#7011

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Signed-off-by: Alexandr Borgatin <[email protected]>


          test: Wait for the metrics server before forking in the worker-exit test

2d0d723

On macOS CI the spawned metrics process can take longer than the 5 s
readiness wait to start listening, so the first /metrics request was
refused. Poll /metrics (up to 30 s) before forking the child.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Signed-off-by: Alexandr Borgatin <[email protected]>


          test: Use an instant stand-in process in the worker-exit test

0e58508

The spawned metrics server did not come up within 30 s inside the full
unit suite on macOS CI. The test is about the process surviving a
forked child's exit, not about serving HTTP, so register a fork-context
daemon process via start_metrics_server instead and assert it is still
alive after the child runs multiprocessing.util._exit_function().
Without the fix it is terminated (exitcode -SIGTERM).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Signed-off-by: Alexandr Borgatin <[email protected]>


          test: Check active_children() in the forked child instead of running …

125f4e8

…_exit_function

Running multiprocessing.util._exit_function() in a fork of the pytest
process runs every registered finalizer. Check what a worker's exit
would act on instead: the metrics process must not be in the child's
active_children(). Also cover _forget_metrics_process_in_child directly,
and kill the stand-in process in cleanup (terminate() can wait out the
join timeout under xdist).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Signed-off-by: Alexandr Borgatin <[email protected]>

ntkathole force-pushed the fix/metrics-process-worker-exit branch from 80dc00f to 125f4e8 Compare

October 18, 2026 13:27

ntkathole merged commit 68c5d54 into feast-dev:master

Oct 18, 2026

21 of 23 checks passed

Uh oh!

There was an error while loading. Please reload this page.

franciscojavierarceo pushed a commit that referenced this pull request

Nov 10, 2026


          chore(release): release 0.68.0

fb5e5d5

# [0.68.0](v0.67.0...v0.68.0) (2026-11-10)

### Bug Fixes

* add watch verb on clusterroles and clusterrolebindings for feast-operator ([#7062](#7062)) ([b6bdda4](b6bdda4))
* Apply on demand feature views in SparkRetrievalJob.to_spark_df() ([#6993](#6993)) ([f416309](f416309))
* Break event timestamp ties by created timestamp in Milvus write batches ([#7084](#7084)) ([bbaca76](bbaca76))
* Break event timestamp ties by created timestamp in the Dask offline store ([#6952](#6952)) ([eb93b08](eb93b08))
* Build the Go feature server image in two stages ([1e5e90c](1e5e90c)), closes [#7060](#7060)
* Fall back to the serialized body when UDF source misses globals ([#6816](#6816)) ([f61079f](f61079f))
* Install patched libbrotli1 to address DSA-6582-1 ([#7096](#7096)) ([d458393](d458393))
* Keep description, tags and nested types of ODFV schema fields ([8ef6e75](8ef6e75))
* Keep entity columns out of StreamFeatureView proto features ([#7033](#7033)) ([d520584](d520584))
* Keep the metrics server running when a Gunicorn worker exits ([#7012](#7012)) ([68c5d54](68c5d54)), closes [#7011](#7011)
* Key registry proto caches on the proto, not its id() ([#7085](#7085)) ([147adad](147adad))
* Log ODFV materialization skips instead of printing them ([#7117](#7117)) ([8fdcee7](8fdcee7))
* Pin build dependencies of source builds to the CI lock ([#7104](#7104)) ([dec2f24](dec2f24))
* Populate spec.project in registry REST feature view responses (#… ([#7037](#7037)) ([e304a84](e304a84))
* Preserve colliding source timestamps in Dask retrieval ([#7071](#7071)) ([1396c88](1396c88))
* Preserve Dask historical requests without eligible features ([#7000](#7000)) ([fff09f7](fff09f7))
* Read None elements of UUID and Decimal lists as None ([#7039](#7039)) ([03f30fe](03f30fe))
* Retire the support of Python3.10 ([#7101](#7101)) ([a56766e](a56766e)), closes [#7104](#7104) [#7089](#7089) [#7106](#7106) [#7105](#7105)
* Serialize UUID and TimeUuid entity keys ([#7032](#7032)) ([0c845a1](0c845a1))
* Stop the Ray engine from hiding materialization errors ([#7026](#7026)) ([7903e2e](7903e2e)), closes [#7024](#7024)
* **ui:** stop treating the monitoring baseline as the latest window ([#7056](#7056)) ([0f01802](0f01802)), closes [#7054](#7054)
* Use date-only bounds for partition filters built from string timestamps ([#7116](#7116)) ([b1f2168](b1f2168))
* Validate entity column types in feast dbt import ([#7067](#7067)) ([abca80c](abca80c)), closes [#5876](#5876)
* Warm up the whole DynamoDB connection pool ([#6985](#6985)) ([8ce548c](8ce548c))

### Features

* Add --gc-freeze option to feast serve ([#7089](#7089)) ([aad0c15](aad0c15))
* Add native complex types and scalar index tags to the Milvus online store ([#7120](#7120)) ([f7068d6](f7068d6))
* Add num_partitions and partition_key_isolation to the Milvus online store ([#7083](#7083)) ([68de84e](68de84e)), closes [#6940](#6940)
* Add Pinecone online store integration ([#6585](#6585)) ([7a84f2a](7a84f2a))
* Add PolarisSource and PolarisProvider for Apache Polaris ([#7006](#7006)) ([386759c](386759c))
* Add row deletion to the Milvus online store ([#7082](#7082)) ([d3e2041](d3e2041))
* Add upstream FeatureView lineage support for PushSource ([#6853](#6853)) ([c9b9d72](c9b9d72))
* Optional nameless pre-computed vectors with a schema fingerprint ([#7077](#7077)) ([9420fac](9420fac))

### Performance Improvements

* Scan Lance sources lazily in the DuckDB offline store ([1299e4e](1299e4e))

patelchaitany mentioned this pull request

Nov 11, 2026

chore: Sync upstream feast-dev/feast stable (0.68.0) opendatahub-io/feast#291

Merged

shuchu pushed a commit to shuchu/feast that referenced this pull request

Nov 11, 2026


          chore(release): release 0.68.0

5b94c9e

# [0.68.0](feast-dev/feast@v0.67.0...v0.68.0) (2026-11-10)

### Bug Fixes

* add watch verb on clusterroles and clusterrolebindings for feast-operator ([feast-dev#7062](feast-dev#7062)) ([b6bdda4](feast-dev@b6bdda4))
* Apply on demand feature views in SparkRetrievalJob.to_spark_df() ([feast-dev#6993](feast-dev#6993)) ([f416309](feast-dev@f416309))
* Break event timestamp ties by created timestamp in Milvus write batches ([feast-dev#7084](feast-dev#7084)) ([bbaca76](feast-dev@bbaca76))
* Break event timestamp ties by created timestamp in the Dask offline store ([feast-dev#6952](feast-dev#6952)) ([eb93b08](feast-dev@eb93b08))
* Build the Go feature server image in two stages ([1e5e90c](feast-dev@1e5e90c)), closes [feast-dev#7060](feast-dev#7060)
* Fall back to the serialized body when UDF source misses globals ([feast-dev#6816](feast-dev#6816)) ([f61079f](feast-dev@f61079f))
* Install patched libbrotli1 to address DSA-6582-1 ([feast-dev#7096](feast-dev#7096)) ([d458393](feast-dev@d458393))
* Keep description, tags and nested types of ODFV schema fields ([8ef6e75](feast-dev@8ef6e75))
* Keep entity columns out of StreamFeatureView proto features ([feast-dev#7033](feast-dev#7033)) ([d520584](feast-dev@d520584))
* Keep the metrics server running when a Gunicorn worker exits ([feast-dev#7012](feast-dev#7012)) ([68c5d54](feast-dev@68c5d54)), closes [feast-dev#7011](feast-dev#7011)
* Key registry proto caches on the proto, not its id() ([feast-dev#7085](feast-dev#7085)) ([147adad](feast-dev@147adad))
* Log ODFV materialization skips instead of printing them ([feast-dev#7117](feast-dev#7117)) ([8fdcee7](feast-dev@8fdcee7))
* Pin build dependencies of source builds to the CI lock ([feast-dev#7104](feast-dev#7104)) ([dec2f24](feast-dev@dec2f24))
* Populate spec.project in registry REST feature view responses (#… ([feast-dev#7037](feast-dev#7037)) ([e304a84](feast-dev@e304a84))
* Preserve colliding source timestamps in Dask retrieval ([feast-dev#7071](feast-dev#7071)) ([1396c88](feast-dev@1396c88))
* Preserve Dask historical requests without eligible features ([feast-dev#7000](feast-dev#7000)) ([fff09f7](feast-dev@fff09f7))
* Read None elements of UUID and Decimal lists as None ([feast-dev#7039](feast-dev#7039)) ([03f30fe](feast-dev@03f30fe))
* Retire the support of Python3.10 ([feast-dev#7101](feast-dev#7101)) ([a56766e](feast-dev@a56766e)), closes [feast-dev#7104](feast-dev#7104) [feast-dev#7089](feast-dev#7089) [feast-dev#7106](feast-dev#7106) [feast-dev#7105](feast-dev#7105)
* Serialize UUID and TimeUuid entity keys ([feast-dev#7032](feast-dev#7032)) ([0c845a1](feast-dev@0c845a1))
* Stop the Ray engine from hiding materialization errors ([feast-dev#7026](feast-dev#7026)) ([7903e2e](feast-dev@7903e2e)), closes [feast-dev#7024](feast-dev#7024)
* **ui:** stop treating the monitoring baseline as the latest window ([feast-dev#7056](feast-dev#7056)) ([0f01802](feast-dev@0f01802)), closes [feast-dev#7054](feast-dev#7054)
* Use date-only bounds for partition filters built from string timestamps ([feast-dev#7116](feast-dev#7116)) ([b1f2168](feast-dev@b1f2168))
* Validate entity column types in feast dbt import ([feast-dev#7067](feast-dev#7067)) ([abca80c](feast-dev@abca80c)), closes [feast-dev#5876](feast-dev#5876)
* Warm up the whole DynamoDB connection pool ([feast-dev#6985](feast-dev#6985)) ([8ce548c](feast-dev@8ce548c))

### Features

* Add --gc-freeze option to feast serve ([feast-dev#7089](feast-dev#7089)) ([aad0c15](feast-dev@aad0c15))
* Add native complex types and scalar index tags to the Milvus online store ([feast-dev#7120](feast-dev#7120)) ([f7068d6](feast-dev@f7068d6))
* Add num_partitions and partition_key_isolation to the Milvus online store ([feast-dev#7083](feast-dev#7083)) ([68de84e](feast-dev@68de84e)), closes [feast-dev#6940](feast-dev#6940)
* Add Pinecone online store integration ([feast-dev#6585](feast-dev#6585)) ([7a84f2a](feast-dev@7a84f2a))
* Add PolarisSource and PolarisProvider for Apache Polaris ([feast-dev#7006](feast-dev#7006)) ([386759c](feast-dev@386759c))
* Add row deletion to the Milvus online store ([feast-dev#7082](feast-dev#7082)) ([d3e2041](feast-dev@d3e2041))
* Add upstream FeatureView lineage support for PushSource ([feast-dev#6853](feast-dev#6853)) ([c9b9d72](feast-dev@c9b9d72))
* Optional nameless pre-computed vectors with a schema fingerprint ([feast-dev#7077](feast-dev#7077)) ([9420fac](feast-dev@9420fac))

### Performance Improvements

* Scan Lance sources lazily in the DuckDB offline store ([1299e4e](feast-dev@1299e4e))

Signed-off-by: Shuchu Han <[email protected]>

rhods-devops-app Bot mentioned this pull request

Nov 12, 2026

Automated Sync from main to stable red-hat-data-services/feast#630

Open

Reviewers

ntkathole

ntkathole approved these changes

+1 more reviewer

patelchaitany

patelchaitany approved these changes

Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

ok-to-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feast serve: /metrics stops responding after the first gunicorn worker exits (regression in 0.67.0, #6855)

Uh oh!

There was an error while loading. Please reload this page.

5 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

aborgatin commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Checks

Testing Strategy

Uh oh!

codecov-commenter commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

patelchaitany Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

aborgatin Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

patelchaitany left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

aborgatin commented Oct 14, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

codecov-commenter commented Oct 14, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.