GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — fix: Keep the metrics server running when a Gunicorn worker exits by aborgatin · Pull Request #7012 · feast-dev/feast

fix: Keep the metrics server running when a Gunicorn worker exits #7012

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

ntkathole merged 4 commits into feast-dev:master from aborgatin:fix/metrics-process-worker-exit

Oct 18, 2026

+89 −1

Merged

fix: Keep the metrics server running when a Gunicorn worker exits #7012

Show file tree

Hide file tree

Changes from all commits

Commits

Show all changes

4 commits

Select commit Hold shift + click to select a range

1f0df5e

fix: Keep the metrics server running when a Gunicorn worker exits

aborgatin Oct 14, 2026

2d0d723

test: Wait for the metrics server before forking in the worker-exit test

aborgatin Oct 14, 2026

0e58508

test: Use an instant stand-in process in the worker-exit test

aborgatin Oct 14, 2026

125f4e8

test: Check active_children() in the forked child instead of running …

aborgatin Oct 16, 2026

File filter

Filter by extension .py (2) All 1 file type selected

Viewed files

Clear filters

Conversations

Failed to load comments.

Uh oh!

There was an error while loading. Please reload this page.

Jump to

Jump to file

Failed to load files.

Uh oh!

There was an error while loading. Please reload this page.

Diff view

Unified

Split

Hide whitespace

Diff view

Unified

Split

Hide whitespace

There are no files selected for viewing

27 changes: 26 additions & 1 deletion 27 sdk/python/feast/metrics.py

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -681,6 +681,27 @@ def _shutdown(signum, frame):
		httpd.serve_forever()


		# The metrics-server process started by start_metrics_server() in this process
		# (the Gunicorn master), and whether the at-fork hook below is registered.
		_metrics_process: Optional[multiprocessing.Process] = None
		_metrics_at_fork_registered: bool = False


		def _forget_metrics_process_in_child() -> None:
		"""Drop the inherited metrics-server process from multiprocessing's registry.

		Runs in every child forked from the process that started the metrics
		server (for example each Gunicorn worker). A forked child inherits
		multiprocessing's registry of child processes, and at interpreter exit
		``multiprocessing.util._exit_function`` terminates every daemon process in
		it. Without this, the first worker exit would SIGTERM the master's metrics
		server and ``/metrics`` would stop responding. The child does not own the
		process; the parent still terminates it on its own exit.
		"""
		if _metrics_process is not None:
		multiprocessing.process._children.discard(_metrics_process) # type: ignore[attr-defined]


		def start_metrics_server(
		store: "FeatureStore",
		port: int = 8000,
Expand Down Expand Up		@@ -714,7 +735,7 @@ def start_metrics_server(
		workers — the ``post_worker_init`` hook starts per-worker
		freshness monitoring instead.
		"""
		global _config
		global _config, _metrics_process, _metrics_at_fork_registered

		if metrics_config is not None:
		_config = metrics_config
Expand All		@@ -739,6 +760,10 @@ def start_metrics_server(
		name="feast-metrics-server",
		)
		metrics_proc.start()
		_metrics_process = metrics_proc
		if not _metrics_at_fork_registered and hasattr(os, "register_at_fork"):
		os.register_at_fork(after_in_child=_forget_metrics_process_in_child)
		_metrics_at_fork_registered = True

		if ready_event.wait(timeout=5):
		logger.info(
Expand Down

63 changes: 63 additions & 0 deletions 63 sdk/python/tests/unit/test_metrics.py

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -13,8 +13,10 @@
		# limitations under the License.

		import http.client
		import os
		import socket
		import threading
		import time
		from datetime import datetime, timedelta, timezone
		from unittest.mock import MagicMock, patch

Expand Down Expand Up		@@ -642,6 +644,67 @@ def test_ready_event_set_on_successful_bind(self):
		mock_event.set.assert_called_once()


		@pytest.mark.skipif(not hasattr(os, "fork"), reason="requires os.fork")
		@pytest.mark.filterwarnings(
		"ignore:This process .* is multi-threaded:DeprecationWarning"
		)
		class TestMetricsProcessSurvivesForkedChildExit:
		"""A forked child (e.g. a Gunicorn worker) must not terminate the metrics server on exit."""

		def test_forked_child_does_not_track_metrics_process(self):
		import multiprocessing

		import feast.metrics as m

		# Stand-in for the metrics-server process: a real daemon process that
		# starts instantly on every platform, registered by start_metrics_server.
		proc = multiprocessing.get_context("fork").Process(
		target=time.sleep, args=(60,), daemon=True
		)
		mock_event = MagicMock()
		mock_event.wait.return_value = True

		with (
		patch("feast.metrics.multiprocessing.Process", return_value=proc),
		patch("feast.metrics.multiprocessing.Event", return_value=mock_event),
		):
		m.start_metrics_server(
		MagicMock(),
		start_resource_monitoring=False,
		start_freshness_monitoring=False,
		)
		try:
		pid = os.fork()
		if pid == 0:
		# A worker's interpreter exit terminates every daemon process in
		# active_children(); report whether the metrics process is in it.
		tracked = proc in multiprocessing.active_children()
		os._exit(1 if tracked else 0)
		_, status = os.waitpid(pid, 0)

		assert os.waitstatus_to_exitcode(status) == 0
		assert proc.is_alive()
		finally:
		proc.kill()
		proc.join(timeout=5)
		m._metrics_process = None


		def test_forget_metrics_process_in_child_removes_it_from_registry():
		import multiprocessing

		import feast.metrics as m

		proc = MagicMock()
		children = {proc}
		with (
		patch.object(multiprocessing.process, "_children", children),
		patch.object(m, "_metrics_process", proc),
		):
		m._forget_metrics_process_in_child()
		assert proc not in children


		class TestMetricsYamlConfig:
		"""Verify metrics config in feature_store.yaml is respected.

Expand Down

Oops, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.