GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — perf: Decouple metrics HTTP server into a dedicated child process by ntkathole · Pull Request #6855 · feast-dev/feast

perf: Decouple metrics HTTP server into a dedicated child process #6855

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

jyejare merged 2 commits into feast-dev:master from ntkathole:perf/decouple-metrics-server

Oct 8, 2026

Uh oh!

There was an error while loading. Please reload this page.

Merged

perf: Decouple metrics HTTP server into a dedicated child process #6855

Show file tree

Hide file tree

Changes from all commits

Commits

Show all changes

2 commits

Select commit Hold shift + click to select a range

00984f7

perf: Decouple metrics HTTP server into a dedicated child process

ntkathole Sep 22, 2026

5f7ea99

Merge branch 'master' into perf/decouple-metrics-server

jyejare Oct 8, 2026

File filter

Filter by extension .py (2) All 1 file type selected

Viewed files

Clear filters

Conversations

Failed to load comments.

Uh oh!

There was an error while loading. Please reload this page.

Jump to

Jump to file

Failed to load files.

Uh oh!

There was an error while loading. Please reload this page.

Diff view

Unified

Split

Hide whitespace

Diff view

Unified

Split

Hide whitespace

There are no files selected for viewing

121 changes: 99 additions & 22 deletions 121 sdk/python/feast/metrics.py

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -34,8 +34,10 @@
		1. ``PROMETHEUS_MULTIPROCESS_DIR`` is set (to a temp dir if the user
		has not already set it) before any metric objects are created.
		2. Gauges specify ``multiprocess_mode`` so they aggregate correctly.
		3. The metrics HTTP server uses ``MultiProcessCollector`` to read all
		workers' metric files.
		3. The metrics HTTP server runs in a dedicated child process
		(not a thread) so that scrape aggregation never contends for the
		GIL with request-serving workers. It uses ``MultiProcessCollector``
		to read all workers' metric files.
		4. Gunicorn hooks (``post_worker_init``, ``child_exit``) are wired up
		in ``feature_server.py`` to start per-worker monitoring and to
		clean up dead-worker files.
Expand All		@@ -44,6 +46,7 @@
		import atexit
		import json
		import logging
		import multiprocessing
		import os
		import shutil
		import socket
Expand Down Expand Up		@@ -588,13 +591,12 @@ def init_worker_freshness_monitoring(store: "FeatureStore"):
		class _QuietWSGIRequestHandler(WSGIRequestHandler):
		"""Request handler that does not log each scrape to stderr.

		The metrics server runs as a thread in the Gunicorn master, which forks
		the workers. The default handler writes an access-log line to stderr for
		every request; a fork while that thread holds the stderr buffer lock
		leaves the new worker with the lock held forever, and the worker hangs
		on its first log line ("Booting worker") before it serves anything.
		Same class of problem as #6647. prometheus_client's own server uses a
		silent handler for the same reason.
		Avoids per-request access-log writes. The metrics server previously ran
		as a thread in the Gunicorn master; a fork while that thread held the
		stderr buffer lock left workers hung on their first log line. Same class
		of problem as #6647. prometheus_client's own server uses a silent
		handler for the same reason. Kept for the dedicated metrics process so
		scrapes stay quiet.
		"""

		def log_message(self, format, *args): # noqa: A002 - signature of the base class
Expand Down Expand Up		@@ -627,6 +629,58 @@ def server_bind(self) -> None:
		)


		def _run_metrics_server(port: int, mp_dir: str, ready_event=None):
		"""Entry point for the dedicated metrics-server child process.

		Runs in its own process so that Prometheus scrape aggregation
		(``MultiProcessCollector`` file reads + text serialization) never
		contends for the GIL with request-serving workers or the Gunicorn
		master.

		The function re-imports ``prometheus_client`` so it works correctly
		on platforms that use the ``spawn`` multiprocessing start method
		(e.g. macOS with Python 3.8+).

		Args:
		port: TCP port for the Prometheus HTTP endpoint.
		mp_dir: Path to the ``PROMETHEUS_MULTIPROCESS_DIR``.
		ready_event: Optional ``multiprocessing.Event`` that is set once
		the server has successfully bound to port. The parent
		process can wait on this to confirm startup succeeded.
		"""
		import signal

		os.environ.setdefault("PROMETHEUS_MULTIPROCESS_DIR", mp_dir)
		os.environ.setdefault("PROMETHEUS_MULTIPROC_DIR", mp_dir)
Comment thread ntkathole marked this conversation as resolved. Resolved Uh oh! There was an error while loading. Please reload this page.

		from prometheus_client import CollectorRegistry, make_wsgi_app
		from prometheus_client.multiprocess import MultiProcessCollector

		try:
		registry = CollectorRegistry()
		MultiProcessCollector(registry)
		httpd = _make_metrics_httpd(port, make_wsgi_app(registry))
		except Exception:
		logger.exception(
		"Failed to initialize Prometheus metrics server on port %d "
		"(multiprocess dir: %s)",
		port,
		mp_dir,
		)
		return

		if ready_event is not None:
		ready_event.set()

		def _shutdown(signum, frame):
		threading.Thread(target=httpd.shutdown, daemon=True).start()

		signal.signal(signal.SIGTERM, _shutdown)
		signal.signal(signal.SIGINT, _shutdown)

		httpd.serve_forever()


		def start_metrics_server(
		store: "FeatureStore",
		port: int = 8000,
Expand All		@@ -637,8 +691,14 @@ def start_metrics_server(
		"""
		Start the Prometheus metrics HTTP server and background monitoring threads.

		Uses ``MultiProcessCollector`` so that metrics from all Gunicorn
		workers are correctly aggregated when Prometheus scrapes port port.
		The HTTP endpoint runs in a dedicated child process so that
		scrape-time aggregation (reading mmap files from every Gunicorn
		worker, merging, and serializing to Prometheus text format) is
		fully isolated from the request-serving GIL.

		Background monitoring threads (resource, freshness) remain in-process
		because they only write to mmap-backed Gauges — an operation that
		is fast and does not benefit from process isolation.

		Args:
		store: The FeatureStore instance (used for freshness checks).
Expand Down Expand Up		@@ -671,18 +731,35 @@ def start_metrics_server(
		audit_logging=False,
		)

		from prometheus_client import CollectorRegistry, make_wsgi_app
		from prometheus_client.multiprocess import MultiProcessCollector

		registry = CollectorRegistry()
		MultiProcessCollector(registry)

		httpd = _make_metrics_httpd(port, make_wsgi_app(registry))
		metrics_thread = threading.Thread(target=httpd.serve_forever, daemon=True)
		metrics_thread.start()
		logger.info(
		"Prometheus metrics server started on port %d (multiprocess-safe)", port
		ready_event = multiprocessing.Event()
		metrics_proc = multiprocessing.Process(
		target=_run_metrics_server,
		args=(port, _prometheus_mp_dir, ready_event),
		daemon=True,
		name="feast-metrics-server",
		)
		metrics_proc.start()

		if ready_event.wait(timeout=5):
		logger.info(
		"Prometheus metrics server started on port %d in dedicated process (pid=%d)",
		port,
		metrics_proc.pid,
		)
		else:
		if not metrics_proc.is_alive():
		logger.error(
		"Prometheus metrics server process exited before becoming ready "
		"(port=%d). Check logs for initialization errors.",
		port,
		)
		else:
		logger.warning(
		"Prometheus metrics server process (pid=%d) did not signal readiness "
		"within 5 s — it may still be starting (port=%d).",
		metrics_proc.pid,
		port,
		)

		if _config.resource and start_resource_monitoring:
		resource_thread = threading.Thread(
Expand Down

Oops, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Oops, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.

You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.