GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — compiler: report an unreachable sccache server in cluster health by jyotshnayaparla-00 · Pull Request #7402 · feldera/feldera

Conversation

jyotshnayaparla-00 commented Oct 6, 2026

Copy link

Copy Markdown

Contributor

When the compiler's sccache server was gone, every Rust compilation on that replica failed, but /v0/cluster_healthz kept reporting the compiler as healthy. Operators could only find the cause through kubectl.

/healthz now accepts check_sccache=true. When builds use sccache and the server must always run (SCCACHE_IDLE_TIMEOUT=0), it fails if nothing answers on the sccache server port. The cluster monitor passes it next to check_storage. Kubernetes probes do not, so this does not restart pods.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions


          compiler: report an unreachable sccache server in cluster health

92d5a7a

When the compiler's sccache server was gone, every Rust compilation on
that replica failed, but /v0/cluster_healthz kept reporting the compiler
as healthy. Operators could only find the cause through kubectl.

/healthz now accepts check_sccache=true. When builds use sccache and the
server must always run (SCCACHE_IDLE_TIMEOUT=0), it fails if nothing
answers on the sccache server port. The cluster monitor passes it next
to check_storage. Kubernetes probes do not, so this does not restart
pods.

jyotshnayaparla-00 self-assigned this

Oct 6, 2026

jyotshnayaparla-00 requested review from gz and swanandx

October 6, 2026 11:39

mythical-fred-oss Bot reviewed

Oct 6, 2026

View reviewed changes

Comment thread

crates/pipeline-manager/src/compiler/main.rs

		Some(
		server_port
		.and_then(\|port\| port.trim().parse().ok())
		.unwrap_or(SCCACHE_DEFAULT_PORT),

mythical-fred-oss Bot Oct 6, 2026

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): sccache also supports SCCACHE_SERVER_UDS (it is in the 0.17 binary), and rust_compiler.rs passes every SCCACHE* var through to cargo. If that var is set, the server listens on a Unix socket and nothing listens on TCP 4226, so this check reports a healthy compiler as unhealthy on every monitor tick. Either return None when SCCACHE_SERVER_UDS is set, or connect to the socket. Please add a test case for it as well.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

mythical-fred-oss Bot reviewed

Oct 6, 2026

View reviewed changes

Comment thread

crates/pipeline-manager/src/compiler/main.rs

		};
		Some(format!(
		"unhealthy: sccache server is not reachable on 127.0.0.1:{port} ({error}); \
		Rust compilation on this compiler fails until the compiler server restarts"

mythical-fred-oss Bot Oct 6, 2026

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): The sccache client starts a server on demand when none is running, so "fails until the compiler server restarts" is not true in every case. A server that crashed may come back on the next build, which would make this report a transient false alarm. If the failure you saw was specifically "Timed out waiting for server startup", say so in the message. Either way, tell the operator what to do (for example, "restart the compiler pod").

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

mythical-fred-oss Bot commented Oct 6, 2026

Copy link

Copy Markdown

Contributor

What I ran: cargo test --locked -p pipeline-manager --lib sccache (debug build). All 3 tests pass: expected_sccache_port, sccache_unavailable_message and the existing extract_sccache_message.

Label	Severity	Item
missing (docs gate)	medium	`/v0/cluster_healthz` can now report the compiler as unhealthy for a new reason, which users can see. `docs/operations/guide.md` (around line 141) documents the storage health check but not this one. Please add a troubleshooting entry for the sccache case.
missing (tests)	medium	No test covers the handler wiring: that `?check_sccache=true` returns 503 with the status message, and that a request without the parameter (the k8s probe) stays healthy while sccache is down.
missing (tests)	medium	No test for `SCCACHE_SERVER_UDS` being set (see inline comment), or for whitespace or a port out of range in `SCCACHE_SERVER_PORT`.

The rest passes the feldera-pr-checks gates. There are no unsafe blocks, no new dependencies and no changes to CI workflows. The commit message is good. The 1 s timeout on a loopback connect is reasonable and does not make the tests flaky.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

swanandx reviewed

Oct 6, 2026

View reviewed changes

swanandx left a comment

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when idle timeout != 0, we aren't really monitoring sccache health right?

I'm not sure how much benefit this provides, but don't have any other recommendations / approaches. would let @gz approve if all good.

Thanks for the PR!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

crates/pipeline-manager/src/compiler/main.rs


		/// Message reported when no sccache server answers on `port`, `None` when one does.
		async fn sccache_unavailable_message(port: u16) -> Option<String> {
		let connect = tokio::net::TcpStream::connect(("127.0.0.1", port));

swanandx Oct 6, 2026

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this enough for determining sccache health? TCP port might be in use, but sccache may be facing some other issues ( e.g. regarding failing to connect to s3? )

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

crates/pipeline-manager/src/cluster_monitor.rs

		// need is down, which makes every Rust compilation fail.
		let compiler_url = format!(
		"{protocol}://{}:{}/healthz?check_storage=true",
		"{protocol}://{}:{}/healthz?check_storage=true&check_sccache=true",

swanandx Oct 6, 2026

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i forgot whether we monitor all replicas of compiler server or just compiler-server-0 , should we monitor all?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

anandbraman mentioned this pull request

Nov 10, 2026

compiler: an sccache server that cargo starts gets no IRSA credentials, so all Rust builds fail #7923

Open

5 tasks

This branch has not been deployed

No deployments

Reviewers

swanandx

swanandx left review comments

mythical-fred-oss[bot]

mythical-fred-oss[bot] left review comments

Awaiting requested review from gz

At least 1 approving review is required to merge this pull request.

Uh oh!

There was an error while loading. Please reload this page.

Assignees

jyotshnayaparla-00

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.