GeezerVison

Fixes #296.

Every probed container in the sourcegraph chart now renders the same three-probe shape from one of four helpers under templates/_helpers/:

Shape: a startup probe (1s period, failureThreshold = startupDeadlineSeconds) that holds liveness and readiness off until it passes; a liveness probe at 10s/5s/3 (restart after ~30s of failures); a readiness probe at 5s/5s. sourcegraph.startupDeadlineSeconds defaults to 300, overridable per service. The three Postgres databases and prometheus default to 3600 for crash recovery, major-version upgrades, and WAL replay.

Why

#296: repo-updater (since removed) had a 1s/3-failure liveness probe with no delay, so a resource-starved pod was killed before it could bind its port. The same pattern, softened by a flat initialDelaySeconds, was still on every Go service: a readiness probe that fires at 0s and fails until startup finishes, and a liveness probe that is useless for the first 60–300s and then can still kill a slow start. /healthz on every sourcegraph/sourcegraph binary is a static 200 the moment the debugserver listens, so a liveness failure during startup only ever means "hasn't bound the port yet". A startup probe is the right tool for that.

Behaviour changes worth a look

• codeinsights-db had no probes at all. The postgresql-16-codeinsights image ships the same /ready.sh and /liveness.sh as postgresql-16; it now matches pgsql and codeintel-db.
• blobstore probed GET / on the S3 API port. It is a svcmain binary like the others, so it now exposes http-debug: 6060 and probes /healthz and /ready there.
• New liveness probes on searcher, cadvisor (/healthz), prometheus (/-/healthy), jaeger (/status), zoekt-indexserver (/debug); new liveness + readiness on grafana (/healthz, /api/health; both are root-level middleware in Grafana 7.5 so they work with serve_from_sub_path).
• zoekt-indexserver gets liveness only. A readiness probe on it would gate the pod's Service endpoints and pull zoekt-webserver out of rotation whenever the indexer blinks.
• zoekt-webserver is unchanged. Per CU-3205 the team decided to keep readiness on /healthz rather than /readyz; zoekt increments Crashes while loading so the aggregator already treats results as incomplete.
• redis-store liveness failureThreshold goes from 2 to the standard 3; redis-cache from 5 to 3. Readiness still fails on LOADING, liveness still passes on an open socket, so a long RDB load keeps the pod alive but out of rotation, same as before.
• syntect-server readiness moves from tcpSocket to /health.
• Startup budgets: frontend 300s (unchanged), workers 60s → 300s, gitserver 120s → 300s. Postgres 3600s (unchanged). Prometheus readiness used to retry for 600s then never kill; it now has a 3600s startup deadline and a /-/healthy liveness probe.
• Pods not touched: the pgsql-exporter and redis-exporter sidecars (never had probes), and the migrator init container.

Every probe, before and after

Rendered from helm template on main and on this branch with syntacticCodeIntel, jaeger, and openTelemetry.agent enabled. Unchanged values are plain; old new marks a change; + value marks a probe that did not exist before. delay is initialDelaySeconds. Where main omitted a field, the Kubernetes default (delay 0 / period 10 / timeout 1 / failures 3) is shown.

startupProbe

livenessProbe

readinessProbe

Checklist

• Follow the manual testing process
• Update changelog
• Update Kubernetes update doc

Test plan

• helm unittest charts/sourcegraph: 33 suites / 204 tests pass, including the new tests/probes_test.yaml (standard shape, global deadline applied through every helper, per-service override, 1h default for databases and prometheus, codeinsights-db probes present, redis/blobstore/grafana/zoekt-indexserver specifics).
• helm lint charts/sourcegraph clean; ./scripts/helm-docs.sh regenerated the README with no stray diff.
• helm template with syntacticCodeIntel, jaeger, and openTelemetry.agent enabled, then dumped every container's startup/liveness/readiness probe to confirm all 21 probed containers render the expected paths, ports, and thresholds.
• Not yet done: deploy to a cluster and confirm blobstore answers on :6060/ready, grafana on :3370/healthz and /api/health, and zoekt-indexserver on :6072/debug. All three were verified against source, not a running pod.