GeezerVison

Problem

Some customers need a read-only root filesystem on every container, and still need Alertmanager to work

The prometheus service couldn't be read_only, because Alertmanager wrote to the root filesystem

Solution

Add read_only: true to the prometheus service

sourcegraph/sourcegraph#18737 moves Alertmanager's files onto the /prometheus data volume.

Do not merge this PR until the pinned prometheus image includes it.

Test plan

Ran this branch with the image Buildkite built from sourcegraph/sourcegraph#18737:

• prometheus starts read-only, with no read-only or permission errors
• observability.alerts and observability.silenceAlerts reach Alertmanager, and a webhook receives the notifications
• Upgrading from main with the released prometheus:8.1.4690 to this branch works on the same volume
• docker compose stop takes 0.5 seconds, so the default 10 second grace period is enough
• Control: the released prometheus:8.1.4690 fails with open /sg_config_prometheus/alertmanager.yml: read-only file system
• The rest of the results are in sourcegraph/sourcegraph#18737

Linked PRs

Merge in this order, because the two deploy PRs need a prometheus image with the fix:

1. Image fix: sourcegraph/sourcegraph#18737
2. Helm: fix/observability: Prometheus read only root volume deploy-sourcegraph-helm#1000
3. Docker Compose: fix/observability: Prometheus read only root volume #1479

Amp threads

• Design, implementation, and the EKS upgrade test
• Runtime test in Docker and kind, with commands and output
• Earlier Helm-only workaround