GeezerVison is showing github.com — ci: Make the smoke test wait for health checks, and check more by marcleblanc2 · Pull Request #1487 · sourcegraph/deploy-sourcegraph-docker
Conversation
Makes the smoke test start faster and check more. It also changes one health check setting customers run, on three services: searcher-0, precise-code-intel-worker, and syntactic-code-intel-worker get a 300 second start period.
- Faster start. Pulls images in parallel (
docker compose pull), with up to three attempts, because Docker Hub times out now and then.
- Waits for health, not for a clock. Instead of
sleep 90, runs docker compose up --wait --wait-timeout 300. That returns once every container is running, and healthy if it has a health check. It fails if a container reports unhealthy.
- Counts services from the Compose file instead of a hard-coded 25.
migrator is left out, because it exits when it is done.
- Fails if any container restarted. This replaces a loop which said "checking every 10s for 5 minutes" but checked twice, one second apart.
- Saves every container's logs on failure. Before, it saved logs only for containers listed by
docker ps, which hides the ones that exited.
- Uses
docker compose (v2) instead of docker-compose (v1). The VM image has v2.18.1.
What customers will see
searcher-0, precise-code-intel-worker, and syntactic-code-intel-worker get start_period: 300s on their health checks, the same as sourcegraph-frontend-internal. searcher-0 had no start period, and the two workers had 60 seconds.
None of the three is ready until sourcegraph-frontend-internal is, which waits for the migrator. Until then the health check fails, and three failed checks, 5 seconds apart, mark the container unhealthy. So on every start searcher-0 showed unhealthy, then healthy, and the two workers did the same whenever the start took longer than their 60 seconds. Each now shows starting until its first check passes.
Nothing else changes for customers: Docker does not restart an unhealthy container, and no service waits on the health of these three. docker compose up --wait now works for them too.
Risk: low. One health check setting on three services for customers; the rest is the test.
Things to know
--wait gives up when any container reports unhealthy, even for a moment. That is why the three services need the start period.
- In build 16853,
--wait ran without the start period, and failed with container searcher-0 is unhealthy. Seconds later docker ps showed searcher-0 as healthy.
- On the 8.2.0 images, build 16905 of this PR and build 16902 of ci: Check every Compose file and shell script, and test Jaeger and the executor #1489 failed with
container precise-code-intel-worker is unhealthy, 71 seconds after the container started. In both, sourcegraph-frontend-internal started about 30 seconds after the worker, once the migrator was done. The second commit gives the two workers the start period.
Checklist
- All images have a valid tag and SHA256 sum (no image changes)
Test plan
- The smoke test passes in this PR's build: build 16891: passed, the smoke test job took 403 seconds. On the 8.2.0 images, with the start period on the two workers, build 16907 passed in 430 seconds
docker compose config shows the start period on searcher-0, and 25 services without migrator (Compose 2.18.1 and 5.7.1)
shellcheck test/smoke-test.sh is clean
- On a test machine (Docker 29.9.3, Compose 5.7.1), with this PR alone on
main: the smoke test passes in 35 seconds, 25 of 25 containers, no restarts. main's smoke test takes 2 minutes there
- The same run with
searcher-0's start period removed fails after 30 seconds with container searcher-0 is unhealthy, exits 1, and saves 26 log files
Closes FE-586
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
This was referenced Nov 11, 2026
This was referenced Nov 11, 2026
Replaces the 90 second sleep with docker compose up --wait, pulls images in parallel with retries, counts the expected containers from the Compose file, fails if a container restarted, and saves every container's logs on failure.
searcher-0 gets a health check start period, like sourcegraph-frontend-internal. Without it, it reports unhealthy while it waits for the frontend, which fails --wait.
Amp-Thread-ID: https://ampcode.com/threads/T-01a1c7a5-6b65-723b-980c-b85e1e9008b5
Co-authored-by: Amp <[email protected]>
precise-code-intel-worker and syntactic-code-intel-worker are not ready
until sourcegraph-frontend-internal serves them the site configuration,
and that container waits for the migrator. With a 60 second start
period, a slow start marks them unhealthy, and `docker compose up
--wait` fails. Build 16902 failed that way: precise-code-intel-worker
was unhealthy 71 seconds after it started.
searcher-0 got the same start period for the same reason.
Amp-Thread-ID: https://ampcode.com/threads/T-01a1c7a5-6b65-723b-980c-b85e1e9008b5
Co-authored-by: Amp <[email protected]>
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|