GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — ci: Make the smoke test wait for health checks, and check more by marcleblanc2 · Pull Request #1487 · sourcegraph/deploy-sourcegraph-docker

Conversation

@marcleblanc2

@marcleblanc2 marcleblanc2 commented Nov 11, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Makes the smoke test start faster and check more. It also changes one health check setting customers run, on three services: searcher-0, precise-code-intel-worker, and syntactic-code-intel-worker get a 300 second start period.

  • Faster start. Pulls images in parallel (docker compose pull), with up to three attempts, because Docker Hub times out now and then.
  • Waits for health, not for a clock. Instead of sleep 90, runs docker compose up --wait --wait-timeout 300. That returns once every container is running, and healthy if it has a health check. It fails if a container reports unhealthy.
  • Counts services from the Compose file instead of a hard-coded 25. migrator is left out, because it exits when it is done.
  • Fails if any container restarted. This replaces a loop which said "checking every 10s for 5 minutes" but checked twice, one second apart.
  • Saves every container's logs on failure. Before, it saved logs only for containers listed by docker ps, which hides the ones that exited.
  • Uses docker compose (v2) instead of docker-compose (v1). The VM image has v2.18.1.

What customers will see

searcher-0, precise-code-intel-worker, and syntactic-code-intel-worker get start_period: 300s on their health checks, the same as sourcegraph-frontend-internal. searcher-0 had no start period, and the two workers had 60 seconds.

None of the three is ready until sourcegraph-frontend-internal is, which waits for the migrator. Until then the health check fails, and three failed checks, 5 seconds apart, mark the container unhealthy. So on every start searcher-0 showed unhealthy, then healthy, and the two workers did the same whenever the start took longer than their 60 seconds. Each now shows starting until its first check passes.

Nothing else changes for customers: Docker does not restart an unhealthy container, and no service waits on the health of these three. docker compose up --wait now works for them too.

Risk: low. One health check setting on three services for customers; the rest is the test.

Things to know

  • --wait gives up when any container reports unhealthy, even for a moment. That is why the three services need the start period.
  • In build 16853, --wait ran without the start period, and failed with container searcher-0 is unhealthy. Seconds later docker ps showed searcher-0 as healthy.
  • On the 8.2.0 images, build 16905 of this PR and build 16902 of ci: Check every Compose file and shell script, and test Jaeger and the executor #1489 failed with container precise-code-intel-worker is unhealthy, 71 seconds after the container started. In both, sourcegraph-frontend-internal started about 30 seconds after the worker, once the migrator was done. The second commit gives the two workers the start period.

Checklist

  • All images have a valid tag and SHA256 sum (no image changes)

Test plan

  • The smoke test passes in this PR's build: build 16891: passed, the smoke test job took 403 seconds. On the 8.2.0 images, with the start period on the two workers, build 16907 passed in 430 seconds
  • docker compose config shows the start period on searcher-0, and 25 services without migrator (Compose 2.18.1 and 5.7.1)
  • shellcheck test/smoke-test.sh is clean
  • On a test machine (Docker 29.9.3, Compose 5.7.1), with this PR alone on main: the smoke test passes in 35 seconds, 25 of 25 containers, no restarts. main's smoke test takes 2 minutes there
  • The same run with searcher-0's start period removed fails after 30 seconds with container searcher-0 is unhealthy, exits 1, and saves 26 log files

Closes FE-586

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@marcleblanc2
marcleblanc2 added this pull request to stack #1491 November 11, 2026 10:45
@github-actions

github-actions Bot commented Nov 11, 2026

Copy link
Copy Markdown

Open the Diff Tour for this PR in Sourcegraph

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

This was referenced Nov 11, 2026
ci: Speed up the smoke test #1468
Closed
ci: Smoke test on a gcloud VM with a 1 hour TTL, a retry, Jaeger, and the executor #1477
Closed
@marcleblanc2
marcleblanc2 force-pushed the marc/smoke-test-wait-for-health branch from 7298ea5 to d198de1 Compare November 11, 2026 22:23
This was referenced Nov 11, 2026
ci: Check every Compose file and shell script, and test Jaeger and the executor #1489
Draft
amp: Add Amp Orb project setup #1497
Open
@marcleblanc2 @ampagent
ci: Make the smoke test wait for health checks, and check more
3290bd2
Replaces the 90 second sleep with docker compose up --wait, pulls images in parallel with retries, counts the expected containers from the Compose file, fails if a container restarted, and saves every container's logs on failure.

searcher-0 gets a health check start period, like sourcegraph-frontend-internal. Without it, it reports unhealthy while it waits for the frontend, which fails --wait.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1c7a5-6b65-723b-980c-b85e1e9008b5
Co-authored-by: Amp <[email protected]>
@marcleblanc2
marcleblanc2 force-pushed the marc/smoke-test-wait-for-health branch from d198de1 to 3290bd2 Compare November 11, 2026 23:46
@marcleblanc2 @ampagent
docker-compose: Give the code intel workers a 300 second start period
3e0f620
precise-code-intel-worker and syntactic-code-intel-worker are not ready
until sourcegraph-frontend-internal serves them the site configuration,
and that container waits for the migrator. With a 60 second start
period, a slow start marks them unhealthy, and `docker compose up
--wait` fails. Build 16902 failed that way: precise-code-intel-worker
was unhealthy 71 seconds after it started.

searcher-0 got the same start period for the same reason.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1c7a5-6b65-723b-980c-b85e1e9008b5
Co-authored-by: Amp <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@marcleblanc2
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.