GeezerVison

Summary

When catalogd runs with 2 replicas and leader election enabled, only the leader pod downloads and stores catalog content locally. Non-leader pods serve the catalog HTTP endpoint but have an empty local cache, so requests that hit the non-leader currently return 404 Not Found.

This is semantically incorrect — the catalog does exist, it is just not available on this particular pod. This change returns 503 Service Unavailable with a Retry-After: 1 header instead, which:

• Correctly signals a temporary, retryable condition
• Prompts well-behaved HTTP clients to retry (many auto-retry on 503)
• Avoids the confusing "not found" signal that implies the catalog does not exist

Changes

• internal/catalogd/server/handlers.go: Changed httpError's fs.ErrNotExist mapping from http.StatusNotFound (404) to http.StatusServiceUnavailable (503), added Retry-After: 1 header, updated message to "catalog content not yet available"
• internal/catalogd/serverutil/serverutil.go: Updated code comments referencing "return 404" to reflect the new 503 behavior
• internal/catalogd/server/handlers_test.go: Updated test assertion for catalog-not-found error to expect 503
• internal/catalogd/storage/localdir_test.go: Updated integration test for non-existent catalog query to expect 503 and the new message

Context

The existing code comments in serverutil.go document this as intentional behavior:

Non-leader pods serve the catalog HTTP port but have an empty local cache (only the leader's reconciler downloads catalog content), so requests to a non-leader return 404. Callers are expected to retry.

The design is correct — callers should retry. But 404 is the wrong HTTP status for this. Per RFC 9110, 404 indicates the resource does not exist, while 503 indicates a temporary unavailability. The Retry-After header provides an explicit hint for when to retry.

No architectural changes — the leader election, readiness probes, and reconciler logic are all unchanged.

AI-generated. Review for accuracy.

@grokspawn requested in Slack thread

Summary by CodeRabbit

• Bug Fixes
  • Catalog requests now return 503 Service Unavailable when content is not yet available on replicas.
  • Added a Retry-After: 1 header and clearer availability message.
  • Leaders continue to return 404 Not Found when catalog content is missing.
  • Invalid If-Modified-Since values now return an explicit 500 Internal Server Error.
  • Improved request handling to prevent incomplete or misleading catalog responses.