There is a change this is by design (quick failures), but I do think there should be a way to handle this more gracefully. Let me know how I can further help this.

Problem

On v0.377.0, an iceberg_input connector in snapshot_and_follow mode reopens the table on every poll. Only throttling errors are retried. Any other error while reopening the table ends the follow loop, and the input stays FATAL until the pipeline restarts.

That includes errors that clear within seconds: a credentials endpoint restarting, an instance-metadata timeout, or a proxy in front of the REST catalog being recreated. Meanwhile health.status stays Healthy, and new snapshots go unread with no visible signal.

We hit this three times on our deployment. Each time a short outage of 20-35 s left the input FATAL:

error refreshing the Iceberg table: ... DispatchFailure ... TimedOutError(5s)
error refreshing the Iceberg table: ... CredentialsNotLoaded("no providers in chain provided credentials")
error refreshing the Iceberg table: ... Failed to execute http request ... /v1/config?warehouse=...

Reproduction

These steps use the stock v0.377.0 image, a Glue catalog on moto, and credentials from AWS_CONTAINER_CREDENTIALS_FULL_URI:

1. Start following the table.
2. Stop the credentials server for 35 s.

The input goes FATAL within about 3 s while health still reads Healthy. After the server returns, the next append is never read: still not read after 180 s.

Proposal

Add an opt-in per-connector option, retry_transient_errors (default false), next to max_retries.

• Off: exactly today's behaviour.
• On:
  • Transient errors on open or refresh are retried with the existing backoff, like throttling. These are credential-load failures, timeouts, connection refused or reset, DNS failures, and HTTP 5xx, 429 and 503.
  • While it retries, the connector reports itself unhealthy.
  • The first open retries a bounded number of times, so a wrong endpoint still fails the pipeline start.
  • Permanent errors stay fatal: access denied, table or namespace not found, invalid metadata.

Open errors would stay text-matched, as the existing is_retryable_open_error does.

Status

We have a draft implementation on current main. It was generated with AI tools (Claude Code and OpenAI Codex) and reviewed by us. It includes unit tests, regenerated OpenAPI and TypeScript types, and docs. We also ran it end to end against the reproduction above.

We will open the PR only if you agree with this approach.