Resilience

Tune retries and the per-provider circuit breaker, and learn when to reach for cross-model failover instead.

GoModel wraps every upstream provider call with two resilience layers:

1. Retry with exponential backoff — repeats a failed request against the same provider with growing delays.
2. Circuit breaker — short-circuits calls to a provider that has been failing repeatedly, then probes once the timeout elapses.

Both layers apply per provider. They do not switch to a different model or provider on failure. For cross-model failover, see Failover.

Defaults

The defaults are tuned to be safe for most deployments. Override only what you need.

Setting	Default	Notes
max_retries	3	Maximum retry attempts per request
initial_backoff	1s	First retry wait
max_backoff	30s	Upper cap on retry wait
backoff_factor	2.0	Exponential multiplier between retries
jitter_factor	0.1	Random jitter as a fraction of the backoff
failure_threshold	5	Consecutive failures before the circuit opens
success_threshold	2	Consecutive successes to close it again
timeout	30s	How long the circuit stays open before probing

To disable a layer, zero out its trigger: max_retries: 0 gives every request exactly one attempt, and failure_threshold: 0 disables the circuit breaker entirely. Both work globally or per provider.

What counts as a failure

Retries fire on transport errors (connection refused, resets, DNS failures) and on 429, 502, 503, and 504 responses. Other statuses — including 500 — are returned to the caller without retrying. Two paths retry less than the table suggests:

• Streaming requests are never retried once dispatched, because partial data may already have been sent.
• Passthrough requests are retried only when they are replay-safe: GET, HEAD, OPTIONS, PUT, or any request carrying an Idempotency-Key header.

The circuit breaker counts transport errors and 5xx responses (including non-retried 500s) as failures. It deliberately ignores:

• 429 rate limits — a rate-limited provider is up, so the backpressure signal is passed through instead of opening the circuit. The one exception is a 429 on a half-open probe, which reopens the circuit.
• 4xx responses — the provider answered; the request was at fault.
• Requests canceled by the client — a disconnect says nothing about provider health. Client-side timeouts do count as failures.

While the circuit is open, requests fail fast with a 503 and the message circuit breaker is open - provider temporarily unavailable. After timeout elapses, a single probe request is let through while concurrent requests keep failing fast; success_threshold consecutive successful probes close the circuit, and any probe failure reopens it.

The circuit breaker is in-memory and per gateway process: each provider gets its own breaker, state resets on restart, and nothing is shared between replicas.

Environment Variables

These set the global defaults that apply to every provider unless overridden in YAML.

Retry

Variable	Type	Default	Description
RETRY_MAX_RETRIES	int	3	Maximum retry attempts per request
RETRY_INITIAL_BACKOFF	duration	1s	First retry wait (e.g. 500ms, 2s)
RETRY_MAX_BACKOFF	duration	30s	Upper cap on retry wait
RETRY_BACKOFF_FACTOR	float	2.0	Exponential multiplier between retries
RETRY_JITTER_FACTOR	float	0.1	Random jitter as a fraction of the backoff

Circuit Breaker

Variable	Type	Default	Description
CIRCUIT_BREAKER_FAILURE_THRESHOLD	int	5	Consecutive failures before opening
CIRCUIT_BREAKER_SUCCESS_THRESHOLD	int	2	Consecutive successes to close again
CIRCUIT_BREAKER_TIMEOUT	duration	30s	How long the circuit stays open before probing

YAML

The same fields are available under the global resilience: block, and can be overridden per provider:

resilience:
  retry:
    max_retries: 2
    initial_backoff: 500ms
    max_backoff: 10s
    backoff_factor: 1.5
    jitter_factor: 0.05
  circuit_breaker:
    failure_threshold: 3
    success_threshold: 1
    timeout: 15s

providers:
  anthropic:
    type: anthropic
    api_key: ${ANTHROPIC_API_KEY}
    resilience:
      retry:
        max_retries: 5 # Anthropic supports long requests — allow more retries

  ollama:
    type: ollama
    base_url: ${OLLAMA_BASE_URL:-http://localhost:11434/v1}
    resilience:
      circuit_breaker:
        failure_threshold: 10 # local service — tolerate more transient failures
        timeout: 5s

Only fields explicitly listed under a provider’s resilience: block are overridden. Everything else inherits from the global section, which in turn inherits from the built-in defaults.

Per-provider tuning must come from YAML. Environment variables set global defaults only — RETRY_MAX_RETRIES cannot target a single provider. See config.yaml gotchas.

Worked example

Given the YAML above, the effective per-provider settings are:

Provider	max_retries	failure_threshold	cb timeout
openai	2 (global)	3 (global)	15s (global)
anthropic	5 (override)	3 (global)	15s (global)
ollama	2 (global)	10 (override)	5s (override)

anthropic and ollama inherit every field they did not explicitly override.

Circuit breaker and dashboard provider health

The dashboard’s provider status combines two independent signals:

• Model discovery — whether the provider’s model inventory could last be fetched (details below).
• Request health — a 10-minute sliding window of real request outcomes per provider and model, including the live circuit breaker state.

The circuit breaker’s state is shown on each provider card: expanding a card reveals a Breaker State chip, an open breaker turns the provider’s status pill to Circuit Open, and a half-open breaker shows Recovering. A model whose recent requests keep failing (at least 3 errors making up half or more of its windowed requests) marks the provider Degraded even while model discovery still succeeds — this catches upstreams that list models fine but fail real calls, e.g. with misreported 4xx errors that deliberately never trip the breaker.

Request-health signals only ever worsen the discovery-based status, never improve it, and the tracking is in-memory per gateway process. The breaker still recovers on its own within timeout (default 30s) once the provider is back; when metrics are enabled its state is also exported as the gomodel_circuit_breaker_state gauge (0 = closed, 1 = half-open, 2 = open).

Model discovery is re-checked:

• at startup,
• on every model registry refresh, controlled by CACHE_REFRESH_INTERVAL (seconds, default 3600 — hourly),
• on the fast recheck loop, which re-probes only providers whose latest refresh failed, controlled by PROVIDER_RECHECK_INTERVAL (cache.model.recheck_interval in config.yaml; seconds, default 60; 0 disables), and
• on demand, when a request asks for a provider-qualified model (provider/model) that is missing from the registry.

What happens while a provider is down

When a provider’s refresh fails, its previously discovered models are kept and marked stale, and the dashboard shows the provider as Degraded (“previous inventory is still available”):

• Direct requests to its models still resolve and are sent to the provider, so callers get an honest 502/503 (and manual failover rules can fire) instead of a misleading “model not found”.
• Virtual-model redirects skip the provider’s targets, so a load-balanced redirect keeps working through its healthy targets.

The fast recheck loop re-probes the provider every PROVIDER_RECHECK_INTERVAL seconds, updating “Last checked” and restoring normal routing typically within a minute of the provider coming back. A provider that was already down at startup (nothing discovered yet) shows as Unhealthy until its first successful fetch.

Failover vs. Resilience

The retry and circuit breaker layers stay on a single provider. If you also want GoModel to try a different model or provider when the primary keeps failing, configure manual failover rules. See Failover.