Fallback Chain — AI Glossary

Reference details

Related terms

• Model Gateway (LLM Gateway)
• Model Routing

How it works

1. The gateway classifies the failure first, because only some are worth failing over: a timeout, a 5xx or a rate limit should move on, while a 400 will fail identically everywhere.
2. On a retryable failure the request is reissued against the next provider in an ordered list, usually after a bounded retry against the primary.
3. The prompt must be translated to the next provider's format — system-prompt placement and tool schemas differ — which is the step that silently degrades quality.
4. A circuit breaker trips after repeated failures so traffic skips a provider that is down rather than paying its timeout on every request.
5. Which provider served each request is recorded, because without that the chain hides the outage it absorbed and nobody learns the primary is unreliable.

Example: The fallback path is the one nobody tested

A chain times out the primary at 10s, then calls a secondary that normally responds in 2s. During an outage every request pays 10 + 2 = 12 seconds, six times the usual latency, and succeeds — so availability dashboards stay green while the product feels broken. Worse, the secondary has been serving a prompt tuned for the primary for the whole incident, at a quality nobody measured. A circuit breaker removes the 10s; only evaluating the fallback path removes the quality risk.

Commonly confused with

A fallback chain reacts to failure; model routing chooses by fit before anything fails. Both may sit in the same gateway, and conflating them hides a gap: the fallback model receives a prompt tuned for the primary, and rarely gets the same validation. If the second model in the chain has never been evaluated on your traffic, the chain converts an outage into a silent quality drop.

When to use it

Reach for it when:

• Availability matters more than consistency of which model answered.
• A single provider outage would otherwise take the feature down.
• Rate limits are hit in bursts and a second provider can absorb the overflow.

Reach for something else when:

• Output must come from one specific model for reproducibility or compliance reasons.
• The fallback has never been evaluated on your prompts, where it trades an outage for silent quality loss.
• Failures are client errors, which will fail the same way on every provider.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Fallback Chain — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path