AI Gateway — AI Glossary

Reference details

Related terms

• Model Gateway (LLM Gateway)
• Model Routing
• Rate Limiting

How it works

1. The gateway terminates the client's request itself, so applications hold a credential for the gateway and never for any provider — which is what makes key rotation a server-side operation.
2. A virtual model name in the request is resolved against configuration to a concrete provider and model, so the string an application sends is an indirection rather than a destination.
3. The request is then rewritten into that provider's dialect — message shape, system-prompt placement, tool format — and the response is normalised back into one envelope the client expects.
4. Streaming has to be proxied frame by frame rather than buffered, because buffering would convert a streaming response into a single late one and remove the reason the client streamed.
5. Policy is applied in that same path: per-team quotas, spend caps, rate limits, caching and audit logging all sit at the point every request already passes through.
6. Failover works because the retry happens inside the gateway against a different provider, so an application that received no error also needed no deploy — and that same centrality is why the gateway is a single point of failure to design around.

Example: What breaks without one

Three services each hold their own OpenAI key. Someone needs per-team spend, so every service gets patched. Then a provider outage needs a failover, so every service is patched again. A gateway makes those one change in one place, because the application only ever talks to the gateway.

Commonly confused with

A gateway is not a router. Routing — choosing which model serves a request — is one feature a gateway may offer. The gateway itself is the proxy layer: one API surface, with auth, logging, rate limits, caching and failover behind it.

When to use it

Reach for it when:

• More than one service or team calls model providers
• You need per-team cost attribution, quotas or a kill switch
• Provider failover has to work without redeploying applications

Reach for something else when:

• One service, one provider — it is infrastructure with nothing to manage
• Latency is critical and you cannot absorb an extra network hop
• You have not planned for it being a single point of failure

Referenced by

This term is used by the following ModelRefs references:

• byo model gateway stack

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to AI Gateway — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path