LiteLLM — AI Glossary

Reference details

Related terms

• PortKey
• AI Gateway
• OpenAI-Compatible API

How it works

1. One request shape is translated to each provider's own format, so application code targets a single interface across vendors.
2. The translation is the hard part and where fidelity is lost: tool-call schemas, system-prompt placement and streaming event shapes all differ.
3. Routing rules choose a deployment per request — by cost, latency, or declared capability — before any failure occurs.
4. Fallback chains react after a failure, reissuing to the next deployment on retryable errors.
5. A proxy mode centralises keys, budgets and logging, which is usually the real reason to adopt it rather than the abstraction itself.

Example: The abstraction is leakiest exactly where it matters

A request using strict JSON-schema tool calling works against the primary provider. The fallback provider supports function calling with a looser schema, so the same request succeeds and returns arguments that do not validate. Availability was preserved and correctness was not — and the failure appears downstream as a parse error, not as a provider incident. This is why a fallback path needs evaluating on your own prompts rather than trusting that a unified interface made the providers equivalent.

Commonly confused with

Two things ship under this name and the choice between them matters: a Python library you import to normalise provider calls in-process, and a proxy server you run so any client can reach many providers through one endpoint. The library adds no hop; the proxy adds one and gives you central keys, budgets and logging. Portkey and a hosted gateway occupy the same role as the proxy form.

When to use it

Reach for it when:

• Several providers must be reachable through one interface, with central keys and budgets.
• Routing by cost or latency, and failing over on provider incidents.
• A gateway is wanted for spend control and logging across teams.

Reach for something else when:

• A single provider is used, where the provider's own typed client is simpler and more complete.
• Provider-specific features matter, which an abstraction tends to flatten.
• Fallback targets have not been evaluated on your prompts, which trades an outage for silent quality loss.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to LiteLLM — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path