PortKey — AI Glossary

Reference details

Related terms

• LiteLLM
• AI Gateway
• Model Routing

How it works

1. A gateway normalises many providers behind one request shape, so switching provider is configuration rather than code.
2. Routing rules select a target per request by cost, latency or declared capability, before any failure occurs.
3. Fallback chains react after a failure, reissuing to the next target on retryable errors.
4. Semantic caching can short-circuit a request whose embedding is close enough to a cached one — which saves a call and introduces a correctness risk a prefix cache does not.
5. Because every request passes through it, the same component holds the observability data, which is why it occupies two product categories at once.

Example: Two controls with very different failure modes

Fallback routing turns a provider outage into a slower successful request: availability improves, and the worst case is degraded quality on a path nobody evaluated. Semantic caching at a 0.95 threshold turns a near-miss into a wrong answer served quickly: cost improves, and the worst case is a confidently incorrect response that monitoring records as a success. Both live in the same gateway and are configured in the same place, so it is worth being explicit that one risks latency and the other risks correctness.

Commonly confused with

An AI gateway with observability attached, which puts it in two categories at once — compare it with LiteLLM on the gateway axis and with Langfuse or Helicone on the observability axis. The gateway function is the load-bearing one: unified access, fallback routing and caching are things a request path does, whereas observability is something that watches it.

When to use it

Reach for it when:

• Many providers must be reachable through one interface with central keys and budgets.
• Routing by cost or latency, with failover on provider incidents.
• Gateway and observability are wanted from one component rather than two.

Reach for something else when:

• A single provider is used, where its own client is simpler and more complete.
• Semantic caching would be enabled without measuring false-hit rate on real traffic.
• Deep per-stage tracing of non-model steps is the primary need.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to PortKey — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path