LLM Router — AI Glossary

Reference details

Related terms

• Model Routing
• Model Gateway (LLM Gateway)

How it works

1. A classifier at the entry point reads the request and predicts which model can handle it, so the routing decision is made before any expensive inference starts.
2. The classifier must be fast and cheap relative to the models it routes between, or the routing overhead erodes the saving it exists to produce.
3. Routing signals vary: a small trained classifier, embedding similarity to labelled historical requests, or a cascade that tries the cheap model first and escalates on a confidence or validation check.
4. The cascade variant trades latency for accuracy — an escalated request pays both models — while a classifier trades accuracy for latency by deciding up front and sometimes deciding wrong.
5. A misroute downward costs more than not routing at all, because the cheap attempt is wasted and the expensive model runs anyway, so the escalation rate is the number that determines whether routing pays.
6. The decision needs continuous evaluation, since request distributions drift and a classifier trained on last quarter's traffic routes this quarter's badly.

Example: Why the misroute rate decides whether routing pays

Suppose 70% of requests can be served by a model costing a fifteenth of the frontier one. With perfect routing the blended cost is 0.7 x (1 / 15) + 0.3 x 1 = 0.0467 + 0.3 = 0.3467, a 1 - 0.3467 ≈ 65% saving. Now add misroutes. If 5% of the requests sent to the cheap model fail and must be escalated, those requests cost both models: 1 / 15 + 1 = 1.0667 each. The blend becomes (0.7 x 0.95) x (1 / 15) + (0.7 x 0.05) x 1.0667 + 0.3 x 1 = 0.665 x 0.0667 + 0.035 x 1.0667 + 0.3 = 0.0443 + 0.0373 + 0.3 = 0.3817, so the saving falls from 65% to 1 - 0.3817 ≈ 62%. A 5% misroute rate costs 3 percentage points of the saving — tolerable. At a 30% misroute rate it is (0.49 x 0.0667) + (0.21 x 1.0667) + 0.3 = 0.0327 + 0.224 + 0.3 = 0.5567, a 44% saving: still positive, and now half the benefit has gone to wasted cheap attempts. The arithmetic stays favourable across a wide range, which is the real finding — routing is robust to moderate misrouting, and what kills it is a classifier whose own cost or latency is comparable to the models it routes between.

Commonly confused with

A router selects between models; a load balancer selects between replicas of one model. The distinction matters because the failure modes are unrelated: a load balancer sending a request to a slow replica costs latency, while a router sending a request to an incapable model costs a wrong answer or an escalation. The second confusion is with a cascade: a router decides up front from the request alone, while a cascade decides after seeing a cheap model's attempt. A cascade is more accurate and pays double on escalation; a router is cheaper and sometimes wrong.

When to use it

Reach for it when:

• Traffic with a wide difficulty spread, where a large fraction of requests genuinely do not need the frontier model.
• High volume, where the saving is large enough to justify building and maintaining the classifier.
• With a classifier whose cost and latency are a small fraction of the models it routes between.
• With escalation measured continuously, since the misroute rate is what erodes the saving and drifts with traffic.

Reach for something else when:

• Uniformly hard traffic, where almost everything needs the capable model and routing adds a step for nothing.
• Where a wrong answer is costly and the misroute is not detectable, since routing down then fails silently.
• Low volume, where the classifier's development and evaluation cost exceeds the token saving.
• With a classifier as expensive as the gap it is arbitrating, which is the configuration that reliably loses money.

Referenced by

This term is used by the following ModelRefs references:

• byo model gateway stack

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to LLM Router — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path