Model Routing — AI Glossary

Reference details

Related terms

• Model Gateway (LLM Gateway)
• Fallback Chain
• Inference Cost

How it works

1. Features are extracted from the request before any model runs — length, detected task type, the presence of code or structured data, sometimes an embedding of the prompt.
2. A decision function maps those features to a model, and it is usually a small classifier trained on labelled examples of which requests the cheap model handled acceptably.
3. That label is the hard part: it requires knowing, per request, whether the cheap answer was good enough, which means the router's training data comes from evaluation rather than from traffic.
4. A cascade is the alternative mechanism and has a different cost shape: the cheap model always answers, a verifier scores the result, and only failures are escalated.
5. So a router pays one call and risks misclassification, while a cascade pays the cheap call on every request and never misroutes — the choice depends on the cheap model's hit rate.
6. Both need outcome measurement to stay honest, because a misroute returns a worse answer rather than an error, and nothing in the request path notices a quietly degraded response.

Example: Where the saving comes from

Suppose 80% of traffic is simple classification and 20% needs frontier reasoning. Sending everything to the expensive model costs 1.0 units. Routing the 80% to a model at one tenth the price gives 0.8 × 0.1 + 0.2 × 1.0 = 0.28 — roughly a 70% reduction, provided the router classifies correctly. The router's accuracy is the whole business case.

all-frontier : 1.00 x 1.0                = 1.00
routed       : 0.8 x 0.1  +  0.2 x 1.0   = 0.28

Commonly confused with

Routing is not the same as fallback. A router chooses a model up front based on the request; a fallback chain reacts after a failure. Systems often need both, and conflating them leads to fallbacks that quietly become the primary path.

When to use it

Reach for it when:

• Traffic is genuinely mixed in difficulty and volume is high enough to matter
• You can evaluate whether the cheap model was adequate for the routed requests
• Cost is a real constraint rather than a theoretical one

Reach for something else when:

• Volume is low — the routing layer costs more to run and maintain than it saves
• Requests are uniform; there is nothing to route
• You cannot detect misroutes, which silently degrade quality where it is least visible

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Model Routing — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path