Inference Provider — AI Glossary

Reference details

Related terms

• Serverless Inference
• OpenAI-Compatible API
• Model Routing

How it works

1. The provider runs the serving stack — accelerators, engine, batching, autoscaling — and exposes models behind an API, so the caller's integration is a request format and a key rather than infrastructure.
2. Most expose an OpenAI-compatible surface, which makes switching a base-URL change at the code level while leaving the harder work of re-tuning prompts and re-validating behaviour untouched.
3. Pricing is per token, usually split between input and output at different rates, because the two phases consume different resources: prefill is compute-bound and decode is memory-bandwidth-bound.
4. Throughput and latency vary substantially between providers serving the same open-weights model, since batch settings, quantisation and hardware are the provider's choices and are rarely disclosed.
5. That is the key asymmetry for open models: the weights are identical, so the provider is selling a serving configuration, and quantisation applied for throughput is a quality decision made on your behalf.
6. Quotas rather than capacity bound you, so scaling is a commercial negotiation and a rate-limit design problem rather than a hardware purchase.

Example: Why the cheaper provider can be the more expensive choice

Two providers serve the same open-weights model. One generates at 50 tokens per second and bills 2 units per million tokens; the other at 200 tokens per second and bills 4. For a 500-token response the latency is 500 / 50 = 10 seconds against 500 / 200 = 2.5 seconds, a 4x difference, while the token cost is 500 x 2 / 1,000,000 = 0.001 units against 500 x 4 / 1,000,000 = 0.002 — 2x. So the faster provider costs twice as much per request and is four times quicker, which is a straightforward trade until you price the 7.5 seconds. In an interactive product that difference is the whole user experience and the 0.001 units is noise. In a batch pipeline processing a million documents overnight, the latency is irrelevant and the gap is 1,000,000 x 0.001 = 1,000 units against 2,000 — a real 1,000-unit difference. The same two providers are the correct answer to two different questions, which is why provider selection belongs per workload rather than per organisation, and why a single default is usually wrong for half the traffic.

Commonly confused with

An inference provider is not the model's developer, and for open-weights models this distinction carries real consequences. When a third party serves someone else's weights, the weights are the same but the quantisation, context limit, sampling defaults and system-prompt handling are the provider's configuration — rarely documented and freely changeable. So two providers' outputs for the same model name can differ measurably, and a benchmark result obtained from one does not transfer to another. For a developer's own hosted model there is no such gap, because the developer controls both.

When to use it

Reach for it when:

• Variable or modest volume, where per-token billing beats paying for idle accelerators.
• Fast evaluation across models, where several are reachable through one request format.
• When inference operations are not where the team's effort should go, which is most teams.
• Chosen per workload, since latency-sensitive and batch traffic have different right answers.

Reach for something else when:

• Data-residency or processing constraints that no contract satisfies, where the data must stay on your infrastructure.
• Sustained high utilisation, where self-hosting's per-token cost is a fraction of an API's.
• Without checking the serving configuration for open-weights models, since undisclosed quantisation is a quality decision taken for you.
• Assuming benchmark results transfer between providers of the same model name — the configuration differs even when the weights do not.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Inference Provider — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path