Inference Endpoint — AI Glossary

Reference details

Related terms

• vLLM
• Model Gateway (LLM Gateway)
• Inference

How it works

1. A model artefact is bound to compute — weights loaded into accelerator memory inside a container running an inference engine — and that binding is what an endpoint actually is.
2. A stable network address fronts a pool of replicas. The address outlives any individual replica, which is what makes rolling replacement and canary traffic possible without clients knowing.
3. Requests are admitted against quotas and queued, then dispatched to a replica. The queue is where latency accumulates under load, and queue depth is the usual autoscaling signal.
4. The engine batches continuously: arriving requests join an in-flight batch between decoding steps rather than waiting for a batch window to close, which is what keeps utilisation high without sacrificing time-to-first-token.
5. Scaling has a floor set by cold start — loading tens of gigabytes of weights takes tens of seconds — so a replica added in response to load arrives after the load that called for it.
6. The endpoint is the versioning boundary. What model and what engine configuration a given address serves is a deployment decision, which is why pinning a model identifier matters as much as pinning a library version.

Example: Why queue depth, not latency, is the autoscaling signal

A replica serves requests taking 500ms of compute each, so one replica handles 1 / 0.5 = 2 requests per second. At 10 requests per second you need 10 / 2 = 5 replicas to keep up exactly. Run 4 instead and the arrival rate exceeds service capacity by 10 - 8 = 2 requests per second, so the queue grows by 2 every second indefinitely: after 30 seconds it holds 60 requests and the newest arrival waits 60 / 8 = 7.5 seconds behind it. Latency is the symptom and it lags — at second one the queue holds 2 and latency looks almost normal, so a latency-triggered autoscaler starts reacting roughly when the queue is already deep, and then waits out a 40-second cold start before capacity arrives, by which point the queue holds about 2 x 70 = 140 requests. Scaling on queue depth sees the 2-per-second accumulation immediately. This is also the argument for a warm floor: no signal can beat the cold start, so capacity for the burst has to exist before the burst.

Commonly confused with

An endpoint is not a model, though configuration files routinely conflate them by storing only a URL. The same model served by two endpoints can behave differently — different engine versions, different quantisation, different maximum context, different sampling defaults — and the same endpoint can be repointed at a different model without the address changing. For reproducibility you need both: which endpoint, and which model identifier it served at the time. Teams that record only one discover the gap when output changes and the configuration they version shows nothing different.

When to use it

Reach for it when:

• Serving a model to any caller other than the process that loaded it, which is the whole purpose of the abstraction.
• Multiple consumers of one model, where the endpoint centralises quota, logging and version control.
• Rolling or canary deployment, which requires a stable address decoupled from the replicas behind it.
• Managed endpoints specifically, when inference operations are not where your team's effort should go.

Reach for something else when:

• Batch jobs with no latency requirement, where a direct offline inference run avoids the serving layer entirely.
• On-device or embedded inference, where the network hop is the thing you are trying to remove.
• A single process using a single model, where in-process loading is simpler and faster.
• As the only thing recorded for reproducibility, since the address does not say which model or engine configuration answered.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Inference Endpoint — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path