Triton Inference Server — AI Glossary

Reference details

Related terms

• Ray Serve
• BentoML
• vLLM

How it works

1. A model repository on disk declares each model, its backend and its configuration, and the server loads from that directory — so deployment is a filesystem layout rather than application code.
2. Multiple backends run side by side, so models from different frameworks are served by one process with one API surface, which is what makes it a serving platform rather than a framework's own server.
3. Dynamic batching holds arriving requests for a configured window, or until a maximum batch size is reached, then submits them as one batch. The window length is the explicit latency-for-throughput trade.
4. Several instances of a model can execute concurrently on one device, which fills the gaps when a single instance leaves the accelerator idle between kernels.
5. Ensembles chain models into a pipeline declared in configuration — tokenise, embed, rerank — so intermediate tensors stay on the device instead of crossing a network boundary between services.
6. A companion analyser sweeps batch sizes and instance counts against your own traffic to find the configuration, because the right values depend on the model, the hardware and the arrival pattern rather than on a default.

Example: Why the batching window hurts at low load

Set a dynamic batching window of 50ms with a maximum batch of 8. At 100 requests per second, 50ms of waiting accumulates 100 x 0.05 = 5 requests, so batches of 5 form and the window is doing its job: one model execution serves 5 requests instead of 5 executions serving one each. At 20 requests per second the same window accumulates 20 x 0.05 = 1 request — the batch is a single item, the execution is exactly what it would have been without batching, and every request has paid up to 50ms of pure waiting for nothing. The window is a tax at low load and a saving at high load, and the crossover is where the arrival rate fills a useful batch inside it: to reach a batch of 8 within 50ms you need 8 / 0.05 = 160 requests per second. Below that the window is partially wasted, which is why the setting cannot be chosen from the hardware alone and why a configuration tuned during a load test at peak will add latency during the quiet hours it was never measured against.

Commonly confused with

Dynamic batching and continuous batching are different mechanisms and the distinction matters for text generation. Dynamic batching groups whole requests before execution, so a batch starts together and the batch is occupied until its longest member finishes — one 2,000-token generation holds the slots of seven 50-token ones. Continuous batching admits and retires sequences between decoding steps, so a finished sequence frees its slot immediately. For generative models the second is what keeps utilisation high, which is why a general-purpose server's dynamic batching is usually paired with an engine that does continuous batching inside it rather than relied on alone.

When to use it

Reach for it when:

• Serving models from several frameworks behind one endpoint, which is the problem this is built for.
• Multi-stage pipelines where keeping intermediate tensors on the device avoids network hops between services.
• Non-generative workloads — classification, embedding, reranking, vision — where whole-request dynamic batching fits the uniform shape of the work.
• Infrastructure that already standardises on it, where adding a model is a repository entry rather than a new service.

Reach for something else when:

• Generative text serving on its own, where an engine doing continuous batching and paged attention is what determines throughput.
• A single model in a single framework, where that framework's own server is less to configure and operate.
• With a batching window tuned only at peak load, which taxes every request during quiet periods.
• Small deployments, where the model repository and configuration surface is more machinery than one model needs.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Triton Inference Server — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path