Text Generation Inference (TGI) — AI Glossary

Reference details

Related terms

• vLLM
• Inference

How it works

1. A Rust-based serving layer handles HTTP, request admission and batching, while the model's forward passes run in Python — a split that keeps the request-handling path out of the interpreter.
2. Continuous batching admits and retires sequences between decoding steps, so a finished short request frees its slot immediately rather than waiting for the batch's longest member.
3. Attention kernels that avoid materialising the full score matrix reduce both memory and bandwidth during prefill, which is what makes long prompts affordable.
4. Weights load from a zero-copy tensor format that is memory-mapped rather than deserialised, so startup is bounded by storage bandwidth instead of by parsing.
5. Tensor parallelism shards a model across devices when it does not fit on one, configured at launch rather than in the model definition.
6. Hub integration means a model identifier is enough to launch a server, which is the practical difference from engines requiring a local conversion step.

Example: Why the weight format shows up as startup time

A 7B model in 16-bit weights is 7e9 x 2 = 1.4e10 bytes, 14GB. Memory-mapped from an NVMe device at 3 GB/s, loading is bounded by transfer: 14 / 3 = 4.7 seconds. A format requiring deserialisation adds a parse and at least one full copy in host memory before the transfer to the device, so the same 14GB can take several times that — and the gap is not throughput but cold-start latency, which is the number that matters for autoscaling. At 14GB from slower network storage at 500 MB/s it is 14,000 / 500 = 28 seconds, which is why the storage path rather than the format is often the real constraint. The practical consequence is that cold start has three terms worth measuring separately — storage read, device transfer and any parse — and only the third is a property of the format. A deployment blaming its 40-second starts on the engine when the weights sit on network storage is optimising the wrong term.

Commonly confused with

This is an inference engine, not a serving platform, and the distinction is the same one that applies to any engine: it implements the token loop, batching and attention kernels for transformer models, and it does not provide multi-model routing, pipeline composition or cluster orchestration. Those belong to a layer above it, which is why production deployments commonly run an engine behind a separate serving layer. Comparisons between engines are specific to a model, a quantisation and a hardware generation, so a throughput claim without those three stated is not a comparison.

When to use it

Reach for it when:

• Serving transformer models from the Hub where a model identifier should be enough to launch.
• Workloads needing continuous batching and memory-efficient attention, which are the engine's core contributions.
• Multi-device serving of a model too large for one accelerator, configured at launch.
• Where fast cold start matters and the weight format's zero-copy loading is part of the budget.

Reach for something else when:

• As a serving platform, since multi-model routing and pipeline composition sit above an engine.
• For non-transformer architectures outside its supported set.
• Chosen on a throughput claim that does not state model, quantisation and hardware, which makes the figure unusable.
• Where cold start is dominated by storage bandwidth rather than by the format, which is a different term to optimise.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Text Generation Inference (TGI) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path