Tokens Per Second (TPS) — AI Glossary

Reference details

Related terms

• Time to First Token (TTFT)
• Throughput
• GPU Memory (VRAM)

How it works

1. The figure is derived rather than observed: output token count divided by the interval from the first token to the last, which deliberately excludes the wait before generation began.
2. Where that interval starts matters, because including the time to first token blends a queueing and prefill measurement into a decode one and produces a number describing neither.
3. The first few tokens of a request are often slower than the steady state, so a short completion measures warm-up and a long one measures the plateau — making the figure dependent on output length.
4. A mean conceals stalls: a scheduler preempting a sequence produces one long gap between consecutive tokens, which barely moves the average and is exactly what a reader perceives.
5. Reporting the distribution of inter-token intervals rather than their reciprocal mean is what makes that visible, since the tail is where a stutter lives.
6. Per-request and server-aggregate figures also diverge under batching — aggregate rises while each individual stream slows — so a number without its batch size, hardware and precision is not comparable to another.

Example: Two numbers, and only one of them users feel

At 40 tokens per second the gap between tokens is 1 / 40 = 25 milliseconds, and a 600-token answer takes 600 / 40 = 15 seconds to finish. Comfortable reading runs around 5 tokens per second, so anything above roughly 20 TPS arrives faster than a person can read it — pushing 40 to 80 buys nothing a user can perceive on a streamed chat response. Where it does matter is when nobody is reading: an agent loop consuming its own output, or a batch job, is throughput-bound, and there the number is the cost.

Commonly confused with

Tokens per second is decode throughput after generation starts; time to first token is the wait before it. They are governed by different things — TTFT by prompt length and queueing, TPS by memory bandwidth and batch size — and a system can be excellent at one and poor at the other. Per-request TPS also differs from server aggregate TPS, which rises with batching while each individual request slows.

When to use it

Reach for it when:

• Sizing hardware for a target concurrency, where aggregate throughput is the constraint
• Agentic and batch workloads, where output feeds a machine rather than a reader
• Comparing serving engines on identical hardware, model and batch settings

Reach for something else when:

• Optimising perceived speed of chat — past roughly 20 TPS, TTFT is the lever
• Comparing published figures without the hardware, quantization and batch size
• As a quality proxy: a faster model is not a better one

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Tokens Per Second (TPS) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path