Time to First Token (TTFT) — AI Glossary

Reference details

Related terms

• Tokens Per Second (TPS)
• P99 Latency
• Chunked Prefill

How it works

1. What distinguishes this measurement from total latency is that it ends before generation begins, so nothing about the answer's length can enter it.
2. Prefill processes every prompt position in parallel, so it is compute-bound and its cost rises faster than linearly with length because the attention term is quadratic.
3. A long prefill monopolises the accelerator, which is why one large request lengthens the first-token wait of every other request behind it rather than only its own.
4. Chunked prefill addresses that by splitting a long prompt into pieces and interleaving them with other requests' decode steps, so the long request slows slightly and everyone else's worst case improves.
5. Disaggregated serving goes further and runs prefill and decode on separate hardware pools, so a compute-bound phase and a bandwidth-bound phase stop competing for the same device.
6. Speculative decoding does nothing for this metric at all, since the drafting only begins once prefill has finished — which is why a stack can be fast at generation and still feel unresponsive.

Example: It scales with the prompt, not the answer

Before any token comes back, the model must process the entire prompt — the prefill pass. A 500-token prompt and a 50,000-token prompt differ by two orders of magnitude in that work, so the long one waits proportionally longer for its first token even if both answers are three sentences. This is why stuffing a document into context is felt as sluggishness rather than cost, why prompt caching improves responsiveness so noticeably (a cached prefix skips most of the prefill), and why trimming a system prompt often does more for perceived speed than switching to a faster model.

Commonly confused with

Time to first token is the wait before output starts; tokens per second is how fast it flows once it has. They are governed by different things — prefill and queueing for the first, memory bandwidth and batch size for the second — and are optimised by different changes. Streaming exposes TTFT directly to the user, so it becomes the number that reads as “speed”.

When to use it

Reach for it when:

• Interactive and voice interfaces, where the pause before output is what users judge
• Deciding whether to shorten prompts, cache prefixes, or move to retrieval
• Capacity planning under load, since queueing lands here first

Reach for something else when:

• Batch and background work, where nobody is waiting on the first token
• Comparing providers without matching prompt length and cache state
• As a total-latency proxy — a long answer's completion time is dominated by decode

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Time to First Token (TTFT) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path