Chunked Prefill — AI Glossary

Reference details

Related terms

• Time to First Token (TTFT)
• KV Cache
• Batch Size

How it works

1. Prefill and decode have opposite profiles: prefill processes every prompt token at once and saturates arithmetic, while decode emits one token at a time and is bound by memory bandwidth.
2. Scheduled naively, a long prefill occupies the device until it finishes, and every request already decoding stalls behind it.
3. Chunked prefill splits the prompt into fixed-size pieces and schedules one piece per iteration instead of the whole prompt.
4. Decode steps for other requests are interleaved between chunks, so an in-flight generation continues while a large prompt is still being ingested.
5. The arriving request pays slightly more total prefill time for the chunking overhead; everyone already generating stops paying for it entirely, which is why the win shows in tail latency rather than in averages.

Example: Where the tail latency actually goes

A 32,000-token prompt takes 2 seconds to prefill. Unchunked, a request mid-generation waits the full 2 seconds for its next token. Split into 8 chunks of 4,000 tokens at about 250ms each, decode steps interleave between chunks and the worst wait becomes roughly one chunk — about 250ms, an eightfold improvement in that stall. Mean throughput barely moves, and p99 time-between-tokens does. This is why the optimisation is invisible in average benchmarks and obvious to users on a busy server.

Commonly confused with

Grouped with continuous batching, which it complements rather than duplicates. Continuous batching decides which requests occupy a batch and admits new ones as others finish; chunked prefill decides how one request's prefill is divided so it cannot monopolise an iteration. Without chunking, a long prompt still blocks the batch it was admitted to. It is also not prefix caching: that avoids recomputing a shared prefix altogether, while chunking recomputes nothing and only changes the schedule. The three stack, and only one of them helps the stall a long prompt inflicts on other requests.

When to use it

Reach for it when:

• Mixed traffic where long prompts and interactive generation share a server.
• Tail time-between-tokens matters, as in streaming interfaces.
• A few large requests are degrading latency for many small ones.

Reach for something else when:

• Uniformly short prompts, where there is no monopolising prefill to break up.
• Offline batch work optimising total throughput, where chunking overhead is a small net loss.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Chunked Prefill — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path