Streaming (Token Streaming) — AI Glossary

Reference details

Related terms

• Latency
• Inference Cost

How it works

1. The response is held open as a server-sent event stream, with each chunk arriving as a line carrying a JSON delta rather than a complete message.
2. Deltas are fragments to be concatenated, so the client accumulates rather than replaces, and a client written to overwrite shows only the final fragment.
3. Chunk boundaries fall on token boundaries rather than character ones, so a multi-byte character can be split across two chunks and must be buffered until complete or it renders as a replacement glyph.
4. The stream is closed by a sentinel line rather than by the connection ending, which lets a client distinguish a finished response from a dropped one.
5. Token usage and the finish reason arrive only in the final chunk, so a stream cannot report its own cost or whether it was truncated until it completes.
6. That ordering is what makes streaming incompatible with a check on the whole output: by the time the finish reason is known, everything it describes has already been delivered.

Example: The wait you remove is most of the wait

A 400-token answer generated at 50 tokens per second takes 8 seconds to finish. Without streaming the user sees nothing for 8 seconds. With streaming they see the first words in a few hundred milliseconds and read while the rest arrives — and since comfortable reading is well under 50 tokens per second, they never catch up to the stream. Total time is identical. Time to first token is the number that changed, and it is the one users experience as speed.

Commonly confused with

Streaming changes delivery, not generation. It does not reduce cost, total latency, or tokens produced. It also does not make a slow model fast — if time to first token is already seconds, streaming reveals the delay rather than hiding it.

When to use it

Reach for it when:

• Any interactive surface where a human reads the output as prose
• Long responses, where the perceived gain grows with output length
• Agent traces you want shown progressively, so users can interrupt a wrong direction

Reach for something else when:

• Output that must be validated as a whole before display — malformed JSON cannot be unsent
• Guardrail or moderation gates on the completion, which need the full text to decide
• Batch and background jobs, where nobody is waiting and streaming only complicates the client

Referenced by

This term is used by the following ModelRefs references:

• streaming inference stack

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Streaming (Token Streaming) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path