Voice Agent — AI Glossary

Reference details

Related terms

• AI Agent
• Tool-Using Agent
• Conversational AI

How it works

1. Audio is captured continuously and segmented by a turn detector, which decides when the user has stopped speaking. That decision is the hardest part of the system and the main source of felt awkwardness.
2. Speech recognition transcribes incrementally rather than waiting for the turn to end, so partial text is available to start downstream work before the user finishes.
3. The transcript enters a model loop that may also call tools, and its output is streamed token by token rather than returned complete.
4. Synthesis begins on the first few tokens, so audio starts playing while the model is still generating. This overlap is not an optimisation — the budget does not close without it.
5. Interruption handling is required throughout: if the user speaks while the agent is talking, playback must stop and the partial utterance must be recorded as having been said, or the conversation state diverges from what the user heard.
6. Newer designs process audio end to end in one model, which removes two conversion boundaries and the latency they add, at the cost of the per-stage control the pipeline gives you.

Example: Why the latency budget does not close without overlap

Target a 500ms gap between the user finishing and the agent starting to speak — roughly the threshold above which a conversation feels like a radio link. Stage the pipeline sequentially: 100ms to finalise the transcript, 300ms for the model's first token, 150ms for the synthesiser to produce its first audio. That is 100 + 300 + 150 = 550ms, already 50ms over budget before networking, and this is with generous per-stage figures. Overlap them instead. Recognition runs incrementally so most of its work is done before the turn ends; synthesis starts on the model's first few tokens rather than its last, so its 150ms runs concurrently with generation rather than after it. The critical path becomes roughly the turn detection plus the model's time-to-first-token plus the synthesiser's startup on a short prefix — on the order of 100 + 300 = 400ms, inside the budget with room for a network hop. The design consequence is that every stage must stream. A single non-streaming component anywhere in the chain adds its full latency to the critical path, and one 300ms non-streaming synthesiser is enough to push a working system out of budget.

Commonly confused with

A voice agent is not a chatbot with speech attached, and the difference is that latency is a correctness property rather than a comfort one. In text, a two-second response is acceptable; in speech, it reads as the system being broken and users talk over it. Turn-taking has no text equivalent at all — there is no ambiguity about when a typed message ends, while deciding whether a pause is a breath or a finished sentence is a judgement the system makes several times a minute and visibly gets wrong. So the engineering centre of gravity is the real-time path, not the model's answer quality.

When to use it

Reach for it when:

• Hands-free or eyes-free contexts — driving, field work, accessibility — where speech is the only practical interface.
• Phone-channel automation, where the alternative is a touch-tone menu and the comparison is favourable.
• Short, bounded transactions: status checks, bookings, confirmations, where a turn or two completes the task.
• With an end-to-end audio model when the pipeline's latency budget will not close and per-stage control is dispensable.

Reach for something else when:

• Information-dense output, where speech is a poor medium — a list of options or a table should be read, not heard.
• Noisy environments or heavy accents outside the recogniser's evaluated range, where transcription errors propagate silently into the model's input.
• Emotionally charged interactions, where turn-taking errors and synthetic prosody make an already difficult conversation worse.
• Anywhere a non-streaming component sits in the chain, since its full latency lands on the critical path.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Voice Agent — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path