Chat Completion — AI Glossary

Reference details

Related terms

• OpenAI-Compatible API
• System Message
• Streaming (Token Streaming)

How it works

1. The messages array is serialised in order into one continuous token sequence using the delimiter tokens the model was instruction-tuned on, so roles exist as learned markers rather than as structure the model sees separately.
2. That serialisation is why the endpoint can be stateless at all: the conversation is reconstructed from scratch on every call and nothing persists between them.
3. The server validates the array's shape but not its plausibility, so a history with the wrong alternation, a duplicated assistant turn, or a fabricated prior response is accepted and answered.
4. Every sampling parameter acts on that single serialised sequence, which is why stop sequences can match text the model was given rather than only text it produced.
5. Requesting several completions branches after the shared prefill, so the input is charged once while output is charged per branch.
6. Because the whole array becomes one prefix, the parts that stay identical across calls are cacheable and the parts that change are not — which makes message ordering a cost decision as well as a semantic one.

Example: The endpoint has no memory, so you pay for it

Each call is stateless: the conversation exists only because your client resends it. If each turn averages 200 tokens, the tenth request carries nine prior turns — about 1,800 tokens of history — plus the new message, before the model reads a word of the actual question. Across a ten-turn conversation the resent history sums to roughly 9,000 tokens you are billed for repeatedly. That arithmetic is the entire motivation for prompt caching, and for summarising or truncating old turns rather than carrying them forever.

Commonly confused with

Chat completions replaced the older text completions endpoint, which took a single prompt string with no roles. Newer stateful endpoints reverse the trade by keeping conversation state server-side. “OpenAI-compatible” providers implement the chat completions shape but not always its full parameter set — tools, response formats and streaming semantics are the usual gaps.

When to use it

Reach for it when:

• Essentially all current LLM application code — it is the portable interface across providers
• Multi-turn interfaces, where roles carry the instruction and history distinction
• Tool calling and structured output, which are defined against this shape

Reach for something else when:

• Assuming the server remembers anything between calls
• Carrying unbounded history — cost and lost-in-the-middle effects both grow with it
• Porting between providers without testing: compatibility covers the shape, not the behaviour

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Chat Completion — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path