Message Threading — AI Glossary

Reference details

Related terms

• Multi-Turn Conversation
• System Message
• User Message

How it works

1. The API contract is a typed sequence: an optional system message first, then alternating user and assistant turns, with tool-result messages attached to the calls they answer.
2. Validity is structural. Two consecutive messages of the same role, a tool result with no preceding tool call, or an assistant turn with nothing before it are rejected or silently degrade quality depending on the provider.
3. The whole thread is re-sent on every request, because the model is stateless between calls. The nth turn therefore costs the sum of all previous turns in input tokens, which is where conversational cost actually lives.
4. Growth is managed by a windowing policy: keep the last k turns, summarise older ones into a synthetic message, or retrieve relevant past turns. Each choice decides what the model can no longer remember.
5. The system message is pinned outside the window, since dropping it to save tokens removes the instructions the rest of the thread assumes.
6. Trimming must preserve pairing. Cutting a thread between a tool call and its result produces an invalid sequence, which is the most common way a naive truncation breaks a working agent.

Example: Why conversational cost grows quadratically

Each turn adds about 200 tokens — 100 from the user, 100 from the assistant. On turn 1 the request carries roughly 100 input tokens; on turn 10 it carries the preceding 9 turns plus the new message, about 9 x 200 + 100 = 1,900. Total input across 20 turns is the sum of an arithmetic series: roughly 200 x (1 + 2 + ... + 19) + 20 x 100 = 200 x 190 + 2,000 = 40,000 tokens, against the 20 x 200 = 4,000 tokens the conversation actually contains. That is 10x, and it grows with the square of turn count — at 40 turns it is 200 x 780 + 4,000 = 160,000 against 8,000, a factor of 20. Capping the window at the last 10 turns makes per-request input constant at about 1,900 rather than unbounded, and the 20-turn total becomes roughly 20 x 1,900 = 38,000 — barely better here, but at 100 turns it is 190,000 against 1,000,000. This is why windowing is a cost-control mechanism, not a context nicety.

Commonly confused with

A thread is not memory, and the distinction decides your architecture. The thread is the literal message array re-sent on each request, bounded by the context window and costing tokens every turn. Memory is an external store — a summary, a vector index, a key-value record — consulted selectively and injected when relevant. Teams that treat the thread as memory hit the context limit and then start truncating, which silently discards the earliest and often most important turns, including the ones where the user stated their actual goal.

When to use it

Reach for it when:

• Any multi-turn chat interface, where the role sequence is the API contract rather than a convention.
• Tool-calling agents, where call and result messages must stay paired for the model to interpret them.
• Few-shot prompting via example turns, which is often more effective than the same examples inside one user message.
• Debugging conversational quality, since an invalid or reordered thread is a common cause of degradation that looks like a model problem.

Reach for something else when:

• As long-term memory. Unbounded threads hit the context window and cost grows quadratically with turn count.
• For single-shot tasks, where one user message is clearer and cheaper than a synthesised conversation.
• Truncated naively from the front, which drops the system message and can split tool-call pairs.
• As the store of record for anything that matters. If a fact must survive the conversation, it belongs in a database, not in turn 4.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Message Threading — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path