Structured Output — AI Glossary

Reference details

Related terms

• JSON Mode (Structured Output)
• Function Calling
• Output Parser

How it works

1. The schema is compiled into a grammar — an automaton whose states describe which continuations keep the document valid.
2. At each generation step the automaton is consulted for the set of tokens that could legally come next, and every other token in the vocabulary is masked to negative infinity before sampling.
3. An invalid token is therefore unreachable rather than unlikely, which is the difference between output that is valid by construction and output that is valid by luck.
4. The mask has to be computed per step, so the grammar is pre-compiled and cached — otherwise the per-token cost of deciding legality dominates generation.
5. Tokens that span structural boundaries complicate this, since one token may contain the end of a string and the start of a delimiter, so the automaton operates over the tokenizer's vocabulary rather than over characters.
6. Nothing in the procedure examines meaning: the grammar constrains shape only, so every field can be populated with a confidently wrong value and the output will still validate.

Example: Asked for JSON vs. constrained to JSON, and the audit that follows

Ask politely for JSON and you will eventually get a fenced code block, a trailing comma, or a helpful sentence before the object — rare enough to pass review, frequent enough to page someone. Constrained decoding makes invalid tokens unreachable during generation, so the output is valid by construction rather than by luck, and your parser stops being a source of incidents. The asymmetry worth internalising is what an unvalidated 3% costs later rather than now. At 500,000 extractions a day, a 3% malformed rate is 500,000 x 0.03 = 15,000 records that either failed loudly or, worse, were coerced into something a downstream system accepted. Finding those afterwards means auditing all 500,000 to locate 15,000 — a search ratio of about 33 to 1 — because nothing in the stored record marks it as the one that went wrong. Validation at the boundary costs a schema; validation in arrears costs a full-table review.

Commonly confused with

Schema-valid is not correct. Constrained decoding guarantees shape, never truth: a response can satisfy the schema perfectly and still put the wrong value in every field. Validation checks the container, not the contents. Constrained decoding is also distinct from post-hoc validation — the first makes invalid output impossible to generate while the second makes it detectable afterwards and requires a retry, and only the first removes that retry from the cost model.

When to use it

Reach for it when:

• The output feeds code, a database or an API rather than a human
• You need fields present and typed so downstream logic can rely on them
• You are extracting records from unstructured text
• Where the schema can be enforced at generation rather than checked afterwards, which removes the retry

Reach for something else when:

• The reader is a person — schemas make prose worse
• The schema is very large or deeply nested; quality degrades and latency rises
• You are treating validity as accuracy and skipping evaluation
• Without a provenance field per record, since an unmarked bad row costs a full-table audit to find

Referenced by

This term is used by the following ModelRefs references:

• json mode
• constrained generation
• json schema tool call
• learn prompt engineering
• build document intelligence
• openai builder path

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Structured Output — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path