Instruction Following — AI Glossary

Reference details

Related terms

• Supervised Fine-Tuning (SFT)
• System Message
• Calibration

How it works

1. Instruction-tuned behaviour is created by supervised fine-tuning on instruction-response pairs, which teaches the model that a prompt is a directive to be executed rather than a passage to be continued.
2. Preference optimisation then ranks competing responses to the same instruction, which is where constraint adherence is learned: a response that respects a stated word limit is preferred over a better-written one that ignores it.
3. At inference the instruction competes with other pressures — the system prompt, retrieved context, conversation history, and the model's prior over fluent text. Constraint violations are usually this competition resolving against the instruction, not a failure to parse it.
4. Verifiable constraints are the tractable evaluation target: word counts, required or forbidden strings, letter casing, JSON validity, language. Each can be checked by a program, so the benchmark needs no judge model and no human.
5. Scoring is reported at several granularities — per instruction and per prompt, under strict and loose normalisation — because a prompt carrying four constraints scores very differently if one is missed and the rest hold.
6. Decoding settings interact with adherence: higher temperature raises violation rates on format constraints, and a grammar or schema constraint at decode time removes format failures entirely rather than reducing them.

Example: Why per-prompt scores fall so far below per-constraint scores

A prompt carries 4 independent verifiable constraints — under 100 words, valid JSON, include a named key, no exclamation marks. Suppose the model satisfies each with probability 0.95 independently. Per-constraint adherence is 95%, which sounds strong. Per-prompt adherence, which requires all four, is 0.95^4 = 0.8145, about 81%. At 6 constraints it is 0.95^6 = 0.7351, and across 1,000 such prompts that is 265 failures. The point is structural rather than about any model: a headline instruction-following figure is almost always per-constraint, while a user experiences per-prompt. If your production prompts carry 6 constraints and you need 95% of requests fully compliant, the required per-constraint rate is 0.95^(1/6) = 0.9915 — not 95% — which is why format constraints are better removed from the model's discretion by schema-constrained decoding than improved by prompting.

Commonly confused with

Instruction following is not reasoning ability, and the two diverge in both directions. A model can execute a 6-constraint formatting instruction perfectly and be wrong about the content; a strong reasoner can produce the correct answer in the wrong format and score zero on a compliance benchmark. It is also not the same as steerability or helpfulness: compliance measures whether stated constraints were met, not whether the response was useful or whether the model inferred unstated intent. Reading an instruction-following score as a general capability number is the usual error.

When to use it

Reach for it when:

• Selecting a model for pipelines where downstream code parses the output and a format violation is an outage.
• Diagnosing whether failures are format or content: a verifiable-constraint suite separates the two cleanly.
• Comparing small models for high-volume structured work, where adherence varies far more than it does between frontier models.
• Testing system-prompt designs, since constraint adherence is measurable and prompt wording measurably moves it.

Reach for something else when:

• As a proxy for reasoning, factuality or domain competence, which it does not measure.
• As a reason to skip schema-constrained decoding. If JSON validity matters, constrain the decoder; do not buy adherence with prompt engineering and hope.
• To compare models whose scores come from different constraint sets or different strictness settings, where the normalisation choice moves the number more than the model does.
• For instructions whose satisfaction cannot be programmatically checked — 'be concise', 'use a professional tone' — which need a judge and carry that judge's biases.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Instruction Following — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path