Fill-in-the-Middle (FIM) — AI Glossary

Reference details

Related terms

• Code Completion
• Decoder-Only Architecture
• Context Window

How it works

1. Training examples are rearranged rather than the architecture changed: a document is split into prefix, middle and suffix, and the pieces are reordered so the middle comes last.
2. Sentinel tokens mark the boundaries, so the model learns that a particular token sequence means 'the suffix follows' and another means 'now produce the middle'.
3. Because the middle is last, ordinary left-to-right training teaches infilling — the model is still predicting the next token, with the suffix already in its context.
4. A fraction of training examples is transformed this way and the rest left in natural order, and that rate is the hyperparameter: too low and infilling is weak, too high and plain continuation degrades.
5. At inference the caller assembles the same sentinel structure, placing the code before and after the cursor into the prefix and suffix slots.
6. The result is bidirectional conditioning from a causal model with no architectural change, which is why the technique is a data transformation rather than a model variant.

Example: What the sentinel overhead actually costs

A transformation adding 3 sentinel tokens per example sounds free, and it is worth confirming. At a one-trillion-token pretraining corpus in examples of 2,048 tokens, the corpus holds 1e12 / 2,048 = 488,281,250 examples. Transforming all of them adds 488,281,250 x 3 = 1,464,843,750 sentinel tokens, which against the trillion is 1.46e9 / 1e12 = 0.146% of the training budget — negligible, and that is the point: the capability is bought with a data reordering rather than with compute. The cost that is not negligible is the rate trade-off. Transforming half the examples leaves 1e12 x 0.5 = 5e11 tokens of natural left-to-right order, so the model sees half as much plain continuation as it otherwise would, and plain continuation is what powers every non-infilling use. That is the actual decision: the sentinel tokens cost 0.146% and the displaced ordinary examples cost a fraction of the training signal equal to the transformation rate, which is why published rates sit in the middle of the range rather than at either end.

Commonly confused with

This produces bidirectional conditioning without a bidirectional architecture, and that distinction matters when comparing it to an encoder. An encoder-only model attends in both directions at every layer, so each token's representation is informed by the whole input — and it cannot generate. A causal model trained with this objective generates normally and receives the suffix as ordinary context that precedes the generation point, which is conditioning rather than bidirectional attention. The practical consequence is that the suffix influences the output through the same mechanism as any other context, including the same position-dependent reliability.

When to use it

Reach for it when:

• Code completion inside existing files, which is the canonical case and where the suffix carries most of the constraint.
• Any infilling task — inserting into a document, filling a template, completing a gap with known surroundings.
• During pretraining, since the capability comes from the data transformation and cannot be added by prompting afterwards.
• With the transformation rate chosen deliberately, since it trades infilling quality against plain continuation.

Reach for something else when:

• On a model not trained with the objective, where the sentinel structure is meaningless and the model will continue rather than infill.
• As a substitute for an encoder on representation tasks, since conditioning on a suffix is not bidirectional attention.
• With a mismatched sentinel format, which is a common integration error and presents as the model ignoring the suffix.
• At a transformation rate high enough to degrade plain continuation, which is the cost the rate controls.

Primary source

• Bavarian et al., Efficient Training of Language Models to Fill in the Middle (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Fill-in-the-Middle (FIM) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path