N Completions — AI Glossary

Reference details

Related terms

• Pass@k
• Sampling (LLM)
• Log Probabilities

How it works

1. The prompt is processed once, producing a single set of key-value tensors for the shared prefix.
2. That prefill state is then branched into n independent decode streams, each sampling from the same distribution with a different random draw.
3. Because the prefill is shared, input tokens are consumed once while output tokens are consumed n times, which is where the pricing asymmetry comes from.
4. Sampling must be stochastic for the completions to differ, so temperature above zero is a precondition rather than an option.
5. The n results are returned together with per-completion indices and finish reasons, since any one of them may terminate differently.
6. What makes the technique useful is a selector downstream — a verifier, a reward model, a test suite — because without one, n samples are n guesses.

Example: The prefill is shared, and the selector is the whole value

Take a 3,000-token prompt and n = 8 completions of 500 tokens each. The prompt is prefilled once, so input is 3,000 tokens and output is 8 x 500 = 4,000 tokens. At $3 per million input and $15 per million output that is 0.003 x $3 = $0.009 plus 0.004 x $15 = $0.060, so $0.069. Issuing 8 separate requests instead charges the prompt 8 times: 8 x 3,000 = 24,000 input tokens at 0.024 x $3 = $0.072, with the same $0.060 of output, so $0.132. The comparison is $0.069 against $0.132, about 1.91 times cheaper, saving $0.063 per task — and the advantage grows with prompt length, since the whole gain is the prefill charged once. The selection arithmetic is where the technique either pays or does not. If a single sample succeeds 35% of the time and the samples are independent, the probability that at least one of 8 succeeds is 1 - 0.65^8 ≈ 96.8%. That figure is what pass@8 reports, and it is an upper bound on what the technique delivers rather than a result. Realising it requires knowing which of the 8 succeeded. With a test suite or a verifier, 96.8% is achievable. Without one, picking a completion at random returns to the 35% a single sample gives, and the extra $0.0525 bought nothing — so the selector is not a detail of the setup, it is the entire mechanism.

Commonly confused with

Requesting n completions is not the same as self-consistency, though the former is how the latter is usually implemented. The n parameter gets you n samples; self-consistency is the specific strategy of marginalising over them by majority vote on the final answer, which works where answers are comparable — a number, a label — and not where they are prose. It is also distinct from beam search, which explores several partial sequences and returns the highest-likelihood completion. Beam search optimises for likelihood and yields near-identical candidates; sampling n completions at temperature optimises for diversity and yields genuinely different attempts, which is what best-of-N selection needs and what beam search cannot supply.

When to use it

Reach for it when:

• With a programmatic verifier such as a test suite, which is what converts the 96.8% pass@8 figure above into a real result.
• For best-of-N selection against a reward model, where a scorer stands in for a verifier.
• In pass@k evaluation, which is the measurement the parameter was designed to support.
• On long prompts, where sharing the prefill made the batched form about 1.91 times cheaper than separate calls.

Reach for something else when:

• Without a selector, since picking at random from 8 samples returns the single-sample success rate of 35% in the case above.
• At temperature zero, where the completions are identical and the output tokens are paid for n times.
• For self-consistency over free-form prose, where there is no comparable answer to take a majority over.
• When one good answer is needed cheaply, where a single call with a better prompt beats sampling and discarding.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to N Completions — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path