Pass@k — AI Glossary

Reference details

Related terms

• LiveCodeBench
• BigCodeBench
• Code Generation

How it works

1. The naive procedure — draw k samples and check whether any passed — is itself a random variable, so repeating it on the same model gives different answers.
2. The metric is therefore an estimator: generate n samples where n is comfortably larger than k, count how many passed, and compute the probability that a random subset of size k would contain at least one.
3. That probability is one minus the chance of drawing k failures from the n samples, which is a ratio of binomial coefficients and needs no simulation.
4. With ten samples of which three passed, the estimate at k = 1 is 1 - C(7,1)/C(10,1) = 1 - 7/10 = 0.3, and at k = 5 it is 1 - C(7,5)/C(10,5) = 1 - 21/252 ≈ 0.9167.
5. Using n much larger than k is what makes the estimate stable, so a figure reported from n = k is the noisy naive version wearing the estimator's name.
6. Sampling temperature enters as a parameter of the generation rather than the estimator, and it moves the two ends in opposite directions — more diversity raises the chance that some sample passes while lowering the chance that the first one does.

Example: Why pass@10 flatters, in one line of arithmetic

If a model solves a problem with probability 0.3 on any single attempt, and attempts are independent, then at least one of ten succeeds with probability 1 − (1 − 0.3)^10 = 1 − 0.7^10 ≈ 0.97. A model that is right under a third of the time reports 97%. Nothing is wrong with the metric — it is answering “can it ever get there”, which is the right question when a verifier exists to pick the passing sample. It is the wrong question when the user sees one answer, and pass@1 is the number that describes that.

Commonly confused with

Pass@k is not accuracy, and the two diverge fast as k grows. It also depends on sampling temperature: higher temperature raises diversity and therefore pass@k while often lowering pass@1. Reporting pass@k without the temperature, the sample count and the value of k describes almost nothing.

When to use it

Reach for it when:

• Settings with a cheap verifier — unit tests, a type checker, a proof assistant — that can pick the winner
• Measuring headroom: how often the ability is present but not reliably surfaced
• Comparing sampling and search strategies against a fixed model

Reach for something else when:

• Predicting single-shot user experience, which pass@1 describes
• Where no verifier exists, since nobody can tell which of the k samples was the passing one
• Comparing across papers without k, n and temperature stated

Primary source

• Chen et al., Evaluating Large Language Models Trained on Code (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Pass@k — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path