HumanEval — AI Glossary

Reference details

Related terms

• Evaluation Benchmark
• SWE-Bench

How it works

1. Each problem supplies a function signature and a docstring, and the model must produce the body. The prompt is the signature plus documentation, so the task is synthesis from a specification rather than completion of partial code.
2. Generated code is executed against hidden unit tests in a sandbox. Correctness is functional — the tests pass or they do not — which avoids the paraphrase problem that defeats string-matching metrics on code.
3. The headline metric is pass@k: the probability that at least one of k sampled completions passes. It is a property of a sampling procedure, not of a single answer, which is why the sampling temperature is part of the result.
4. Estimating pass@k by drawing exactly k samples is biased, so the standard practice draws n > k samples, counts the c that pass, and computes the unbiased estimator 1 - C(n-c, k) / C(n, k).
5. The original test suites are thin — a handful of assertions per problem — so code that is wrong in an untested way passes. Extended suites add many more generated tests and measurably lower scores for the same completions.
6. The set is small, public and old by the standards of this field, which makes contamination the default assumption rather than a caveat and means the protocol must be stated for a number to mean anything.

Example: Why pass@1 and pass@10 are not the same measurement

Suppose a model solves a problem with independent probability 0.3 per sample. Then pass@1 is 0.3, and pass@10 is 1 - (1 - 0.3)^10 = 1 - 0.7^10 = 1 - 0.0282 = 0.972. The same model is 30% reliable and 97% capable on that problem, and both numbers are correct answers to different questions: pass@1 describes what a user gets from one request, pass@10 describes what is reachable with a verifier and ten attempts. Now the estimator. Draw n = 20 samples and observe c = 6 passing. The unbiased pass@10 is 1 - C(20-6, 10) / C(20, 10) = 1 - C(14, 10) / C(20, 10) = 1 - 1001 / 184756 = 1 - 0.00542 = 0.9946. Note it is not 1 - (1 - 6/20)^10 = 1 - 0.7^10 = 0.972 — the naive plug-in understates it here, which is exactly the bias the estimator exists to remove. The practical consequence: quoting pass@10 to describe an interactive coding experience overstates it by the gap between 0.97 and 0.30, and that gap is largest precisely where the model is weakest.

Commonly confused with

HumanEval measures function synthesis from a docstring, not software engineering, and the distance between them is the main thing to understand before using it to choose a model. Its problems are self-contained, 10-to-20-line functions with no repository, no dependencies, no existing conventions to match and no ambiguity in the specification. Benchmarks built on real repository issues measure a different and much harder task — locating the right file, understanding surrounding code, making a change that does not break other tests. A strong HumanEval result predicts competence at the first task and says little about the second.

When to use it

Reach for it when:

• A fast smoke test of basic code synthesis, where failing it is disqualifying and passing it is not informative.
• Comparing small or specialised code models, where the range is wide enough that the set still separates them.
• With an extended test suite, which closes the thin-test loophole and gives a more honest number for the same completions.
• Reporting pass@1 for interactive use and pass@k only where a verifier genuinely selects among k attempts.

Reach for something else when:

• As a measure of engineering capability on real codebases, which repository-level benchmarks address and this does not.
• Comparing frontier models, where the set is saturated and remaining differences are within contamination and protocol noise.
• Quoting pass@k as if it described one request. Without a verifier picking the passing sample, the user experiences pass@1.
• Without stating temperature, sample count and test suite, each of which moves the number more than most model differences do.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to HumanEval — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path