HumanEval+ Leaderboard — AI Model Scores

What this benchmark measures

• function-level Python code synthesis
• hidden-test correctness

Relevant to:

• bounded code-generation screening
• coding evaluation portfolios

Failure modes it exercises:

• functional errors on generated functions

Method and its limits

Pass@1 function-generation evaluation using hidden tests; the displayed HumanEval+ label requires harness-specific review.

• Function-level tasks do not represent repository-scale engineering.
• The registered source describes HumanEval; extended HumanEval+ test provenance is not fully represented by this source alone.

Dataset

Dataset

HumanEval / HumanEval+

Type

function-level Python synthesis tasks with hidden tests

Freshness

aging

HumanEval is a 2021 hand-written Python function benchmark. HumanEval+ extends test coverage through EvalPlus, but scores must name the exact harness and test-suite revision.

How to read this score

Pass@1 is easy to read under a fixed harness; it should not be generalized to software-engineering quality.

Similarity to real tasks: Low — HumanEval covers 164 hand-written Python function-from-docstring tasks scored with unit tests. HumanEval+ substantially expands test coverage, but both remain isolated function-synthesis evaluations rather than repository, dependency, security, integration, review, or maintenance work.

Data contamination risk: High — HumanEval is public, long-running, and widely used; HumanEval+ adds tests but does not by itself establish that a model has not seen the prompt or related solutions.

Benchmark gaming risk: High — Scores are sensitive to pass@k, sampling count, execution sandbox, test-suite revision, and benchmark-specific tuning.

What you still need to test yourself

• Test repository context, dependencies, security, maintainability, and the target language mix.
• Execute generated code in an isolated environment and require project-specific tests and review.

This benchmark supports decisions about:

• Screen bounded Python function synthesis under a fixed HumanEval or HumanEval+ harness.
• Compare functional-correctness results only when sampling, pass@k, and test-suite conditions match.

Limitations

• Does not test secure coding, architecture, repository context, or long-term maintainability.
• Cross-report scores may use different sampling and pass@k settings.
• Every score needs a canonical run record with model version, dataset/harness revision, pass@k and sampling settings, source scope, and limitations; training overlap remains unknown unless separately documented.

Sources re-reviewed 2026-07-11. HumanEval is treated as an aging 2021 benchmark; HumanEval+ results remain harness-revision-specific rather than a single timeless dataset claim.

Sources

• HumanEval evaluation harness OpenAI · accessed 2026-07-11Official HumanEval implementation and sandbox warning; it does not document HumanEval+ or production software-engineering quality.
• Evaluating Large Language Models Trained on Code OpenAI · accessed 2026-07-11Primary source for HumanEval's function-from-docstring functional-correctness task and its reported limitations.
• EvalPlus EvalPlus authors · accessed 2026-07-11Primary implementation reference for HumanEval+ and its expanded tests; exact version and execution settings still matter.

How this benchmark is scored

Primary source: https://github.com/openai/human-eval

Published results

Each score reflects the protocol and date of its own source run. Results from different harnesses are not directly comparable.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to HumanEval+ Leaderboard — AI Model Scores.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path