SWE-Bench — AI Glossary

Reference details

Related terms

• Evaluation Benchmark
• HumanEval

How it works

1. Each instance is a real repository pinned at a base commit, the text of the issue that was filed against it, and the tests that accompanied the eventual fix.
2. Two test sets are recorded separately: those that fail before the fix and must pass after, and those that pass before and must continue to pass.
3. The agent receives the repository and the issue text and must produce a patch — nothing identifies which files are relevant, so locating them is part of the task rather than a given.
4. Evaluation applies the patch in an isolated container, installs the environment at that commit, and runs both test sets against it.
5. Resolution requires every fail-to-pass test to pass and every pass-to-pass test to still pass, and the second set is what rejects a patch that addresses the issue by breaking something else.
6. Everything between receiving the issue and emitting the patch belongs to the scaffold rather than the model — how files are retrieved, how many attempts are made, whether tests are run between edits — which is why the harness has to be described for a figure to mean anything.

Example: The number describes a system, not a model

The task is: read an issue, find the relevant files in a real repository, write a patch, and have the maintainers' existing tests pass. Almost none of that is generation. Retrieving the right files, deciding how many attempts to make, whether to run tests between edits, how much of the repository to put in context — all of it lives in the scaffold around the model, and all of it moves the result. Two reported figures for the same model can differ substantially because the harnesses differ. Read any result as a claim about model *and* scaffold, and treat a comparison that does not describe the harness as uninterpretable.

Commonly confused with

The Verified subset is human-validated for solvable, well-specified issues; the full set contains tasks with underspecified reports and tests that cannot be passed from the issue alone. They are different exams and figures are not interchangeable. This is also not a code-generation benchmark in the usual sense — writing a correct function from a clear spec is a different skill from locating what to change in an unfamiliar codebase.

When to use it

Reach for it when:

• Assessing agentic engineering ability where navigation and iteration are the point
• Comparing scaffolds against a fixed model, which is what the harness sensitivity makes it good for
• Alongside a self-contained coding benchmark, which isolates generation from navigation

Reach for something else when:

• Comparing models across papers without identical harnesses and the same subset
• Generalising beyond Python and this repository set
• As a hiring-style proxy for engineering judgement — passing tests is not the same as a good patch

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to SWE-Bench — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path