SWE-Bench Verified — AI Glossary

Reference details

Related terms

• Coding Agent
• LiveCodeBench
• Aider Polyglot

How it works

1. Every instance in the original set was reviewed by professional developers against a fixed rubric rather than filtered by any automated heuristic.
2. Annotators judged two things separately: whether the issue description specifies the required change unambiguously, and whether the test patch actually tests that change.
3. Instances whose tests could pass without solving the described problem, or fail despite solving it, were excluded as ungradeable regardless of difficulty.
4. Instances requiring environment state the harness does not reproduce were also removed, since those fail for reasons unrelated to the agent.
5. The surviving instances form a fixed, published subset, so a score computed on it is comparable across submissions in a way a self-selected sample is not.
6. Grading itself is unchanged — apply the patch, run the test suite, require the fail-to-pass tests to pass and the pass-to-pass tests to keep passing — so only the instance pool differs.

Example: What the smaller pool costs in measurement precision

The subset keeps 500 of 2,294 instances, which is 500 / 2,294 ≈ 21.8%, excluding 2,294 - 500 = 1,794 as ambiguous or ungradeable. That exclusion buys validity and spends precision, and the amount spent is calculable. For an observed resolution rate p on n instances, the standard error is sqrt(p(1-p)/n), maximised at p = 0.5, which is the conservative case to plan with. At n = 500 that is sqrt(0.25 / 500) ≈ 0.0224, or about 2.24 percentage points, so a 95% interval extends roughly 1.96 x 2.24 ≈ 4.38 points either side. At n = 2,294 the same calculation gives sqrt(0.25 / 2294) ≈ 0.0104, about 1.04 points, and an interval of roughly 1.96 x 1.04 ≈ 2.05 points. So the two pools support intervals of about 4.38 against 2.05 percentage points, a factor of about 2.14 — which is just sqrt(2294 / 500) ≈ 2.14, since precision follows the square root of sample size. The practical reading is a resolution floor on claims, and it needs one more step than it first appears. The 4.38 figure is the interval around a single system's own rate. Distinguishing two systems is a stricter test, because the standard error of a difference combines both: sqrt(2 x 0.25 / 500) ≈ 0.0316, so about 1.96 x 3.16 ≈ 6.20 points separate two independently evaluated systems before the difference is significant. Leaderboard entries are scored on the same 500 instances, so a paired comparison does better than 6.20, since only the instances where the two systems disagree carry any information about which is better. What the subset trades is precision of this kind for the guarantee that every instance in it is gradeable at all.

Commonly confused with

SWE-Bench Verified is not an easier SWE-Bench, and the distinction matters for how its numbers are read. The filtering removed instances that were ungradeable — underspecified issues, tests that did not test the described change — rather than instances that were hard. Resolution rates are higher on the subset not because the problems were softened but because the original set charged agents for failures the grader created. It is also not a different task: the repositories, the patch format and the grading procedure are identical, so the only change is which instances are in the pool. And it is not interchangeable with the full set in a comparison — a figure from one cannot be set beside a figure from the other, since they are rates over different populations.

When to use it

Reach for it when:

• As the comparison set when evaluating coding agents, since a published fixed pool is what makes submissions comparable.
• When a reported figure needs an uncertainty attached, where about 4.38 points is the honest interval on one rate and about 6.20 the bar for separating two.
• For tracking a single agent's progress over time, where a stable gradeable pool isolates real change from grader noise.
• As the example to reach for when arguing that benchmark validity is a property of the instances, not only of the metric.

Reach for something else when:

• Compared directly against full-set figures, which are rates over a different and partly ungradeable population.
• To separate two systems on a difference below their comparison interval, which for an independent comparison at this sample size is about 6.20 points.
• As evidence of general software-engineering ability, given the pool is Python library issues with existing test coverage.
• Described as a difficulty-filtered subset, which misstates what the annotation removed and why the rates differ.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to SWE-Bench Verified — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path