Aider Polyglot — AI Glossary

Reference details

Related terms

• SWE-Bench
• LiveCodeBench
• Coding Agent

How it works

1. Each exercise supplies existing source files and a task requiring a modification, so the model must locate the right place to change rather than write a function from a blank docstring.
2. The same exercises are posed across several programming languages, which separates general problem-solving from per-language competence and is where the name comes from.
3. Edits must be expressed in an applicable format — a search-and-replace block or a diff — so producing a correct change the harness cannot apply counts as a failure, which is deliberate and reflects real tool use.
4. Scoring runs the repository's test suite after the edit, so correctness is functional rather than textual and a differently-written correct change passes.
5. A second attempt is permitted after a failure, with the test output supplied, which measures whether a model can use an error message — a distinct capability from getting it right first time.
6. The exercise count is in the low hundreds, which bounds the precision of any comparison drawn from it.

Example: How large a difference this set can actually resolve

With 225 exercises, the standard error of a proportion is sqrt(p(1-p)/n), largest at p = 0.5: sqrt(0.5 x 0.5 / 225) = sqrt(0.001111) = 0.0333, about 3.3 points. Comparing two models, the standard error on the difference is sqrt(2) x 3.3 = 4.7 points, so a gap under roughly 5 points is within one standard error and carries almost no information. Nearer the ends of the range the interval tightens — at p = 0.8 it is sqrt(0.8 x 0.2 / 225) = 0.0267, about 2.7 points, giving 3.8 points on a difference — but the order of magnitude holds: this set resolves differences of several points, not fractions of one. To halve that you need four times the exercises, since the standard error falls with the square root of n: 900 exercises would bring the difference's standard error at p = 0.5 to 4.7 / 2 = 2.4 points. The practical reading is that a leaderboard ordering among closely-placed models on 225 exercises is mostly noise, while a gap of 15 or 20 points is a real finding.

Commonly confused with

This measures code editing, not code synthesis, and the distinction is why it exists alongside function-writing benchmarks. Synthesis benchmarks hand over a signature and a docstring with no surrounding code, so they test whether a model can write a self-contained function. Editing requires reading existing code, finding the right location, matching local conventions, and producing a change in a format a tool can apply — which is closer to what a coding assistant is actually asked to do. A model can be strong at the first and weak at the second, most often by failing the edit format rather than the logic.

When to use it

Reach for it when:

• Selecting a model for a code-editing assistant, where applying diffs to existing files is the actual task.
• Assessing cross-language breadth, which the repeated-across-languages design isolates.
• Measuring whether a model can act on a test failure, which the second-attempt protocol captures.
• Reading differences of ten points or more, which is the resolution the exercise count supports.

Reach for something else when:

• Ordering closely-placed models, where gaps of a few points are inside one standard error.
• As a measure of repository-scale engineering, which involves navigating a large codebase rather than editing supplied files.
• Comparing results obtained under different edit formats or attempt counts, both of which move the number materially.
• As a general capability proxy, since code editing does not predict reasoning, writing or agentic reliability.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Aider Polyglot — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path