LiveCodeBench — AI Glossary

Reference details

Related terms

• SWE-Bench
• Pass@k
• BigCodeBench

How it works

1. Problems are scraped continuously from competitive-programming sites and stamped with the date their contest published them, which is the field everything else depends on.
2. The benchmark publishes numbered releases, each covering a stated date range, so a run is citable as a specific release rather than as the project generally.
3. That makes the evaluation window a required parameter of any reported figure: two models released months apart are scored on disjoint sets of different difficulty.
4. The benchmark defines four scenarios over those problems, not one — generating a solution, repairing a failing one given its error output, predicting a program's output, and predicting whether code executes correctly.
5. A bare figure almost always refers to the first scenario, so the name alone does not identify which capability was measured.
6. Grading runs the contest's own hidden tests in a sandbox, which is why the measurement inherits the contest's notion of correctness including its time limits.

Example: The same benchmark, a different exam per model

Problems are timestamped from when the contest published them, and a model is evaluated only on problems dated after its own training cutoff. That is the point — it cannot have memorised them. It is also the caveat: a model released in March and one released in September are scored on different problem sets, of different difficulty, drawn from different contests. Two papers can both report “LiveCodeBench” and not be comparing the same exam. Any comparison you make must state the evaluation window, or it is not a comparison.

Commonly confused with

It is not a replacement for SWE-bench. LiveCodeBench asks for a self-contained solution to a stated competitive-programming problem; SWE-bench asks for a patch to a real repository that makes a failing test pass. Competitive-programming skill and codebase navigation are different capabilities, and models rank differently on them.

When to use it

Reach for it when:

• Checking algorithmic coding ability without contamination from memorised training data
• Tracking one model family over time, where the window can be held fixed
• Alongside a repository-level benchmark, not instead of one

Reach for something else when:

• Cross-model league tables that do not state the problem window
• Predicting performance on production engineering work — the task shape is unrelated
• Evaluating a model on problems predating its cutoff, which silently reintroduces contamination

Primary source

• Jain et al., LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to LiveCodeBench — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path