ModelRefs / ARC Challenge — AI Glossary

ARC Challenge — AI Glossary

A grade-school science question-answering benchmark with adversarially filtered questions that retrieve-then-answer systems fail.

Overview

ARC Challenge (Clark et al. 2018) contains 1,172 questions from US 3rd–9th grade science exams that neither word-overlap nor retrieval-based models could answer correctly. Tests systematic reasoning and scientific knowledge. Frontier LLMs score >90%; 7B models typically 75–85%.

Reference details

Topic	evaluation
Last reviewed	2026-06-24

How it works

Scoring an item means ranking four supplied strings, and the ranking is not uniquely determined — the protocol has a free choice that changes which option wins.
That comparison can be made three ways, and they are not equivalent: the raw sum of log-probabilities, that sum divided by token count, or the sum divided by the option's likelihood without the question.
The third form removes the model's prior for strings that are simply common in English, which is the correction that matters most when options differ in how ordinary they sound.
Each normalisation produces a different ranking over the same model outputs, so two reported figures can differ by several points with no difference in the model at all.
Few-shot examples are prepended from a separate split, and the count is a further free parameter that moves results independently of everything above.
None of this is visible in a bare score, which is why the split, the normalisation and the shot count together are what make a figure comparable to another.

Example: Filtered by what retrieval could not solve

The challenge split was built by keeping the grade-school science questions that word-overlap and retrieval-based solvers of the time got wrong. That construction selects hard cases efficiently, and it also shapes what a score means: the set is enriched for questions where surface matching fails, which is a narrower claim than “tests reasoning”. It shares this property with other adversarially filtered benchmarks — progress partly measures not failing the way a particular era's solver failed. Useful as a floor check on a small model, weak as a discriminator between strong ones.

Commonly confused with

The challenge split is the filtered subset; an easy split of the same questions exists and is much less informative. Figures quoted without naming the split are ambiguous, and multiple-choice scoring by option likelihood also differs from asking the model to answer generatively — the two do not produce the same number.

When to use it

Reach for it when:

Regression checks on small models after quantization or fine-tuning
Cheap, deterministic scoring with no judge model in the loop
As one of several floor checks, never as a headline capability claim

Reach for something else when:

Distinguishing frontier models, which cluster at the top of the range
Any claim about generation, agentic ability or long-form reasoning
Comparing figures without the split, prompt format and normalisation stated

Primary source

Clark et al., Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to ARC Challenge — AI Glossary.

Frequently asked questions

What is ARC Challenge?