WinoGrande — AI Glossary

Reference details

Related terms

• HellaSwag
• ARC Challenge
• Commonsense Reasoning

How it works

1. Each item is a sentence with an ambiguous pronoun and two candidate referents, and only one is consistent with the described situation.
2. Items come in near-identical pairs differing by one word that flips the correct answer, so surface cues cannot resolve them.
3. Candidate items were adversarially filtered against a model, removing those a model could answer from statistical artefacts alone.
4. That filtering is the design's core: it targets the shortcut rather than the task, which is why earlier commonsense sets became easy without the underlying ability improving.
5. Scoring is binary choice, so the chance baseline is 50% and a reported score must be read against that floor.

Example: Reading a binary-choice score against its floor

A score of 70% on a two-option task sounds like 70% competence; measured against a 50% chance baseline, it is 20 points of 50 available above chance — about 40% of the way from guessing to perfect. A model at 55% is barely above noise, and on a 1,000-item sample the standard error on a proportion near 0.5 is sqrt(0.25/1000) ≈ 1.6 points, so 55% versus 52% is roughly two standard errors and 55% versus 54% is not a difference.

Commonly confused with

Treated as a general commonsense benchmark, which is broader than what it tests. It targets pronoun resolution requiring physical, social or causal inference — one slice of commonsense, deliberately narrow so the signal is clean. It is also the adversarially-filtered successor to an earlier, smaller Winograd set, and the filtering is the point rather than the scale: the predecessor became solvable by pattern-matching, and this was constructed to remove exactly those shortcuts.

When to use it

Reach for it when:

• Testing commonsense pronoun resolution specifically rather than reasoning broadly.
• A benchmark resistant to statistical shortcuts is wanted.
• Explaining why a chance baseline must be stated with a binary-choice score.

Reach for something else when:

• As a measure of general reasoning, which it deliberately narrows away from.
• Without reporting the 50% baseline, which makes the score look better than it is.
• Where contamination is a concern, since it is a public fixed dataset.

Primary source

• Sakaguchi et al., WinoGrande: An Adversarial Winograd Schema Challenge at Scale (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to WinoGrande — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path