ModelRefs / HellaSwag — AI Glossary

HellaSwag — AI Glossary

A commonsense NLI benchmark requiring models to choose the most plausible continuation for a partially described activity.

Overview

HellaSwag (Zellers et al. 2019) uses adversarial filtering to generate hard negatives that fool BERT-era models but are easy for humans (95%). Tests grounded commonsense reasoning about physical and social situations. GPT-4 achieves ~95%; smaller 7B models score 80–88%.

Reference details

Topic	evaluation
Last reviewed	2026-06-24

How it works

Each item pairs a short context with four continuations, one of which is the genuine caption from the source video or article and three of which were machine-generated.
Candidates come from a language model conditioned on the same context, so every distractor is locally fluent and topically apt before any filtering happens.
That filtering is iterative rather than a single pass — candidates are generated, a discriminator is trained to tell real from generated, the candidates it fails on are kept, and the loop repeats against the strengthened discriminator.
Each round therefore raises the bar for exactly the kind of model used as the discriminator, which is what makes the final set hard for that model class and easy for people.
Scoring compares the likelihood the model assigns to each of the four continuations and takes the highest, so the task is discrimination among supplied strings.
Because the adversary was a model of one era, improvement on the set partly measures no longer failing in that era's characteristic way — a narrower claim than the task name suggests.

Example: What the four options actually are

The model gets a short context — someone begins a familiar physical activity — and four possible continuations, of which one is the real caption from the source and three were machine-generated. The wrong options were not written to be obviously wrong: they were kept precisely because a discriminator model of the period was fooled by them while humans were not. That construction is the whole design, and it is also the caveat. Progress on the set partly measures not being fooled by a 2019-era generator, which is a narrower claim than commonsense reasoning.

Commonly confused with

HellaSwag is multiple choice scored by likelihood over fixed options, so it tests discrimination, not generation. A model that picks the right ending cannot necessarily write one. Read it beside a generative benchmark rather than as a summary of reasoning ability.

When to use it

Reach for it when:

Regression checks after quantization, pruning or distillation, where a drop signals real damage
Smoke-testing a small or fine-tuned model against a widely reported baseline
Cheap, fast, deterministic scoring that needs no judge model

Reach for something else when:

Separating frontier models, which cluster near the human ceiling
Any claim about agentic or long-form ability — the format cannot reach it
Comparing published figures without checking the prompt format and normalisation used

Primary source

Zellers et al., HellaSwag: Can a Machine Really Finish Your Sentence? (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to HellaSwag — AI Glossary.

Frequently asked questions

What is HellaSwag?