DROP — AI Glossary

Reference details

Related terms

• MATH Benchmark
• Multi-Document Question Answering
• Reasoning (LLM)

How it works

1. Passages are drawn from Wikipedia text dense in numbers, dates and repeated entities, so the material supports operations rather than only lookups.
2. Questions are written adversarially against a span-extraction baseline: an annotator is shown a model's answer and rewarded for composing a question it gets wrong.
3. Answering requires a discrete operation over values found in the passage — subtracting two quantities, counting how many entities satisfy a condition, or ordering events — rather than locating a phrase.
4. The consequence is that the correct answer frequently does not appear anywhere in the passage, so no extractive system can return it however well it reads.
5. Answers come in three types — a number, a date, or a set of spans — and the grader handles each differently, since a set answer can be partially right.
6. Scoring is token-level F1 against the gold answer alongside exact match, which grants partial credit on multi-span answers and none on a wrong arithmetic result.

Example: Why the answer space outgrows the passage

Consider what span extraction has available against what the task asks for. A passage mentioning n numbers offers n numeric spans a model could return, growing linearly with the passage. The pairwise differences those numbers support grow as n(n-1)/2: at n = 8 that is 8 x 7 / 2 = 28 distinct differences against 8 extractable spans, a factor of 3.5, and at n = 20 it is 190 against 20. Counting and sorting questions expand the space further. So the gap between what is present in the text and what the task can ask about widens as passages get richer, which is why the benchmark is constructed around numeric density rather than length. The grading rule is worth understanding separately, because it changes what a score means. Token-level F1 gives partial credit on set answers: if the gold answer is three spans and a system returns two of them, both correct, precision is 1.0 and recall is 2/3, so F1 = 2 x 1.0 x 0.667 / (1.0 + 0.667) = 0.8 while exact match scores 0. On a numeric answer there is no such cushion — an arithmetic slip produces a different token entirely and scores zero on both measures. A single reported figure therefore mixes a metric that is generous on set answers with one that is unforgiving on arithmetic.

Commonly confused with

DROP is not a mathematics benchmark, though it requires arithmetic. A maths benchmark supplies the quantities in the question and tests whether the operation is performed correctly; DROP buries the quantities in prose and tests whether the right ones are identified before any operation happens. The operations it requires are elementary — addition, counting, ordering — so its difficulty sits in selecting the right quantities out of prose rather than in computing with them once selected. It is also not an extractive reading-comprehension benchmark despite its lineage, and this is the distinction its name encodes: the answers were deliberately placed outside the set of passage spans, so a system architecture that can only point at text has a ceiling on it that no amount of training removes.

When to use it

Reach for it when:

• To test whether a system can operate on quantities it has read, rather than retrieve phrases containing them.
• As a diagnostic for numerical grounding, where the question is which numbers were selected and not whether arithmetic was done.
• When reporting set-answer tasks, where the F1 and exact-match figures carry genuinely different information.
• As an argument for generative over extractive architectures, which it was constructed to make.

Reach for something else when:

• As a measure of mathematical reasoning, which it confounds with reading comprehension by design.
• Reported as a single number without saying which metric, since F1 is generous on sets and exact match is not.
• To compare against extractive systems, which the construction deliberately handicaps.
• As evidence about long-context behaviour, given the passages are single paragraphs.

Primary source

• Dua et al., DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to DROP — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path