VQA (Visual Question Answering) — AI Glossary

Reference details

Related terms

• MMMU (Massive Multidiscipline Multimodal Understanding)
• ChartQA
• Multimodal Architecture

How it works

1. Each question is answered independently by ten human annotators, so the ground truth is a distribution of short answers rather than a single string.
2. A model's answer scores the number of annotators who gave it divided by three, capped at one — so matching three of the ten is already full credit.
3. That rule exists because open-ended visual questions have several correct phrasings, and requiring unanimity would penalise a right answer for being worded differently.
4. It also makes answer normalisation load-bearing: articles, plurals and number words are rewritten to a canonical form before matching, and a model whose phrasing escapes that normalisation loses credit it earned.
5. The dataset is built in complementary pairs — the same question asked of two images with different answers — so a model ignoring the image cannot exploit the answer prior.
6. Those pairs are the structural fix for a measurement problem rather than a difficulty increase, and the general form of the lesson is that an evaluation a text-only model passes is measuring the language prior.

Example: Balanced pairs, and the prior they were built to break

Early visual question answering could be gamed without looking at the image: “what colour is the banana” is answered correctly by a model that has learned bananas are yellow. The fix was structural — for each question, include a second image where the same question has a different answer. Now a model that ignores the image is right about half the time on that pair instead of nearly always. That construction is the benchmark's real contribution, and it generalises: whenever a multimodal eval is easy for a text-only model, the eval is measuring the language prior.

Commonly confused with

Visual question answering is a task, and the dataset of that name is one instance of it. Chart and document question answering are separate benchmarks with different failure modes — reading a value off an axis is not the same skill as recognising an object, and a model can be strong at one and weak at the other.

When to use it

Reach for it when:

• Checking that a multimodal model actually conditions on the image
• Regression testing a vision encoder or projection layer change
• As a floor check, alongside document and chart benchmarks that probe different skills

Reach for something else when:

• As a summary of multimodal ability — the question distribution is narrow
• Judging OCR or dense-text reading, which it barely exercises
• Comparing figures without the answer-normalisation rules, which move results materially

Primary source

• Goyal et al., Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to VQA (Visual Question Answering) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path