ChartQA — AI Glossary

Reference details

Related terms

• MMMU (Massive Multidiscipline Multimodal Understanding)
• VQA (Visual Question Answering)
• Multimodal Architecture

How it works

1. Charts are collected from real sources and paired with the underlying data tables they were drawn from, so every item has both an image and its ground-truth values.
2. Questions come in two splits generated differently: one written by human annotators looking at the chart, the other generated programmatically from the data table.
3. The machine-generated split is systematically more arithmetic and more literal, so a model can score well overall while being weak on the human-authored half, and the split proportions therefore matter to the figure.
4. Because the tables are published, the same questions can be asked with the table supplied instead of the image, which gives a perception-free control run.
5. That control is the diagnostic the benchmark makes possible: comparing image-given against table-given accuracy separates a reading failure from a reasoning failure.
6. Grading normalises the answer string before comparison, so formatting differences are forgiven while the tolerance governs how far a numeric answer may be wrong.

Example: Relaxed accuracy means near-misses count as correct

Numerical answers are graded within a tolerance — commonly 5%. If the true value is 40, an answer of 42 is inside 40 × 1.05 = 42 and is marked correct. That is a defensible choice, since reading a value off an unlabelled bar is inherently approximate. It also means the metric cannot distinguish a model that read the axis from one that estimated the bar height and landed close, and it means a reported score is not a claim that the extracted numbers are exact. For anything downstream that does arithmetic on the extracted value, that gap matters.

Commonly confused with

Chart question answering is not OCR and not table extraction. OCR reads characters that are present; chart reasoning infers values from geometry, often without a printed label to read. A model can be strong at transcribing a data table and weak at reading the same data as a bar chart, because only one of those requires estimating from pixels.

When to use it

Reach for it when:

• Evaluating document and dashboard workflows where charts carry the information
• Comparing vision encoders on reasoning rather than transcription
• Deciding whether a chart pipeline needs the underlying data rather than the image

Reach for something else when:

• Assuming extracted figures are exact enough to compute with
• As a general multimodal score — the domain is narrow by design
• Comparing results without the tolerance and normalisation rules used

Primary source

• Masry et al., ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to ChartQA — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path