ModelRefs / MKQA Leaderboard
MKQA Leaderboard
Cross-lingual question retrieval benchmark built from Multilingual Knowledge Questions & Answers across 26 languages. Current leaders and methodology for MKQA.
Overview
Cross-lingual question retrieval benchmark built from Multilingual Knowledge Questions & Answers across 26 languages.
How it is measured: Aggregate Recall@100 for multilingual query-to-English-passage retrieval; model records must preserve the reported dense, sparse, or hybrid configuration.
What this benchmark measures
- multilingual open-domain question answering
- cross-lingual answer consistency
Relevant to:
- multilingual assistant screening
- language-coverage planning
Failure modes it exercises:
- answer-quality collapse in low-resource languages
Method and its limits
10k question-answer pairs aligned across 26 languages, for 260k pairs in total, with answers curated in a language-independent representation rather than tied to language-specific passages.
- Answers are deliberately retrieval-independent, so the benchmark does not test grounding against a corpus.
- The paper reports the set is challenging even in English and markedly harder in low-resource languages, so cross-language comparisons reflect dataset difficulty as well as model ability.
Dataset
- Dataset
- Multilingual Knowledge Questions and Answers
- Type
- aligned open-domain question answering pairs
- Freshness
- aging
- Benchmark release
- 2020-07-30
10k questions aligned across 26 languages as registered in the primary paper. The date records paper submission; world knowledge in the answers may since have changed.
How to read this score
Scores are comparable across languages by design, which is the benchmark's main contribution; absolute values depend on the answer-matching rule used.
Similarity to real tasks: Medium — Established from the primary methodology: MKQA aligns 10k question-answer pairs across 26 typologically diverse languages, with answers held in a language-independent representation so results are comparable across languages. Open-domain question answering resembles assistant workloads. It does not reproduce a deployed system, because retrieval, grounding, and refusal behaviour are outside the benchmark.
Data contamination risk: High — MKQA derives from Natural Questions and has been public since 2020, so overlap with training corpora is likely. No run-specific audit is registered.
Benchmark gaming risk: Low — There is no prominent public leaderboard creating direct optimisation pressure, though answer-matching rules still affect reported numbers.
What you still need to test yourself
- Add retrieval and grounding, which this benchmark deliberately excludes.
- Re-check answers whose correctness may have changed since 2020.
This benchmark supports decisions about:
- Compare answer quality across languages on a like-for-like question set.
- Identify languages where a candidate model degrades.
Limitations
- Not evidence of grounded or retrieval-augmented answer quality.
- Time-sensitive answers may now be outdated.
Primary source reviewed 2026-10-11. MKQA is treated as an aging 2020 benchmark whose answers encode 2020 world knowledge.
What this evidence does not cover
- Only one model is currently scored on this page, so no cross-model comparison is supported.
Sources
- MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering Longpre, Lu, Daiber · accessed 2026-10-11Supports the 10k-question, 26-language alignment, the language-independent answer representation, and the stated low-resource difficulty; it does not support any model score shown here.
- MKQA dataset release Apple · accessed 2026-10-25Apple's MKQA release, which states the evaluation set comprises 10k question-answer pairs aligned across 26 typologically diverse languages, 260k pairs in total. It establishes the dataset's size and language coverage; it does not establish retrieval-corpus currency or any model's score.
How this benchmark is scored
| Category | retrieval |
|---|---|
| Maximum score | 100 average Recall@100 |
| Direction | Higher is better |
Primary source: https://arxiv.org/abs/2007.15207
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to MKQA Leaderboard.
Frequently asked questions
What does the MKQA benchmark measure?
Cross-lingual question retrieval benchmark built from Multilingual Knowledge Questions & Answers across 26 languages.
What is the methodology?
Aggregate Recall@100 for multilingual query-to-English-passage retrieval; model records must preserve the reported dense, sparse, or hybrid configuration.
Which AI model is the current MKQA leader?
BGE-M3 (BAAI) currently leads with 75.5 average Recall@100.