LongFact Concepts Leaderboard

What this benchmark measures

• long-form factual accuracy
• factual precision in open-domain generation

Relevant to:

• content generation risk screening
• factuality comparison across models

Failure modes it exercises:

• unsupported factual claims in long-form answers

Method and its limits

Long-form prompts across 38 topics, scored by SAFE, which splits a response into individual facts and checks each one via multi-step reasoning over Google Search results, aggregated as an extended F1 at a target response length.

• The grader is itself a language model, and the paper reports 72% agreement with crowdsourced human annotators.
• Recall depends on a hyperparameter for preferred response length, so scores are not comparable across different length settings.
• Search-based verification inherits the coverage and ranking behaviour of the search engine at evaluation time.

Dataset

Dataset

LongFact

Type

model-generated long-form factuality prompts

Freshness

aging

Benchmark release

2024-03-27

38 topics as registered in the primary paper. The prompt set was generated with GPT-4, which the paper states directly.

How to read this score

Precision is the share of supported facts and is directly interpretable. The aggregate F1 depends on the chosen length hyperparameter and should not be compared across settings.

Similarity to real tasks: Medium — Established from the primary methodology: LongFact prompts elicit long-form answers across 38 topics, and responses are decomposed into individual facts that are checked against Google Search results. Long-form generation with per-claim checking resembles real writing tasks more closely than multiple-choice formats. It does not reproduce an edited publishing workflow, because no human review, source selection, or citation requirement is imposed.

Data contamination risk: Medium — Prompts are public, and because grading is search-based rather than answer-key based, direct memorisation is a weaker concern than for fixed-answer benchmarks. No run-specific audit is registered.

Benchmark gaming risk: Medium — Because recall is defined against a preferred response length, response-length tuning changes the score without changing factual quality.

What you still need to test yourself

• Re-check on your own domain, where search coverage and terminology differ.
• Decide your own precision and recall trade-off rather than inheriting the paper's length setting.

This benchmark supports decisions about:

• Compare factual precision in long-form generation across candidate models.
• Estimate unsupported-claim rates before using a model for drafted content.

Limitations

• An automated grader agreeing with humans 72% of the time is not a verification of factuality.
• Not evidence of citation quality, since the benchmark does not require sources in the response.

Primary source reviewed 2026-10-11. LongFact is treated as an aging 2024 benchmark.

What this evidence does not cover

• Only two models are currently scored on this page, and the response-length setting behind those scores is not registered.

Sources

• Long-form factuality in large language models Wei, Yang, Song, Lu, Hu, Huang, Tran, Peng, Liu, Huang, Du, Le · accessed 2026-10-11Supports the 38-topic prompt set, the SAFE grading procedure, the 72% human-agreement figure, and the extended F1 metric; it does not support any model score shown here.
• LongFact prompt set and SAFE evaluation code Google DeepMind · accessed 2026-10-25The authors' released code for the paper, described as the original code for "Long-form factuality in large language models". It establishes the prompt construction and the automatic evaluation procedure; it does not establish any model's score or the response-length setting behind the scores shown here.

How this benchmark is scored

Primary source: https://arxiv.org/abs/2403.18802

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to LongFact Concepts Leaderboard.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path