RAGAS — AI Glossary

Reference details

Related terms

• Faithfulness
• Answer Relevance
• Context Recall

How it works

1. Each metric decomposes its input into small units and asks the judge a narrow question about each, rather than requesting one holistic quality score.
2. Faithfulness splits the generated answer into individual claims and asks whether each is entailed by the retrieved context, scoring the proportion supported.
3. Answer relevance works backwards: the judge generates questions that the answer would answer, embeds them, and compares them to the original query — so a fluent answer to a different question scores poorly.
4. Context precision judges each retrieved chunk for relevance and weights the verdicts by rank, which is what makes it sensitive to ordering rather than only to membership.
5. Context recall requires the ground-truth answer and measures how much of it the retrieval covered, which is why it cannot run where no reference exists.
6. Aggregating many binary judgements rather than one graded opinion is what gives the scores their stability, and it is also the limit: every unit is still a judge's verdict, so the metric moves when the judge model changes.

Example: Reference-free, what that costs, and what the judge bill is

Faithfulness asks whether each claim in the answer is supported by the retrieved context — checkable without a gold answer, which is why it can run on live traffic. Context recall cannot: it needs ground truth. So the metrics split naturally into those you can monitor continuously and those you can only run against a labelled set, and of the four core measures exactly two fall on each side — half the suite is available on production traffic and half is not. The volume is the part to plan for, because every metric is itself model calls. Scoring one answer across all four typically costs around a dozen judge calls once claims are decomposed and checked individually. Sampling 1% of 100,000 daily answers is 1,000 evaluations, so 1,000 x 12 = 12,000 judge calls a day purely for measurement. That is a real line item, and it is the reason the sampling rate is an engineering decision rather than a default.

Commonly confused with

RAGAS scores are produced by an LLM judge, not measured. They inherit the judge's biases and its failure modes, and they move when the judge model changes. Treat them as a consistent instrument for tracking direction, not as ground truth about quality. It is also not a retrieval evaluation on its own: two of its four measures describe the generator and two the retriever, so reporting an average of them produces a number that cannot attribute a regression to either stage.

When to use it

Reach for it when:

• You need automated RAG evaluation in CI or on sampled live traffic
• Separating retrieval failures from generation failures systematically
• You have validated the judge against human labels at least once
• With the judge model and version recorded, so a time series survives a provider update

Reach for something else when:

• As the sole signal — an unvalidated judge can be confidently wrong at scale
• Comparing scores across different judge models or versions
• Where the domain is specialised and the judge lacks the expertise to grade it
• At a sampling rate nobody costed, given the 12,000 judge calls a day above

Primary source

• Es et al., Ragas: Automated Evaluation of Retrieval Augmented Generation (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to RAGAS — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path