Answer Relevance — AI Glossary

Reference details

Related terms

• Faithfulness
• Context Recall
• RAGAS

How it works

1. The metric asks a narrow question: does the response address the question that was asked. Not whether it is true, not whether it is grounded — only whether it is on target.
2. It is computed by inversion. Candidate questions are generated from the answer, each is embedded, and their similarity to the original question is averaged, so an answer that implies a different question scores low.
3. That construction makes it penalise two distinct faults with one number: answering an adjacent question, and padding a correct answer with material that implies questions nobody asked.
4. It requires no ground-truth answer, which is why it is usable on production traffic rather than only on a labelled evaluation set.
5. It is deliberately blind to correctness. A fluent, entirely false response that addresses the question directly scores well, which is the division of labour with the faithfulness metric that accompanies it.
6. So it belongs in a suite rather than alone, and the suite's metrics must be read together because each is constructed to ignore what the others measure.

Example: Why four orthogonal metrics do not combine into one number

A RAG evaluation suite typically reports four metrics — whether the answer is grounded in the retrieved context, whether it addresses the question, whether retrieval found the relevant material, and whether what it found was precise. Suppose a system scores 0.9 on each, which sounds like a strong result. If the four failures are roughly independent, the fraction of responses that clear a 0.9-equivalent bar on all four at once is 0.9^4 = 0.6561, so about 1 - 0.656 ≈ 34% of responses fail at least one dimension. At 0.95 each it is 0.95^4 = 0.8145, so 18.5% still fail somewhere. The arithmetic is the same shape as per-prompt constraint adherence, and the conclusion is the same: a per-metric average is not the user's experience, because a user meets a whole response and a response fails if any dimension fails. It also means averaging the four into a single score is actively misleading — a system at 1.0, 1.0, 1.0, 0.6 and one at 0.9 across the board both average 0.9, while the first has one catastrophic dimension and the second has none.

Commonly confused with

Answer relevance is not faithfulness, and the pair is constructed so that each is blind to the other's failure. Faithfulness asks whether every claim traces to the retrieved context, so it catches invention and ignores whether the answer was on topic. Relevance asks whether the response addresses the question, so it catches drift and ignores whether the content is true. A response can score 1.0 on faithfulness and near zero on relevance by accurately summarising a passage that has nothing to do with the question — which is a common RAG failure and one that a single composite score would hide.

When to use it

Reach for it when:

• Monitoring production RAG traffic, where no ground-truth answer exists and this metric does not need one.
• Detecting drift and padding, which are the two faults its construction targets.
• Alongside faithfulness and the retrieval metrics, read separately rather than averaged.
• As a regression signal on prompt or retrieval changes, where a drop localises the problem to the generation step.

Reach for something else when:

• As a correctness measure, which it is explicitly constructed to ignore.
• Averaged with the other metrics into one score, which hides a single catastrophic dimension behind three good ones.
• On questions with no well-defined answer, where the generated-question inversion has nothing stable to compare against.
• As the only metric, since a fluent, confident falsehood that addresses the question scores well.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Answer Relevance — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path