Measure Retrieval Recall and Grounding — Tutorial

Goal

By the end of this tutorial you will be able to answer three separate questions about a retrieval-augmented system:

1. Did retrieval find the information needed to answer the question? 2. Did the generated answer stay within the evidence that was retrieved? 3. When the answer was wrong, which of those two steps caused it?

Most teams measure only end-to-end answer quality. That tells you something failed, but not what to fix. A low score can mean the right passage never reached the prompt, or that it did and the model ignored it. Those need opposite fixes — better retrieval versus better prompting, grounding constraints, or a different model — so measuring them together wastes effort.

Prerequisites

You should have a working retrieval pipeline (a corpus, an embedding or search index, and a generation step) and be able to log what it retrieves.

Concepts used throughout, each defined in the ModelRefs glossary: grounding, context recall, context precision, reranking, citation accuracy, chunking, embedding model, and representative workload. You do not need to have measured any of them before.

You also need a corpus you can inspect. If you cannot open the documents and confirm what the correct answer should be, you cannot label evidence, and without labels there is nothing to measure against.

Architecture: the evaluation loop

The loop under test is short, and each arrow is a place a measurement can be taken:

representative queries → expected evidence (labelled) → retrieval → optional reranking → answer generation → grounding and citation checks → failure analysis

The important property is that retrieval is observable on its own. You capture what came back and at what rank before generation runs, so retrieval quality can be scored without involving the model at all. Generation is then scored against the evidence that was actually supplied, not against the corpus as a whole.

This separation is the whole point of the setup. Keep the retrieved set for every query — you will need it to attribute failures later.

Step 1. Define the workload and build a query set

Start from the questions the system will really be asked, not from questions that are easy to score.

Sample from real traffic where you have it, or write queries with the people who will use the system. Aim for a mix that reflects production: straightforward lookups, questions that need evidence from more than one document, ambiguous phrasings, questions using internal jargon, and — critically — questions the corpus genuinely cannot answer.

That last group is not optional. A system that answers everything confidently looks excellent on a set where every question is answerable, and fails the moment a user asks something out of scope. Include unanswerable questions so you can measure whether the system declines.

Freeze the set once you start comparing configurations. If the questions change between runs, the comparison is meaningless.

Step 2. Label the evidence each query needs

For every query, record which passages would let a careful person answer it. This is your relevance label set, and it is the part teams most often skip.

Label at the granularity you retrieve at. If you retrieve chunks, label chunks; labelling whole documents while retrieving passages will make recall look better than it is.

Expect this to be the slow step, and expect disagreement. Have two people label a sample independently and compare. Where they disagree, the query is usually ambiguous, which is itself a finding worth keeping. Write down the rule you settled on so later labellers apply the same standard.

Keep a note of queries where no passage is relevant. Those are your abstention cases.

Step 3. Measure retrieval on its own

Run retrieval for every query and capture the returned chunks with their ranks, before any generation.

The core question is coverage: of the passages you labelled relevant, how many appeared in the retrieved set? That is context recall, and it caps everything downstream — evidence that never arrives cannot be used, no matter how good the model is.

The complementary question is how much of what came back was actually relevant, which is context precision. Low precision does not block a correct answer the way low recall does, but it fills the context window with distractors, raises cost, and gives the model more opportunity to cite the wrong thing.

Look at rank, not just presence. Evidence retrieved at position 20 when you pass the top 5 to the model is, in practice, not retrieved.

Step 4. Inspect grounding

Now score the generated answer against the evidence that was actually supplied to it.

For each claim in the answer, decide whether the retrieved context supports it, partially supports it, or does not support it at all. An answer built entirely from supplied evidence is grounded; one that introduces facts the context does not contain is not, regardless of whether those facts happen to be true.

That distinction matters and is easy to lose. A model can produce a correct answer from its training data while the retrieval system failed completely. Scored end-to-end, that looks like success. Scored as grounding, it is a warning that the system is not actually using your corpus.

Where the interface shows citations, check them separately: does the cited passage support the specific claim attached to it? Citation accuracy is about attribution, and an answer can be correct and grounded while still pointing at the wrong passage.

Step 5. Attribute the failure

For every unsatisfactory answer, assign a cause using what you captured in the previous steps:

- Relevant evidence was not retrieved → a retrieval problem. Look at chunking, the embedding model, query rewriting, hybrid search, or filters. - Relevant evidence was retrieved but ranked below the cutoff → a ranking problem. Look at reranking or a larger top-k. - Relevant evidence was supplied and ignored or contradicted → a generation problem. Look at the prompt, the grounding instructions, context ordering, or the model. - Nothing relevant exists in the corpus and the system answered anyway → an abstention problem. The system needs to be able to say it does not know.

Keep the failing cases. A short, well-understood set of failures is more useful for the next iteration than an aggregate score, and it becomes the regression set you re-run after each change.

Decision points

Several choices change your numbers, so decide them deliberately and hold them fixed while comparing:

Chunk size and overlap trade recall against precision. Larger chunks tend to capture more context per hit and retrieve fewer distinct facts; smaller chunks do the reverse. There is no size that is correct in general — it depends on your documents.

Top-k sets how much evidence reaches the model. Raising it usually improves recall and costs precision, latency, and tokens.

Reranking re-orders an over-fetched candidate set, which often helps when recall is acceptable but the right passage sits too low. It adds a step and latency, so it should earn its place.

Metadata filters and hybrid retrieval help when pure vector similarity misses exact identifiers, product codes, or rare terms.

Retrieval mode is its own decision when your embedding model supports more than one. Some open-weight retrieval models expose dense, sparse, and multi-vector representations from a single artifact — BGE-M3 is one published example. Each additional mode is a separate pipeline to measure: dense-only, sparse-only, hybrid, and multi-vector will not score alike on the same corpus, and the extra modes add index size, latency, and operational complexity. Evaluate them separately rather than enabling everything by default, and treat provider-reported multilingual results as a starting hypothesis about your own corpus and languages, not as evidence about them.

Relevance labelling and human review are themselves decisions: who labels, against what rule, and how disagreement is resolved.

Change one of these at a time. Changing several at once tells you the result moved without telling you why.

What each measure does and does not tell you

These four are related and not interchangeable. Confusing them is the most common reporting error in RAG evaluation:

Context recall — did the needed evidence reach the prompt? Bounds everything downstream. Says nothing about the answer.

Grounding — did the answer stay within the supplied evidence? Says nothing about whether the evidence was right, or whether the answer is factually true.

Citation accuracy — does each cited passage support the claim attached to it? An answer can be correct and grounded and still cite badly.

Answer correctness — is the final answer right? Can be satisfied for the wrong reasons, including by model memory when retrieval failed.

Report them separately. A single blended score hides exactly the information you need to act on.

Resist fixed universal targets. What counts as acceptable recall depends on your corpus, question mix, and the cost of a miss; a number that is fine for internal search may be unacceptable in a regulated workflow.

Failure modes

- Relevant evidence is never retrieved, so the ceiling is set before the model runs. - Distractors rank above the correct passage and crowd it out of the cutoff. - Correct evidence is supplied and the model answers from memory instead. - The answer adds claims the context does not support. - Citations point at plausible but unsupporting passages. - The evaluation set is not representative — too easy, too clean, or missing the unanswerable cases. - Labels are inconsistent, so the metric moves when the labeller changes rather than when the system does.

Risks and limitations

Relevance labels are subjective. Two reasonable people will disagree on borderline passages, so treat small differences between configurations as noise unless you have checked agreement.

Small evaluation sets move a lot. A handful of queries can swing a percentage substantially, which makes it easy to "improve" a system by chance.

Corpora drift. Documents are added, edited, and removed, so a frozen evaluation set slowly stops describing the live system and needs periodic refresh.

Query leakage is a real risk when the evaluation set was built from the same examples used to tune chunking or prompts — results will look better than production.

Results are domain-dependent and do not transfer. A configuration that works on policy documents may not work on code or support transcripts.

Finally, none of this proves production readiness on its own. It measures the retrieval and grounding behaviour of a pipeline on a fixed set, not live user satisfaction, cost, or safety.

What to compare next

Once retrieval and grounding are measured separately, the natural next comparisons are:

- Chunking and top-k variants on the same frozen set. - With and without reranking, watching latency alongside recall. - Two generation models on identical retrieved context, which isolates grounding behaviour from retrieval. - The same pipeline on a representative workload, so quality sits next to cost per successful task and tail latency before a production decision.

The companion tutorial on creating representative workload evaluations covers that last step, and the ModelRefs guides on evaluating model quality and choosing a model for RAG cover the selection decisions that follow.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Measure Retrieval Recall and Grounding — Tutorial.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path