Evaluate RAG Retrieval Quality Before Changing the Model

Start with evidence, not model upgrades

When a retrieval-augmented generation system produces a weak answer, teams often change the model first. That is an expensive guess. The answer may be wrong because the right document was never retrieved, because irrelevant chunks crowded it out, because the document was stale, or because the model ignored evidence that was present.

Create a small evaluation set before touching the pipeline. Twenty to fifty real questions are enough to expose the shape of the problem if they include easy, ambiguous, unanswerable, and permission-sensitive cases. For each question, record the relevant document or chunk IDs, the IDs actually returned, whether the answer is grounded, and whether the answer includes the expected citation.

Measure retrieval separately

For one question, retrieval recall asks: did the system return the evidence that a reviewer marked relevant? A simple form is:

recall = relevant retrieved items / all relevant items
precision = relevant retrieved items / all retrieved items

These are not perfect measures, but they separate two common failures. Low recall means chunking, indexing, filters, or query rewriting may be hiding the answer. Low precision means the top-k result is noisy and the answer model has too much irrelevant context to sort through.

Use the same top_k, metadata filters, and reranker settings that production uses. Do not evaluate retrieval with a larger context window than the application actually provides. That produces a comforting score that the user will never see.

Then inspect answer behavior

A retrieved answer can still fail. Mark whether every material claim is supported by the retrieved text, whether a citation points to the supporting chunk, and whether the system says it does not know when no evidence is available. Keep these labels separate. Citation presence is not faithfulness, and faithfulness is not completeness.

Include adversarial rows such as:

• a question whose answer changed between two policy versions
• a question that requires joining two documents
• a question with a plausible but unauthorized document
• an unanswerable question that should produce a refusal

Change one layer at a time

If recall is poor, inspect document extraction, chunk boundaries, metadata, and query normalization before choosing a new model. If recall is healthy but precision is poor, test smaller chunks, metadata filters, or reranking. If both are healthy but answers are unsupported, inspect the prompt, citation format, and refusal behavior.

Run the evaluation again after every change and keep the old scores. The point is not to find one magical number. The point is to know which layer moved, which queries regressed, and whether a local improvement made the overall system less trustworthy.

The RAG Retrieval Evaluation Workbook calculates a directional first pass from pasted evaluation rows. It does not replace human review or a production monitoring set.