Your RAG Answers Are Wrong? Stop Blaming the Model and Inspect Retrieval First
Trace a weak RAG answer through parsing, indexing, filtering, ranking, context assembly, and generation with a practical diagnostic workflow.
A fluent answer can hide a broken pipeline
When a retrieval-augmented generation system gives an incorrect answer, teams often switch models or rewrite the prompt. That may help, but it can also leave the root cause untouched. The system may have extracted a table incorrectly, indexed an old policy, filtered out the right tenant, ranked a near-match above the source, or passed only half a relevant section to the model. A better generator cannot reliably recover evidence it never received.
RAG is a chain of transformations. Each stage can lose meaning, freshness, permissions, or provenance. Debugging only the final text is like debugging a web request by staring at the rendered page and ignoring the database and network. The practical task is to make each transformation observable.
Before changing anything, capture one failed query and its exact production context: user scope, query rewrite, filters, retrieved document IDs, scores, assembled prompt sections, model version, and final answer. Redact personal data. Without this trace, a reproduction may be testing a different system.
Follow the evidence from source to answer
First inspect the original source. Did ingestion capture the correct version? Were headings, page numbers, table columns, footnotes, and access labels retained? Compare extracted text with the visible document. OCR can quietly turn a minus sign into a dash, merge columns, or omit a warning that changes a policy. If source parsing is wrong, embedding experiments will optimize a corrupted representation.
Next inspect chunk boundaries and metadata. Does the relevant exception remain with the rule it qualifies? Are version, owner, effective date, and access scope attached to every chunk? A chunk without its section path can be difficult to interpret. A chunk carrying the wrong tenant ID can become a serious data leak.
Then inspect retrieval. Ask whether the needed evidence appears anywhere in the candidate set before reranking. If it does not, investigate query formulation, filters, embedding compatibility, and index freshness. If it appears but ranks too low, evaluate reranking and hybrid lexical search. If it appears at the top but the answer is still wrong, inspect context assembly, truncation, conflicting sources, and generation instructions.
Record a retrieval trace with the raw query, rewritten query, filter expression, document IDs, version metadata, ranking scores, and final order. Scores from different models are not directly comparable; use labeled outcomes to determine whether a score threshold is meaningful.
Build a diagnostic evaluation set
Gather real questions from support tickets, search logs, and domain experts. For each query, label the authoritative evidence, acceptable answer, relevant version, and whether the system should abstain. Include exact-name questions, paraphrases, ambiguous requests, multi-document reasoning, table lookups, outdated sources, and no-answer cases.
Measure retrieval separately from generation. Useful retrieval measures include whether relevant evidence appears in the top results, ranking quality, and whether the returned content is sufficient to answer. Generation measures include factual support, completeness, citation accuracy, and appropriate abstention. A single “answer quality” score makes it hard to tell where a regression came from.
Review errors by class. Missing evidence calls for indexing or retrieval work. Wrong evidence may indicate ranking or permissions. Correct evidence with a wrong claim suggests prompt, synthesis, or model behavior. Correct answer with a false citation suggests the model is not reliably connecting claims to sources. Keep every instructive failure as a regression case.
Fix the stage that failed
If parsing is weak, improve extraction and preserve structure before changing chunk size. If freshness is the issue, add version-aware indexing, deletion handling, and a clear reindex process. If filters exclude the answer, test filter logic against identity and tenant cases. If lexical names matter, add keyword retrieval or exact-match indexes. If chunks lose context, experiment with semantic boundaries and limited overlap using the same evaluation set.
Context assembly deserves deliberate design. Deduplicate near-identical results, retain source labels, avoid burying a decisive exception, and set token limits per section. Put evidence in a stable format that the model can distinguish. Ask the model to state when the sources conflict or do not support an answer, and verify that behavior with examples rather than assuming the instruction is obeyed.
Deploy changes without losing the baseline
Change one stage at a time. Freeze the query set, model, prompt, and filter while comparing a retrieval change. Store index and pipeline versions with each trace. Run offline evaluation, then shadow the new pipeline on real queries before switching production traffic. Watch latency, empty retrieval rate, citation coverage, answer corrections, and access-control failures.
RAG quality rarely comes from a single magic chunk size or one premium model. It comes from preserving trustworthy evidence at every stage and knowing where a failure entered. When an answer is wrong, inspect the evidence path first. That turns guesswork into a sequence of testable engineering decisions.