Test RAG Abstention With Questions the Corpus Cannot Answer
A retrieval system must recognize when evidence is absent, not merely return the nearest chunk. Build unanswerable cases and define a safe abstention outcome.
Nearest neighbor does not mean sufficient evidence
Vector databases return the closest available chunks even when every result is irrelevant. If the application always asks the model to answer, an unrelated policy or old ticket can become a confident fabrication. The system needs a tested way to say that evidence is insufficient.
Create unanswerable questions based on realistic user wording: ask for a policy the corpus does not contain, a person's private record outside the caller's permissions, or a future fact that has not been published. Include near-miss documents with similar vocabulary so the test is not trivially solved by an empty index.
Define the abstention signal
Possible signals include a calibrated reranker score, a retrieval threshold, evidence coverage, or a model-level groundedness check. None is universally reliable. Measure false answers and unnecessary refusals separately. Raising a threshold may reduce unsupported answers but frustrate users whose evidence is phrased differently.
Make the response useful when abstaining: state that the available sources do not establish the answer, identify what information is missing, and offer a search refinement or escalation path. Do not expose documents that the caller cannot access just to explain the refusal.
Track abstention rate by query type and evaluate sampled outputs with human review. A low refusal rate is not inherently good, and a high rate is not proof of safety. The target is appropriate abstention on missing or unauthorized evidence while retaining answers that are well supported. Keep these cases in the regression set whenever retrieval, prompts, or model versions change.
Calibrate the threshold against evidence
A score from a vector search or reranker is not automatically a probability that the answer is supported. Collect labeled examples of answerable questions with relevant evidence, unanswerable questions, ambiguous questions, and questions blocked by permissions. Compare the score distributions and choose a threshold on a validation set. Keep a separate test set for measuring whether the threshold generalizes to new topics and document types.
Abstention can happen at several stages. The retriever may return no authorized candidates, the reranker may find only weak matches, or an answer checker may find that a claim lacks support. Record which stage made the decision. A single “no answer” metric hides whether ingestion, search, permissions, or generation is responsible for the user experience.
Give the user a useful recovery path. Ask for a document, date, identifier, or narrower question when that would improve retrieval. Do not reveal that a forbidden document exists merely by saying it was withheld. For permission failures, return the same safe behavior as an absent source unless the product has an explicit authorized explanation. Test prompt injection attempts that try to force an answer despite weak evidence; the threshold must be enforced outside the model's instructions.
Review threshold performance after corpus and model changes. A new embedding model can shift scores without changing evidence quality, so thresholds need recalibration rather than copied constants. Keep false-confidence incidents as high-priority evaluation examples and track the cost of unnecessary abstention separately. The right operating point depends on the harm of an unsupported answer versus the cost of asking the user for more information.
Keep separate thresholds for workflows with different risk. A brainstorming assistant can tolerate a cautious suggestion, while policy, financial, or account-access answers may require stronger evidence and human escalation. Document the threshold owner and recalibration trigger. Never let a prompt instruction alone override an external abstention decision.
Include threshold decisions in release evaluations with a fixed set of intentionally unanswerable questions. Review false confidence more heavily than verbosity or style. A score shift after changing the embedding model is a reason to recalibrate, not to copy the old numeric threshold. Keep the model's prose separate from the external decision that evidence is insufficient.
Include index version and freshness policy in cache keys and traces. A rollback should restore a known index intentionally, not leave a mixture of old retrieval and new metadata. Run a date-specific test after every ingestion change so current and historical questions remain distinguishable.
Define how overlapping effective windows are handled and who resolves an ambiguous version. If no deterministic rule exists, route the item for review or abstain. Do not use file modification time as a substitute for policy dates; reformatting a document can change it without changing the policy.
For a policy question that names a date, carry that date into the structured retrieval filter and answer trace. A current document may be irrelevant even when its wording is identical. Include effective dates in the citation display when they explain why one version was selected. This makes freshness behavior understandable to both users and reviewers.
When a document is deleted or access is revoked, propagate that event to the active index and caches. A correct freshness field does not help if an old chunk remains searchable. Measure deletion and permission-propagation delay and alert for records that remain active beyond the policy window.