Choose RAG `top_k` With a Context and Latency Budget
Increasing top-k can raise evidence recall but also add noise, token cost, and latency. Select the candidate count from query-level evaluation instead of habit.
More retrieved text is not automatically more evidence
Teams often raise top_k when an answer is incomplete. That can recover a missing document, but it can also bury the relevant chunk in near-duplicates, consume the model's context budget, and increase generation latency. A larger context window does not remove those tradeoffs.
Run the same evaluation set at several candidate values, such as 3, 5, 8, and 12. For each query, record whether the required evidence appears, how much irrelevant material enters the prompt, whether citations remain precise, and the total prompt tokens. Use the production reranker and filters in the test.
Allocate the budget by query shape
Simple fact lookup may need only a few chunks. A comparison across two policies may need more, while an identifier query may need one exact passage. Consider a candidate pool followed by reranking, then pass only a smaller final set to the model. Make sure the reranker sees enough candidates and that its cost fits the latency budget.
Measure p50 and p95 retrieval plus generation latency, not only the mean. A configuration that improves average recall but makes tail latency unacceptable may be a product regression. Test long chunks and duplicated versions because they can consume many tokens while adding little new information.
Choose the smallest top-k that meets the evidence and answer-quality target on important query classes. Keep a fallback for low-confidence retrieval rather than dumping more context indiscriminately. Re-run the evaluation whenever chunking, reranking, or model context behavior changes, since the best top-k depends on the whole pipeline.
Budget candidates and final context separately
The retrieval candidate pool and the model context are different controls. A reranker may need twelve candidates to find the best three, while the generator should receive only the evidence that fits the answer's complexity. Record both numbers in evaluations. If a query needs several independent facts, select coverage across sources rather than three near-duplicate chunks from one document.
Measure token cost before and after packing. Include prompt instructions, source labels, citation metadata, and the actual chunk text. A configuration that fits the average request can still exceed the budget on long-tail documents, so report p95 and the truncation rate. Truncation should be explicit and observable, not a silent string slice that removes the evidence or citation marker.
Latency should be attributed by stage: vector search, lexical search, filter evaluation, reranking, prompt construction, and generation. This shows whether increasing top_k actually costs retrieval time or mainly inflates generation. Set a product limit for tail latency and define the response when the budget is exceeded. A concise abstention with a retry or narrowing suggestion is usually safer than sending an oversized, noisy context.
Keep the evaluation set versioned with the top-k decision. When chunk size, embedding model, reranker, or model context limit changes, rerun the same queries and compare evidence recall, answer support, citation precision, latency, and cost. The correct value is not a universal number; it is the smallest setting that meets the workflow's evidence requirement under its budget.
Also test repeated or near-duplicate chunks. They can consume the context while giving the model no additional evidence. Use source diversity or coverage rules when appropriate, but verify that diversity never removes the only authoritative passage. Keep low-confidence behavior explicit so more candidates are not used as a substitute for a missing answer.
Set a hard candidate and context budget per endpoint. A long-tail request should degrade predictably, perhaps by returning fewer sources or asking for a narrower question. Do not let a user-controlled query force an unbounded retrieval fan-out. Record the budget decision so quality and cost regressions can be reviewed together.
Evaluate the same setting on short, long, multi-part, and ambiguous questions. The best value for a fact lookup may be too small for a comparison and too large for a simple answer. Keep endpoint-specific defaults explicit rather than hiding a single global number in a shared helper.
Compare the cost of a larger candidate pool with the cost of a reranker. If a reranker is used, make its output carry the original stable IDs and scores. Do not let a model-generated citation label become the only source identity; labels are presentation, while IDs must be validated by the application.