Add Hybrid Search to RAG When Exact Product Terms Matter

Semantic similarity can miss exact identifiers

Vector search is useful when a user asks about a concept using different words from the source. It can be weaker on ticket numbers, error codes, package names, model IDs, and uncommon product labels. A query for ERR_AUTH_721 should not depend on a nearby semantic phrase being embedded similarly.

Hybrid retrieval combines a lexical signal such as BM25 with dense-vector similarity. First classify a sample of real queries: conceptual questions, exact identifiers, quoted phrases, and mixed requests. Measure whether lexical results recover evidence that vector search misses before operating both paths in production.

Merge rankings deliberately

Raw BM25 scores and vector distances are often on different scales. Do not add them directly without calibration. Reciprocal-rank fusion is a simple starting point because it combines positions rather than assuming score comparability. Deduplicate by stable chunk ID and preserve which retriever found each result for debugging.

Evaluate top-k recall and precision separately for each query type. Hybrid search can improve exact-term recall while adding irrelevant keyword matches to conceptual questions. A reranker may help, but it adds latency and cannot recover a document neither retriever found.

Keep authorization filters identical across both retrieval paths. If one path omits tenant or access scope, fusion can reintroduce a forbidden chunk after a secure vector query. Log source, rank, and filter version without exposing document text. Hybrid search is a ranking design, not a default checkbox: keep it only when an evaluation set demonstrates a useful tradeoff.

Classify queries before choosing the blend

Build an evaluation table with exact identifiers, natural-language questions, quoted phrases, spelling variants, and conceptual requests. Mark the evidence that should be retrieved for each row and note whether lexical matching or semantic similarity is expected to help. A single aggregate recall number can hide the fact that hybrid search fixes error codes but damages broad discovery queries. Report results by query class and rank cutoff.

Reciprocal-rank fusion is useful because it avoids pretending that BM25 scores and vector distances share a scale. It still has parameters: the rank constant, each retriever's candidate depth, and the final cutoff. Tune these on a held-out set rather than on the same examples used to choose the configuration. Preserve contributing ranks in evaluation output so a reviewer can understand why a result moved.

Deduplication deserves explicit tests. The same source passage may be indexed as a dense chunk and a lexical document with different internal IDs. Deduplicate by a stable source-chunk identity, then retain provenance for both retrievers. If a reranker consumes merged candidates, ensure its input preserves the authorization filter and source version. A better rank is not useful if it loses the evidence location needed for citation.

Measure operational cost alongside quality. Hybrid retrieval can double backend calls, increase candidate transfer, and add reranking latency. Compare p50 and p95 latency, empty-result rate, and context size with the quality gain on the query classes that matter. Keep a fallback only when its behavior is defined and secure; an unfiltered lexical fallback is a data-leak path, not a reliability feature.

Keep the evaluation set versioned and include the query class in every result. Re-run it after changing analyzers, tokenization, embeddings, rerankers, or access filters. A hybrid system is justified by measured gains on the queries users actually ask, not by the assumption that two rankings must be better than one.

Use a held-out set when tuning fusion weights and report exact-term recall separately from conceptual relevance. Preserve the originating rank and retriever in traces. This lets the team explain a result change and remove hybrid retrieval if its extra latency does not buy measurable evidence coverage.

Include analyzer changes in the evaluation because tokenization, stemming, and case handling can change identifier recall. Keep lexical and dense candidates tagged until the final citation is assembled. A fusion improvement is real only when the final answer uses better authorized evidence, not merely when a ranking score moves.

Keep a small set of exact identifiers that must be found lexically and a set of conceptual questions that should benefit from semantic matching. Test quoted phrases and spelling variation separately. This prevents tuning a blend that wins on one easy category while making the most important operational queries worse.