Test RAG Tenant Filters Against Cross-Account Leakage

A prompt instruction is not an access-control boundary

Telling a model not to reveal another customer's information does not prevent a retrieval layer from supplying that information in context. Authorization belongs in the application and data query. If a chunk is outside the caller's scope, it must not enter the candidate set sent to the model.

Create two test tenants with deliberately similar documents and unique canary phrases. Authenticate as tenant A, ask directly for tenant B's canary, and inspect the retrieved IDs as well as the answer. The expected result is that no B chunk is returned at any stage, including reranking and fallback search.

Exercise every retrieval path

Apply the same access filter to vector, lexical, cache, and reranker paths. Test missing tenant metadata, malformed claims, empty permissions, and a filter service failure. The safe behavior for an authorization uncertainty is to fail closed, not run an unfiltered search.

Check caches too. A response or retrieval cache key should include the relevant identity and authorization scope, or contain only results proven safe for all callers. A cache that returns a prior user's answer can bypass otherwise correct database filters.

Log the effective principal scope and filter version without storing unnecessary private content. Alert on results whose tenant does not match the request. Treat this as a security regression suite and run it whenever ingestion schema, search backend, or authorization logic changes. Prompt-level refusal is a second line of defense; it cannot compensate for evidence that should never have been retrieved.

Test every retrieval path adversarially

Create two tenants with similar documents and unique canary phrases. Authenticate as each tenant and ask directly for the other tenant's canary, then inspect raw candidate IDs, reranker inputs, cache hits, and final citations. The expected result is no cross-tenant record at every stage, not merely a refusal in the final text. Include a document with missing tenant metadata and a request with an empty or malformed principal scope.

Apply the authorization predicate before vector ranking, lexical search, hybrid fusion, reranking, and context caching. A filter added only to the primary vector query can be bypassed by a fallback path or a cached response. Cache keys must include relevant identity and permission scope, or the cache must contain only results safe for every caller. Test cache hits separately from misses.

Fail closed when the authorization service or claim parser is unavailable. Do not substitute an unfiltered query to preserve availability. Log principal scope, filter version, and result tenant IDs with privacy controls. Alert on a mismatch even if the model did not reveal the content; retrieval of forbidden evidence is already a security incident.

Run the suite when changing ingestion metadata, search backend, reranker, cache, or authorization middleware. Keep canary content synthetic and rotate it when needed. A secure RAG system treats the model as an untrusted consumer of evidence, not as the component responsible for deciding who may see it.

Add a test for a cache key created by tenant A and read by tenant B, plus a test where a document's permissions change after indexing. The expected result is no stale authorized answer. Run the suite on cache hits, misses, vector search, lexical search, and fallback paths, logging only stable IDs, principal scope, and policy versions. A permissions regression should fail closed and alert even when no answer was generated.

Include a malformed principal, an expired role, and a backend timeout. The safe behavior for each is no unauthorized evidence. Review cache invalidation when a tenant is deleted or a document changes ownership. Permission tests belong in release gates because a polished answer cannot compensate for a retrieval boundary failure.

Use synthetic canary phrases that cannot be mistaken for real customer data and rotate them with the test fixture. Test both direct retrieval and an indirect question that could surface the phrase through summarization. Inspect candidate IDs before generation; waiting for the model to leak the canary is too late to detect a retrieval authorization defect.

Exercise response and retrieval caches with the same adversarial identities. A correct backend filter can still be bypassed by a cached answer keyed only by query text. Include authorization scope in cache design and invalidate entries when permissions or document ownership changes. Alerting should cover near misses as well as confirmed answer leakage.