Your AI Knowledge Base Can Be Poisoned Without Anyone Touching the Model
Reduce RAG poisoning and stale-answer risks with source allowlists, provenance, version control, ingestion review, and deletion tests.
The model may be innocent
When a RAG assistant gives a dangerous or misleading answer, the underlying model may be behaving exactly as instructed with the evidence it received. The knowledge base could contain a malicious upload, a compromised connector, an outdated procedure, or duplicate content that outranks the authoritative source. Teams often focus security reviews on prompts and model settings while treating ingestion as a routine data job.
That creates a quiet attack path. An employee account is compromised, a public page is edited, or an import pipeline accepts a document from an unexpected source. The content enters the index, gets retrieved for a plausible question, and appears to the model as trusted context. No one needs to alter model weights to influence the product's answers.
The right defense starts with source governance. Decide which systems may contribute knowledge, who can publish into them, how changes are approved, and how quickly content is removed. A vector database is an index, not an authority registry.
Preserve provenance at ingestion
For every source, store origin, owner, ingestion timestamp, source version, effective date, checksum, and access scope. Keep the relationship between a chunk and its original document. If a connector cannot provide a stable identity or permission signal, quarantine its content until you have a safe policy.
Use allowlists for connectors and content types. Validate file size, parsing output, MIME type, and metadata. Scan uploads with existing security controls. Review unusual bulk changes and sudden increases in document volume. Prevent an untrusted user from making a document appear authoritative merely by adding words such as “official policy” to its title.
Prefer source-system permissions and signatures where available. For critical procedures, maintain an explicit owner and approval state. Separate draft from published content. Retrieval should exclude drafts unless a user is explicitly working in an authorized review flow.
Manage updates, duplicates, and deletion
Indexing should be version-aware. When a source changes, mark old chunks superseded and ensure they no longer compete with the current version. If two sources repeat a policy, record which one is authoritative and how conflicts are resolved. A ranking model cannot reliably infer your organizational source hierarchy from wording alone.
Deletion must propagate across derived data: chunks, embeddings, summaries, caches, evaluation snapshots, and any stored answer. Define a revocation service-level objective and test it. A deleted source that remains retrievable for days is still present from the user's perspective.
Track ingestion lag and failed syncs. If a connector stops updating, indicate that freshness is uncertain rather than silently treating old records as current. Use alerts for unexpected permission changes, missing owner metadata, and documents that fail parsing or validation.
Detect poisoning and low-quality content
Monitor for repeated instruction-like strings, suspicious external links, hidden text, unusually large files, and documents that suddenly appear across many queries. These signals are not proof of an attack; they are useful triage clues. Keep a way to quarantine a source quickly without rebuilding the entire platform.
Run retrieval tests against known canary documents. Ask questions whose correct answers distinguish trusted from untrusted sources. Include a deliberately misleading document with a high keyword overlap and verify that the source policy or ranking strategy excludes it. Test whether a prompt injection inside a document can cause unauthorized tool use, not only whether it changes a summary.
Review source diffs for high-impact collections. Require a second approver for changes to policy, security, legal, or financial content. Keep a tamper-evident audit record of who published or deleted a source and when.
Build a response for contaminated knowledge
If poisoning is suspected, disable retrieval from the affected source, invalidate derived caches, identify answers generated from its chunks, and notify owners according to the incident plan. Preserve enough evidence to investigate without spreading the malicious content further. Reindex from a known-good source version and rerun regression tests before restoring access.
Tell users when the knowledge base is incomplete or under review. Do not let the assistant fill the gap with confident guesses. For critical questions, provide a path to the responsible source owner.
Good model behavior cannot repair a knowledge supply chain that has no provenance or revocation. Treat documents as inputs with trust levels, build ingestion and deletion controls, and test retrieval with adversarial content. The model may be the most visible part of an AI system, but the quietest risk often enters before the first token is generated.