RAG safety paper limits who can read untrusted documents

A new arXiv paper proposes a more selective way to protect retrieval-augmented generation systems from poisoned documents.

Retrieval-augmented generation, or RAG, lets a language model answer with the help of outside documents. The weakness is that a malicious or wrong document can still influence the final answer even when the model notices the document is unreliable.

The authors build on the “Cordon Principle,” which keeps the final answer writer away from raw evidence. Their refinement is narrower: only agents capable of deliberative “System 2” reasoning should be allowed to inspect untrusted documents. They also introduce metrics for measuring the gap between detecting misinformation and being influenced by it.

The result is a security design aimed at reducing overhead without giving every model component direct access to risky text. The paper is research rather than a deployed product, but it targets a practical problem for companies that connect chatbots to web pages, knowledge bases, and user-uploaded files.