LayerRAG-Bench tests where agentic RAG systems actually fail

A new arXiv paper introduces LayerRAG-Bench, a benchmark for testing reliability failures in agentic retrieval-augmented generation systems. Retrieval-augmented generation, or RAG, lets AI systems answer using external documents and tools, but the benchmark shows that grounded-looking answers can still fail for different reasons.

The authors test eight enterprise domains, 240 tasks, nine fault scenarios and nine models from OpenAI, Anthropic and Gemini. They separate problems across layers such as evidence freshness, tool contracts, authorization and session state.

One result is especially practical: schema normalization helped with schema drift, but did not fix stale evidence, missing tool output, denied permissions or wrong-session context. Groundedness-only evaluation also produced false positives when the underlying evidence was stale or tied to the wrong session.

The lesson is that RAG reliability needs layer-specific testing. A system should get credit for fixing the layer it actually repairs, but teams should not mistake one improvement for a universal reliability solution.