New RAG metric tests whether models use context or prior knowledge

A new arXiv paper argues that common RAG evaluations can miss a key failure mode: models may appear grounded while actually relying on prior knowledge. The authors introduce Normalized Context Utilization, a token-probability-based metric designed to measure contextual information gain across zero-shot, oracle, and adversarial settings.

The results challenge a simple scaling story for factual extraction. In the paper’s tests, efficient smaller language models can match or outperform larger architectures when the task is strict extraction rather than broad reasoning.

For teams building RAG systems, the work is a reminder to test whether retrieval is being used, not just whether the final answer looks correct.