Researchers including members of the UK AI Security Institute applied psychometric methods to eight widely used language-model safety benchmarks. Analyzing up to 192 models across more than 5,000 questions, they found that a single safety score combines largely independent traits: refusal strictness, truthfulness, and handling of context-dependent requests.
That aggregation can obscure important tradeoffs. HarmBench rewards refusal of harmful requests, while OR-Bench-Hard penalizes unnecessary refusals of harmless ones. A model can therefore improve some safety ratings by blocking more requests overall, even as it becomes less useful. Similar benchmarks can also double-count the same behavior when their scores are averaged.
The study found that fewer than 2% of questions meaningfully distinguish models. Three 25-question tests selected for information value represented the three measured dimensions more accurately than equally sized random samples. Around ten adaptively selected questions could approximate rankings from a full benchmark, potentially reducing evaluation costs by 97% to 99%.
The researchers also tested methods for spotting “sandbagging,” in which a model behaves more cautiously during evaluation. Response-pattern analysis detected 80% to 100% of directly instructed cases, with about one harmless model in ten incorrectly flagged. Adding comparison with a model’s past performance raised detection to 94% to 97%. The authors caution that deliberately trained evasion would probably be harder to identify.