The Allen Institute for AI has released BenchMIRT, a method for showing which abilities individual questions in an AI benchmark actually measure. Rather than treating every task as evidence for one headline capability, it estimates question difficulty and how strongly each item separates models along several underlying dimensions.
Researchers trained the method on results from 100 language models across 16 benchmarks and more than 34,000 questions. Without being told the intended categories, BenchMIRT repeatedly recovered two dominant dimensions: safety and general reasoning. Some familiar labels did not align neatly. BBQ, commonly used to assess social bias, tracked more strongly with reasoning, while different subsets of HarmBench measured different combinations of safety and reasoning.
BenchMIRT draws on multidimensional item response theory, a psychometric approach used to infer abilities from patterns of answers. The team says selecting the most informative 10% of questions generally preserved nearly the same model rankings on the inferred dimensions, suggesting a path to cheaper evaluations. The result does not prove that existing benchmarks are invalid; it shows why a single average can hide what drives the score and should be interpreted with item-level evidence.