Reasoning Jury uses model consensus to evaluate long reasoning traces

A new arXiv paper proposes Reasoning Jury, a multi-model method for evaluating long reasoning traces from large language models. The authors argue that single-model judges, even strong ones, often miss defects in complex reasoning.

The problem matters for training and evaluation. Reasoning models produce long intermediate traces, and developers need ways to identify mistakes for data curation, reinforcement learning signals and runtime feedback. If the judge is unreliable, the training signal can reward flawed reasoning.

Reasoning Jury uses consensus across multiple models to surface defects rather than relying on one frontier judge. The paper also notes a practical constraint: some providers’ terms restrict using frontier models during online training, so evaluation designs need to account for policy as well as accuracy.

The approach does not remove the need for ground truth or human review in high-stakes settings. It does show that reasoning evaluation is becoming a systems problem, combining model judgments, rules and deployment constraints.