A new arXiv paper studies how to make LLM-as-a-judge systems less vulnerable to bias. These systems use language models to evaluate other model outputs, a practice that is increasingly common because human evaluation is expensive and slow.
The paper, titled “Chain-of-Models,” argues that judge models can be affected by cognitive biases and that simple prompt-based debiasing is brittle across different bias types. Its proposed approach uses cross-model auditing, where multiple models help check one another’s judgments.
The work matters because automated evaluation is becoming part of model development, benchmark reporting, and product testing. If the judge has systematic blind spots, teams may optimize models toward scores that look objective but reflect the evaluator’s bias.
This is early research, not a ready-made replacement for human review. Its useful warning is that AI evaluation infrastructure needs its own validation, especially when model judgments are used to compare systems or make deployment decisions.