Researchers have released JudgeArena, an open-source framework for evaluating language models with other language models as judges. The project unifies benchmarks including AlpacaEval, Arena-Hard, MT-Bench, and m-Arena-Hard under a common interface.
LLM-as-a-judge testing is popular because it is faster and cheaper than large human studies, but many benchmarks hardcode a particular judge model or protocol. That makes it difficult to know whether a ranking reflects model quality or the evaluation setup.
JudgeArena allows swappable judge models through systems such as vLLM, llama.cpp, and OpenRouter, while logging metadata needed for reproducibility. The authors also report tuned open-model judge configurations that can match or outperform closed-model judges on human preference datasets.
The work does not eliminate the need for human evaluation. It does make automated judging easier to inspect, compare, and reproduce, which is increasingly important as leaderboards influence model choices.