A new benchmark called UPHELD tests language models against long, human-written conversations rather than short questions or synthetic dialogue. Professional scriptwriters produced hundreds of complete exchanges containing more than 30,000 turns, accompanied by over 36,000 per-turn human annotations.
The dataset is designed to measure qualities that matter over an extended interaction, not only factual correctness. The researchers evaluated traditional automatic metrics and reference-free “LLM as a judge” methods, then compared their scores with judgments from human experts.
Common approaches proved unreliable in that comparison, according to the paper. The team used the benchmark to build a mixture-of-judges system that combines multiple evaluation signals and improved correlation with human assessments by about 30%.
UPHELD remains a research dataset, and stronger correlation does not mean an automated score fully captures whether a conversation is helpful, safe, or appropriate for every user. Its human-authored references are nevertheless important because many dialogue evaluations use AI-generated examples to judge other AI output. The findings suggest developers should validate conversational metrics against sustained human interactions before relying on them to select or tune models.