A new study argues that evaluating an AI agent only from its final response can overlook serious mistakes made along the way. The researchers built trajectory-judge to score the sequence of tool calls and decisions behind an answer, not just whether the visible outcome looks correct.
The testbed is a deterministic support-desk environment with a scripted policy that always completes the task correctly. Researchers then inject one controlled fault into an otherwise valid run. Because the environment and intended policy are known, they can determine exactly where the agent deviated and measure whether an evaluator notices it.
This setup targets a common production shortcut: giving a judge the original request and final response while hiding the process. An agent may still reach the right result after using the wrong record, violating a rule or taking an unnecessary action. The paper is a preprint and its controlled support setting is narrower than messy real applications, but it offers a reproducible way to test whether process-aware judges catch faults that outcome-only scores systematically miss.