The best model tested on Taste-Bench selected the better long-term path in fewer than 60% of questions, according to researchers from Microsoft and City University of Hong Kong. The benchmark measures an agent’s ability to choose productive intermediate steps, rather than simply checking whether it eventually finishes a task.
Taste-Bench contains 502 “decision forks” drawn from recorded runs on software engineering and AI research tasks. At each fork, a model sees the work completed so far and must choose between two possible next steps while the later outcome remains hidden. Researchers label the better option using hindsight from completed trajectories, including independent attempts that diverged and runs where an agent corrected an earlier mistake.
The team tested 14 models from the Claude, GPT, Grok, DeepSeek, GLM, MiniMax and Mistral families. Each question appeared twice with the options reversed, and a model received credit only by answering both versions correctly. Under that rule, random guessing would score 25%. Human reviewers confirmed nearly all automatically generated labels, though the dataset is limited to coding and AI-research traces. The findings suggest that final success rates can conceal costly detours: an agent may complete a job while repeatedly choosing approaches that waste time or compute before recovering.