AWS has proposed a metric for evaluating agents across multi-turn conversations rather than scoring only the final outcome. The Agent Evaluation Metric, or AEM, is designed to identify the first turn that caused a failure and distinguish it from later turns that simply inherited incorrect state.
A single wrong tool argument can contaminate an entire workflow. For example, an agent asked to refine a sales report may use profit instead of revenue early on, causing every later calculation to be wrong. A task-level pass or fail reveals the bad result but not the point that needs repair.
AEM instead decomposes quality into named sub-metrics evaluated at each turn with the complete trajectory in view. The initial implementation measures truthfulness, whether values and statements match expectations, and completeness, whether required fields and information are present. Tool and action selection form a structural layer underneath those scores.
AWS presents the design as extensible to dimensions such as safety and instruction retention without replacing the evaluation mechanism. It is a methodology rather than a guarantee of objective scoring: teams still need high-quality expected values, representative trajectories and careful judge calibration before the resulting metric can guide production decisions.