Researchers have proposed a metric for a part of AI agent behavior that success rates often hide: whether an agent behaves consistently across different tasks.
The Behavioral Consistency Metric trains a model to predict task success from features of agent execution traces, then compares feature-attribution patterns across trajectories. In experiments covering roughly 9,000 trajectories from six language-model agents on software engineering tasks, the authors find that within-task consistency and cross-task consistency can diverge. Some agents may behave reproducibly on similar tasks while changing strategy sharply across the broader workload.
That distinction matters for teams deciding whether an agent is dependable enough for production. A high success rate can mask brittle or unpredictable process choices, while consistent behavior can make failures easier to diagnose. The metric does not replace outcome evaluation, but it adds a lens for understanding how an agent gets to its answer.