IBM Research has added consistency analysis to ALTK-Evolve, a system that turns an AI agent’s previous execution traces into reusable operating guidelines. The goal is to reduce cases where an agent completes a task in one run but fails the identical task the next time.
The researchers distinguish average success, called Mean@k, from Pass^k: the share of tasks completed successfully in every one of k attempts. On the AppWorld benchmark, a ReAct agent using GPT-4.1 achieved 77.4% Mean@5, but only 53% Pass^5. That 24.4-point gap would be hidden by a conventional average.
Their Consistency Analyzer replays individual decision steps with controlled resampling to locate unstable choices. These are points where the model assigns similar probabilities to multiple next actions, so small numerical changes can alter a tool call or argument. The issue can persist at temperature zero because hosted inference may still shift near-tied probabilities slightly.
ALTK-Evolve then writes guidelines aimed at the unstable steps and injects them during inference. The authors frame consistency as separate from raw capability: a model can often know how to solve a task without doing so reliably on every run. That distinction matters most in workflows such as contract checks or financial reconciliation, where occasional success is not enough.