Tool-using AI agents can lose roughly 70% of their clean-context capability by the sixth step of a task because earlier mistakes corrupt later decisions, according to a study of five open-weight models. The researchers measured both selection of the correct tool and construction of its arguments.
Tests covered contamination-free tasks with chains from one to eight steps. Models were evaluated with a clean, teacher-provided history and with their own free-running history. The widening difference quantified how much failure came from accumulated mistakes rather than the current step alone.
The paper also identifies a flaw in exact-match evaluation against one fixed “gold” trajectory. Once an agent diverges, later correct values may depend on tool outputs it never received. Under that scoring rule, recovery can appear impossible by construction, producing confident but misleading estimates of failure severity.
The authors propose conditional-on-state scoring, which judges the next action using the state the agent actually reached. Applied to cached outputs, it produced more informative estimates without new model calls. The work does not establish reliability for proprietary models or every tool environment, but it shows that multi-step agent tests need to measure both compounding errors and legitimate recovery paths.