Apple researchers have proposed a way to turn failed AI-agent attempts into training data instead of discarding them. PROOF-Gen examines each unsuccessful tool-use trace, generates targeted corrective guidance and asks the teacher model to try that scenario again.
The guidance is removed before the result enters the training set, leaving the smaller student model with a clean successful demonstration rather than a task-specific hint. This targets a weakness in common “generate and filter” pipelines, where passing runs are kept and failures provide no learning signal.
On the tau2-bench tool-use benchmark, 57% of teacher trials failed, and the researchers classified about two-thirds of those failures as near misses. PROOF-Gen recovered passing trajectories for 93% of failed scenarios. Training with the expanded set raised Qwen3-4B-Instruct-2507’s reported pass-at-one score from 0.132 to 0.529. A Gemma model gained 7.2 percentage points on a separate multi-turn benchmark.
Apple also reports smaller gains in a deployed pipeline and on an on-device model. Those figures come from the research team’s own evaluations; the method’s cost and benefit will depend on the teacher, evaluator and task distribution used in another deployment.