A new arXiv paper tackles one of the harder problems in language-model reliability: getting models to notice and fix their own mistakes. The authors introduce Self-Fix Step-DPO, or SFS-DPO, a two-stage reinforcement learning framework for step-level self-verification and self-correction.
The first stage strengthens step-level reasoning through preference optimization. The second stage explicitly trains models to verify and correct their own answers. The paper also introduces SFS-DPO-R, a teacher-assisted variant that adds explanatory rationales for error verification to provide stronger corrective signals.
Across in-domain and out-of-domain tests on multiple language models, the authors report that SFS-DPO and SFS-DPO-R outperform earlier step-level training baselines. Their analysis also finds improvements in how often models attempt self-correction and how effective those corrections are.
The work matters because many current models can produce plausible explanations for wrong answers, and simply asking them to check themselves is often unreliable. Training correction at the reasoning-step level may give developers a more targeted way to improve robustness, though the claims still need reproduction outside the paper's evaluation setup.