High-confidence AI errors may be more stable than expected

An arXiv paper studies a practical failure mode for language models: the confident wrong answer that stays stable under small changes. The author calls this stable miscalibration, distinguishing it from errors that disappear when a prompt is perturbed.

The work combines an output-level audit score with an internal sensitivity probe that measures hidden-state movement. On a multi-domain binary factual audit set, the score helps identify domains where abstention-aware self-critique reduces decision loss. Internally, self-critical prompting consistently reduces hidden-state sensitivity across layers in three open-weight models.

For users, the useful takeaway is that confidence and consistency are not the same as correctness. A model can be reliably wrong, and a response that survives small prompt changes may still need external verification. For developers, the paper suggests that audits should separate unstable mistakes from stable overconfidence because the two may require different mitigations.