Language models may be able to learn when to say “I’m not sure” without a labeled dataset of correct and incorrect answers. A new study uses each model’s own probability for its answer as the training signal for abstention.
The researchers froze a model’s confidence estimates, then used low-rank fine-tuning to make it answer when confidence was high and abstain when it was low. They tested six open-weight models from two families, ranging from one billion to eight billion parameters, on short factual question answering.
At matched answer coverage, the label-free method showed no statistically detectable difference from abstention training that used correctness labels. A control experiment that trained models harder on difficult examples did not help, suggesting the improvement came from calibration rather than memorizing more facts.
The method has a clear blind spot: a model cannot catch facts it states incorrectly with high confidence. Correctness was also judged by a separate model, and the study covered short answers rather than long reasoning or tool use. Internal doubt could therefore reduce some unsupported answers cheaply, but it cannot serve as a complete factuality system on its own.