Language models can solve a numeric expression such as “2+5” while stumbling when the same problem is written in words. A new interpretability study finds that this brittleness can be predicted by whether the model reuses the same internal computation across those surface formats.
Researchers used attribution patching to independently locate the circuits recruited for numeric arithmetic and for verbal problems in English, Spanish and Italian. They then measured how much each verbal circuit overlapped with the model’s own numeric circuit. Greater overlap tracked stronger transfer from digits to number words.
The relationship held at three levels: it accounted for the relative difficulty of the three verbal formats, differences in which models generalized best, and whether individual questions were answered correctly. Its predictive performance rivaled supervised probes while requiring no labeled training data for a separate classifier.
The work is an arXiv preprint focused on arithmetic and three languages, so it does not establish that circuit overlap will predict every kind of reasoning transfer. It offers a useful diagnostic, however: benchmark accuracy on one prompt format can hide a different internal strategy on another. Inspecting whether representations converge may reveal fragile capabilities before a model is deployed on varied real-world language.