A new benchmark called PhysMent tests whether language models can investigate a physical system rather than answer from a fully specified word problem. Models interact with a MuJoCo simulator by applying forces, inspecting object states, advancing time and changing scene geometry before giving an answer.
The benchmark contains 105 classical-mechanics scenes across easy and hard, single- and multi-concept regimes. It also includes standard scenes, object creation, hidden objects and a category requiring scene manipulation. The researchers assess performance through six scoring dimensions.
Across seven tested models, overall accuracy ranged from 25% to 67%. Systems handled some qualitative single-concept questions reasonably well, reaching up to 80%, but most scored below 30% on the hardest quantitative single-concept category. The authors trace failures to premature answers, inefficient exploration and poor use of simulator feedback.
That pattern suggests the bottleneck is often procedural: deciding which experiment to run, adapting after each observation and combining the results accurately. A model may understand the relevant physics concept yet still fail to gather the necessary evidence. PhysMent is described in a new arXiv preprint and its results have not been peer reviewed, but the interactive setup offers a more demanding test than static science question answering.