A new arXiv paper titled “Same Question, Different Answers: Evaluating LLM Reliability Beyond Accuracy” argues that measuring only whether a model gets an answer right can miss an important failure mode: inconsistency. The paper focuses on how large language models respond when the same underlying question is asked in different ways.
That distinction matters for real applications. A model that answers correctly once but changes its response when wording shifts slightly may be hard to trust in education, customer support, compliance, or decision-support settings. Reliability is not just a score on one prompt; it is the ability to preserve the same reasoning target across variations.
The paper is research-stage work, so it should not be read as a product benchmark. Its value is in pushing evaluation toward behavior that users actually experience.
As AI systems are embedded into workflows, consistency becomes a practical safety and usability issue. People need to know not only whether a model can be right, but how easily that correctness changes under ordinary rephrasing.