A new arXiv paper proposes a broader way to test hallucinations in multimodal large language models. The authors introduce UniHall, a dataset organized around object, instruction, and knowledge failures, and SAMF, a self-adaptive fuzzing method that mutates inputs to stress-test models.
The goal is to move beyond static benchmarks that models can quickly saturate. Multimodal systems interpret images, text, and instructions together, so failures can appear when small changes create misleading visual or contextual cues. Fuzzing, a testing method that searches for edge cases through repeated mutations, can reveal weaknesses that fixed test sets miss.
The paper reports that state-of-the-art multimodal models degrade under these dynamic stress tests compared with conventional evaluation. That matters for applications where visual understanding affects safety, accessibility, medicine, or operations. The framework is still an academic proposal, but it points toward a more realistic evaluation standard: testing not only whether a model answers known examples correctly, but how robustly it behaves when the world is messy and inputs evolve.