TestHallVQA is a new benchmark for vision-language models that combines difficult visual examination questions with document-scale, multi-image context. It is designed to test whether a model can locate relevant evidence while ignoring extra pages and visual tokens.
Existing visual question-answering tests often isolate one challenge. Some use long documents but require limited reasoning, while others demand deeper visual reasoning in a clean single-page setting. TestHallVQA combines both and can inject controlled levels of redundant context.
The researchers also propose a metric called F1-R² that jointly scores computational reasoning and evidence-retrieval robustness as irrelevant material increases. Their experiments with mainstream vision-language models found weaknesses across several dimensions, though the abstract does not provide a single headline accuracy figure.
The practical issue extends beyond exams. Systems that read reports, manuals or collections of scanned documents may receive far more visual material than a question requires, and additional context can reduce rather than improve performance. The authors have released the dataset, code and theoretical derivations. The work is a new arXiv preprint, so its benchmark design and conclusions still need independent scrutiny.