A new Apple research benchmark evaluates a video caption by testing whether it contains enough correct information to answer questions about the clip. The approach, called CapQuiz, is intended to avoid a basic weakness in caption scoring: two accurate descriptions can use different words or focus on different valid details.
CapQuiz creates fine-grained, multiple-choice questions from each video and has people verify them. A candidate caption is then judged by how useful it is for answering those questions, without requiring it to resemble one reference sentence. The benchmark covers 10 question types, split between descriptive and inferential tasks, across 24 video domains.
The researchers also introduce CapF1, a combined score built from CapP, which measures factuality, and CapR, which measures coverage of important visual information. Separating the two matters because a caption can mention many details while inventing some of them, or remain accurate by saying very little.
In the reported experiments, CapQuiz correlated more closely with human judgments than existing caption metrics and exposed more specific strengths and weaknesses. The method evaluates captions rather than generating them, and its value will depend on the quality and breadth of the verified questions. It offers developers a more interpretable test for comparing visual-language models.