Wearable assistants may remember long first-person video more reliably by turning it into text than by repeatedly sending frames to a vision model. CapMem, a new benchmark, tests that idea with 75 videos totaling 33.7 hours and 1,000 human-written questions across 16 scenarios.
Current vision-language models can process only a limited number of frames, and visual tokens become expensive as recordings grow. Long contexts also make it harder to retrieve a small event. CapMem instead creates captions at regular intervals and treats that text as a reusable episodic memory that can be searched when a question arrives.
For videos longer than 20 minutes, captions generated every 30 seconds beat direct video question answering for 10 of 12 models. Sixty-second windows won for eight of 12. A matched-frame control across six Qwen models still showed average gains of 3.22 and 2.55 percentage points, respectively. A separate retrieve-and-verify workflow improved accuracy by as much as 5.3 points.
Captions discard visual detail and can introduce errors of their own, so they are not a complete replacement for source footage. They offer a cheaper index that can identify likely moments before a model inspects the video. That pattern could help wearable assistants answer “when did I last see this?” without keeping every frame inside an active model context.