A new arXiv paper introduces MemArena, a benchmark for personal memory assistants that run on devices and handle private conversation histories. The benchmark uses a simulator to create a single-world setting with 50 agents over 15 days.
The goal is to test memory systems under conditions closer to real personal assistants: many interactions, an ego-centered point of view, and coherent events across sessions. The authors report 10.3 million dialog-text tokens and 24,100 ego-observed text-only tokens per agent per day.
Their evaluation found that the memory backend mattered more for content accuracy than simply scaling the reader model in some settings. It also found that permission-aware access failed across tested systems, with some leaking information and others refusing too much.
The benchmark is not a deployed assistant, but it highlights a central challenge for local AI tools. Remembering useful context is only valuable if the system can also respect boundaries and retrieve information quickly enough to be practical.