A new benchmark called MERIT tests whether long-term memory changes what a tool-using AI agent actually does. Most existing memory benchmarks ask models to recall facts from conversation history; MERIT instead uses instrumented tasks whose successful completion depends on information learned in an earlier episode.
The benchmark includes three task domains, a difficulty ladder that ends with changed facts, controlled corruption of stored memories and explicit metering for every memory operation. An automated leak check is intended to confirm that the needed answer is not available elsewhere in the task environment.
Across 23,440 scored episodes, the researchers report that memory raised success on dependent tasks from zero to between 55% and 100%, depending on the setup. Updated facts were more difficult. Embedding-based retrieval ranged from 30% to 95% across models and runs, and agents acted on a correctly retrieved value only 55% of the time in that condition. The experiment cost $42.57 in model usage.
The results suggest that storing information is not enough: systems must retrieve the right version and then use it consistently. MERIT is a new preprint, so broader model coverage and independent replication will be needed before treating its figures as general performance estimates.