Giving an AI agent more conversation history generally improved its assigned scores, but the benefit diminished as the context window grew, according to a new study using 1,100 BEAM questions.
The researchers kept the retrieval ranking and reader model fixed while testing five passage budgets. Answerable questions usually scored better with more context, while pooled abstention scores sometimes increased when the budget shrank. That suggests shorter histories can make a system more cautious, but not necessarily more correct.
A separate controlled test compared retaining user messages with keeping an equally sized slice of the most recent conversation. Qwen matched the full conversational state more often for direct requirements when user messages were preserved. Its performance still fell when an answer depended on references to earlier offers, and Mistral produced near-floor results across the tested retention conditions.
The paper does not establish a universal memory setting. Its own scoring audit leaves the correctness of BEAM answers unresolved, so the reported gains are changes in assigned scores rather than proof of better answers. The practical lesson is to preserve the evidence a task needs, then validate both the memory policy and the metric used to judge it.