Web-agent study questions whether skill and memory modules justify their token cost

A new arXiv paper asks whether online skill and memory modules are always worth their token cost for web agents. The authors compare augmented agents with a token-matched vanilla baseline across WebArena domains and several models.

The finding is that extra modules can improve performance, but their test-time overhead is often underreported. Under a fixed inference budget, a simpler baseline using more actor steps can match or beat the augmented systems.

The study is a useful warning for agent evaluation: success rates should be reported alongside total token cost, especially when memory and workflow modules run on every task.