A new arXiv paper introduces AgentOdyssey, a benchmark designed for agents that keep learning during long-horizon deployment. The framework procedurally generates open-ended text games with rich entities, world dynamics, and tasks that require extended planning.
Unlike benchmarks that assume learning stops at test time, AgentOdyssey interleaves learning and inference throughout the agent’s run. It measures game progress as well as world knowledge, retention, and planning behavior.
The benchmark is relevant because real agents often operate in environments where they must adapt continuously rather than solve a static prompt.