Web agents gain less from routing than noisy benchmarks suggest

A web-agent study compares six ways to represent a browser, including text, pixels, and combined views, across VisualWebArena and WebArena tasks. The authors find the modes are complementary: each solves tasks the others miss, and the best choice changes across sites and models.

At first glance, that seems to make per-task routing highly valuable. The paper argues much of the apparent gain is inflated by run-to-run noise, because rerunning the same mode can change 12 to 14 percent of outcomes.

The practical takeaway is a cost bound rather than a magic router. Developers may save compute by sending only unsolved tasks to additional observation modes, but benchmark variance makes it hard to learn a perfect task-level selector.