METR has expanded its influential AI “time horizon” evaluation, adding longer and more open-ended tasks. The nonprofit’s earlier chart found that the length of software tasks frontier models could complete doubled about every four months, but the new work shows that this trend does not transfer cleanly to jobs requiring judgment.
In one new test, models designed AI evaluations for METR researchers to assess. None was considered useful enough for actual research. Models also trailed people when scoring philosophical arguments. These tasks are harder to train and measure because success cannot be checked automatically as it can in coding or mathematics.
The newest models are improving: some reached METR’s “borderline above the bar” category for evaluation design, compared with clearly inadequate results a year ago. Yet the sample of judgment-heavy tasks remains small, and estimated human completion times replace direct measurements for the longest jobs. The update therefore supports rapid progress on verifiable work without proving that autonomous research is advancing at the same rate.