A new preprint argues that measuring AI productivity only by elapsed time can reward workers for checking less, even when more defects escape. The author proposes a “ledger” approach that records both time saved and additional errors, then calculates the cost per defect at which AI assistance stops being worthwhile.
The paper reanalyzes public data from METR’s developer study. It says the same 246 issues support estimated slowdowns of 19% or 24%, depending on the statistical quantity being measured, and 34% without adjustment for a chance imbalance. It also finds that the share of issues helped by AI could range from 3% to 82% under different assumptions. These wide ranges illustrate how a headline average can hide uncertainty and task variation.
In a small preregistered experiment, AI reviewers checked AI-written briefs based on recent SEC filings. When told the writer’s true error rate, reviewers increased checking as errors rose. When told the writer erred ten times less often than it did, they checked seven fewer of nine figures and allowed an incorrect figure through in 22.5% of briefs, versus none under accurate information. The study is a preprint and its small experiment should not be generalized without replication.