AI pre-review helped conference reviewers, but did not replace them

An arXiv experience report from the Bioinformatics Open Source Conference describes using AI to pre-review software submissions before human reviewers made decisions. The goal was to help volunteers handle a surge of submissions, not to automate acceptance.

The BOSC team built a system that checked six review criteria, including whether code was public, properly licensed and runnable. A tool called Runabilly built and tested projects inside disposable Docker containers so reviewers could see evidence about reusability and safety.

The authors report that most surveyed reviewers who responded found the pre-review useful, but preferred to check the AI’s conclusions rather than accept them unquestioningly. That is an important design lesson for AI-assisted peer review.

The useful role was evidence gathering and triage. The risky role would be letting a model make final judgments about scientific or community value. For conferences facing submission overload, the paper suggests AI can reduce reviewer burden if human authority remains explicit.