Large language models are not yet dependable judges of whether a scientific idea is genuinely new, according to a new Ai2 study. Across six models, changing only a few words in an evaluation prompt shifted accuracy by more than 50 percentage points in one experiment.
The researchers tested GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. More reasoning time and literature retrieval did not consistently improve their decisions. Two systems built specifically for novelty evaluation also performed worse than a straightforward Claude Opus 4.6 prompt.
To build the benchmark, the team mined ICLR 2026 peer reviews for explicit comments about originality. It selected 154 ideas from highly rated accepted papers whose novelty reviewers supported, and 145 from low-rated rejected papers whose originality reviewers questioned. A second test paired the 154 validated human ideas with basic AI-generated proposals. Those generated examples carry weaker labels because the researchers assumed, rather than individually verified, that they were less novel.
Ai2 has released the paper, data, code and a pipeline for constructing more evaluation sets. The practical warning is clear: research systems should not treat one model’s novelty verdict as ground truth. Prompt wording, comparison format and dataset composition can dominate the answer, so consequential judgments still need literature review and independent human scrutiny.