The software wrapped around a coding agent can influence benchmark results more than a change to its reinforcement-learning recipe, according to a new repository-level study. Researchers ran 24,000 sealed evaluations using tasks and records from Aider, OpenHands, Qwen Code and SWE-agent.
Starting from the same Qwen3-8B supervised model, they replayed frozen task-and-harness records with the same number of updates. One training rule calculated relative advantages separately for each task-harness pair, while the other pooled multiple harnesses for the same task. Every checkpoint was then tested with a sealed SWE-bench Verified oracle on four source harnesses and one minimal harness withheld from training.
Changing the evaluation harness moved the mean solve rate from 2.14 percent to 9.27 percent, a factor of 4.3. The training rule moved it by 1.16 times. On the held-out harness, the pooled rule’s advantage was only 0.25 percentage points, with a confidence interval spanning a disadvantage to a small benefit; results also varied by training seed.
The findings are specific to one base model and experimental setup, and the preprint does not show that training recipes never matter. It does show that coding-agent reports need to specify and vary the harness, because a score can reflect tool orchestration and execution behavior as much as the underlying policy.