Microsoft and Hugging Face turned deterministic database checks into reinforcement-learning rewards for a business-task agent, improving both average success and repeatability on a held-out benchmark.
The team fine-tuned Qwen3.8-27B for 50 steps on tasks where a correct final database state produces a binary reward. On ThinkingBox-Bench’s 507 tasks, average success across 20 attempts rose from 51.70 percent to 60.78 percent. Tasks completed correctly in every attempt increased from 38 to 107. However, tasks solved at least once fell from 453 to 446, showing a tradeoff between consistent behavior and breadth.
Training focused on tasks the base model solved sometimes. Tasks it always passed or always failed produced no contrast for the GRPO reinforcement-learning method. The selected pool contained 187 tasks and remained separate from the public benchmark, although both sets share the same framework and similar task style.
The test checks resulting records and side effects rather than trusting an agent’s claim that work is complete. That makes it useful for workflows such as bookings, refunds and support cases. The reported serving-cost comparison excludes training and data-selection costs, and all benchmark tasks are synthetic. The stronger conclusion is about evaluation design: production agents need repeated trials and durable state checks, not a single successful run.