Datadog has added Online Experiments to its Agent Observability platform, allowing teams to split real production traffic between agent versions and compare both answer quality and operating behavior. Feature flags can limit a new model, prompt, retrieval method or tool configuration to a small share of users.

The system places evaluator scores beside latency, token use, estimated cost and workflow errors. Engineers can then open individual traces to inspect tool calls, arguments and returned data. The goal is to catch trade-offs hidden by a single quality score, such as a cheaper model that retries more often or a retrieval change that improves evidence but slows every request.

In one Datadog example, a newer model reduced estimated cost by 44.3% while quality scores remained close at 3.84 and 3.86 out of five. It also appeared to take longer and produce more workflow errors, although confidence intervals were too wide to establish a regression. Traces showed more invalid tool parameters followed by successful retries, giving the team a specific issue to fix.

Production experiments do not replace offline safety tests, and exposing users to variants requires careful limits for consequential workflows. Their value is in finding interactions a fixed dataset missed. Datadog’s approach feeds those failures back into the regression suite, connecting controlled rollout, operational evidence and future pre-deployment testing.