
Datadog is extending its Agent Observability platform with Online Experiments, giving engineering teams a way to compare AI agent variants against real production traffic instead of relying only on predefined evaluation datasets. The update combines agent scoring with operational metrics such as latency, token consumption, estimated cost and errors, aiming to show not just whether an agent passed a test, but whether it performs better in live conditions.
Why Datadog is adding production experimentation
The new capability is meant to address a familiar problem in AI agent development: a change that looks strong in a controlled evaluation can behave very differently when exposed to unpredictable user requests. Datadog’s approach brings controlled experimentation, production telemetry and execution traces into one workflow so teams can assess both outcomes and the behavior behind them.
Offline evaluations still matter. Teams can run different versions against predefined datasets, score responses and catch known regressions before deployment. But those tests are only as useful as the datasets and scoring criteria behind them, and they can miss new workflows, unexpected requests and changing tool responses.
How Datadog’s Online Experiments work
With Online Experiments, teams can compare a control version of an agent against a modified variant by randomly splitting production traffic between them. That reduces the impact of different user requests and creates a more realistic picture of how the system behaves under actual workloads.
Datadog also integrates its Agent Observability evaluators with its Experiments capability. That means evaluation scores can be viewed alongside conventional operational metrics, helping teams judge response quality, accuracy and completeness while also tracking execution duration, token consumption, estimated cost and workflow errors.
- Traffic allocation: Feature Flags are used to expose changes to a small percentage of users before a wider rollout.
- Quality metrics: Teams can measure response quality, accuracy and completeness.
- Operational metrics: Latency, token usage, estimated cost and workflow errors are tracked at the same time.
- Trace visibility: Experiment results connect back to individual agent traces for deeper inspection.
Looking beyond a single score
Datadog says the main advantage of this approach is that improvements rarely come without trade-offs. A cheaper model may need more retries, while a revised retrieval strategy may improve evidence quality but increase execution time. Looking only at a quality score can hide those differences.
By linking results to traces, engineers can inspect tool calls, supplied arguments and returned results to understand why a variant behaved differently. That makes it easier to spot issues that might not affect the overall score but still matter in production.
Example results from the GPT-5.6 Luna and GPT-6 Luna comparison
According to the article, Datadog used the system to compare GPT-5.6 Luna and GPT-6 Luna, splitting traffic evenly between the two. An LLM-based evaluator scored investigation quality from one to five, considering accuracy, completeness and whether conclusions were supported by the available evidence.
The reported results showed a 44.3% reduction in estimated cost for the newer model, while quality scores stayed close at 3.84 and 3.86. That suggests the newer version was able to cut cost without a major drop in the measured quality of the output.
What the traces revealed
The operational picture was more mixed. Datadog said the newer model appeared to increase execution duration and workflow errors, although the confidence intervals were too broad to confirm a clear regression. The traces also showed that the newer model supplied invalid tool parameters more often, though it usually recovered after retries.
That kind of issue was not fully visible in the overall quality score, which is exactly the sort of gap Datadog is trying to close. In this case, the trace-level evidence led the team to improve tool handling before moving ahead with the model change. The failed calls also became candidates for new offline regression tests.
How Datadog fits into a broader market shift
Datadog is not the only company trying to help teams manage AI agents in production. Tools such as LangSmith and Langfuse offer agent tracing and evaluation capabilities, while experimentation platforms such as LaunchDarkly support controlled feature rollouts. Datadog’s pitch is that it can connect those concerns inside its existing observability ecosystem.
That matters because AI agents are not static applications. Their behavior can shift with changes to models, prompts, retrieval strategies or tools, and those shifts may not show up in predeployment tests. Production experimentation gives teams a way to detect those differences sooner and turn unexpected failures into future regression cases.
What this means for agent engineering
Datadog’s broader argument is that online experiments can create a feedback loop between production behavior and development testing. Unexpected failures become new regression cases, and follow-up experiments can check whether changes actually fix the problem.
The shift is subtle but important: instead of asking only whether an agent passes a predefined evaluation, teams can ask whether it performs better in real conditions. For AI agent engineering, that may become as important as traditional software testing.
Source: Original report
Was this helpful?
Explore more: Software Testing Services More AI & Automation Tech News
Last Modified: November 17, 2026 at 10:33 pm
0 views
