Large Language Model (LLM) applications are nondeterministic: their outputs are never exactly the same. Traditional testing methods and operational metrics are not sufficient to measure output quality, accuracy, or safety in agentic workflows. When you change a prompt, model, or your application’s architecture, how do you know if the changes actually made things better?

In this workshop, you'll learn how to use Datadog LLM Observability's Experiments and Evaluations features to systematically measure and improve the quality of your agentic AI applications. Evaluations enable you to measure quality in production, and Experiments help you validate changes offline using real production traces. Together, they enable deliberate, data-driven improvement instead of relying on guesswork or trial and error.

Through a hands-on lab, you'll walk through a full development loop: identifying a quality issue, creating an evaluator that defines what "good" means for your use case, running an experiment to compare variations, and confirming improvements with runtime evaluations and monitors. By the end of the workshop, you'll have the skills to build a continuous feedback loop between production monitoring and pre-deployment testing, so you can ship improvements with confidence.