Skip to main content
DASH NYC, June 9-10 | AI + Observability.

Back to Catalog
End-to-End Observability
Harnessing AI
Workshop
Intermediate

Evaluate and Optimize AI Agent Performance

About this Session

Large Language Model (LLM) applications are nondeterministic: their outputs are never exactly the same. Traditional testing methods and operational metrics are not sufficient to measure output quality, accuracy, or safety in agentic workflows. When you change a prompt, model, or your application’s architecture, how do you know if the changes actually made things better?

 

In this workshop, you'll learn how to use Datadog LLM Observability's Experiments and Evaluations features to systematically measure and improve the quality of your agentic AI applications. Evaluations enable you to measure quality in production, and Experiments help you validate changes offline using real production traces. Together, they enable deliberate, data-driven improvement instead of relying on guesswork or trial and error.

 

Through a hands-on lab, you'll walk through a full development loop: identifying a quality issue, creating an evaluator that defines what "good" means for your use case, running an experiment to compare variations, and confirming improvements with runtime evaluations and monitors. By the end of the workshop, you'll have the skills to build a continuous feedback loop between production monitoring and pre-deployment testing, so you can ship improvements with confidence.

Related Sessions

End-to-End Observability
Expo Theater Talk

From Ingestion to AI: Ensuring Data Reliability Across the Full Lifecycle

End-to-End Observability
Harnessing AI
Breakout Session

From Reactive to Proactive: How SREs Can Optimize Their Application Services Before Users Are Affected

Speakers

Nicolas Homble, , OnePay
End-to-End Observability
Harnessing AI
Workshop

Build with LLM Observability: From Setup to Signal

End-to-End Observability
Harnessing AI
Developer Autonomy

Datadog Core Skills for Developers - Pre-Day

End-to-End Observability
Harnessing AI
Scaling Systems

Datadog Core Skills for Site Reliability Engineers (SREs) - Pre-Day

End-to-End Observability
Workshop
intermediate

Serverless Observability on AWS
End-to-End Observability
Expo Theater Talk

From Ingestion to AI: Ensuring Data Reliability Across the Full Lifecycle

End-to-End Observability
Harnessing AI
Breakout Session

From Reactive to Proactive: How SREs Can Optimize Their Application Services Before Users Are Affected

Speakers

Nicolas Homble, , OnePay
End-to-End Observability
Harnessing AI
Workshop

Build with LLM Observability: From Setup to Signal

End-to-End Observability
Harnessing AI
Developer Autonomy

Datadog Core Skills for Developers - Pre-Day

End-to-End Observability
Harnessing AI
Scaling Systems

Datadog Core Skills for Site Reliability Engineers (SREs) - Pre-Day

End-to-End Observability
Workshop
intermediate

Serverless Observability on AWS
End-to-End Observability
Harnessing AI
Breakout Session

From Reactive to Proactive: How SREs Can Optimize Their Application Services Before Users Are Affected

Speakers

Nicolas Homble, , OnePay
Harnessing AI
Expo Theater Talk

How AI Is Redefining the Datadog Experience—and How to Make the Most of It

Harnessing AI
Developer Autonomy
Breakout Session

The AI Engineering Playbook: How to Evaluate & Iterate at Every Phase of Development

End-to-End Observability
Harnessing AI
Workshop

Build with LLM Observability: From Setup to Signal

End-to-End Observability
Harnessing AI
Developer Autonomy

Datadog Core Skills for Developers - Pre-Day

End-to-End Observability
Harnessing AI
Scaling Systems

Datadog Core Skills for Site Reliability Engineers (SREs) - Pre-Day
End-to-End Observability
Expo Theater Talk

From Ingestion to AI: Ensuring Data Reliability Across the Full Lifecycle

End-to-End Observability
Harnessing AI
Breakout Session

From Reactive to Proactive: How SREs Can Optimize Their Application Services Before Users Are Affected

Speakers

Nicolas Homble, , OnePay
End-to-End Observability
Harnessing AI
Workshop

Build with LLM Observability: From Setup to Signal

End-to-End Observability
Harnessing AI
Developer Autonomy

Datadog Core Skills for Developers - Pre-Day

End-to-End Observability
Harnessing AI
Scaling Systems

Datadog Core Skills for Site Reliability Engineers (SREs) - Pre-Day

End-to-End Observability
Workshop
intermediate

Serverless Observability on AWS