Skip to main content
DASH NYC, June 9-10 | AI + Observability.

Back to Catalog

The AI Engineering Playbook: How to Evaluate & Iterate at Every Phase of Development

About this Session

AI coding tools are accelerating development velocity, creating a release challenge most teams aren’t equipped for. Without controlled rollout, higher change velocity makes it harder to know which specific release drove the results you’re seeing in production.

And when teams use AI, to build AI – LLM apps and AI agents– complexity multiplies. Traditional observability can’t ensure AI agent quality, performance, and cost-efficiency at production scale.

In this session, you'll learn how Datadog Feature Flags and LLM Observability work together to drive equilibrium across the software delivery lifecycle. You’ll learn how leading AI teams move fast while maximizing reliability at each stage: 

  • Evaluating and iterating agents with structured experiments in pre-production
  • Controlling rollout with automated guardrails that pause or roll back releases 
  • Instantly link feature releases to traces, user sessions, and downstream behavior
  • Monitoring agent behavior in production, tracing every step, retrieval and tool call

You and your team will leave with a practical AI engineering playbook showing you how to ship faster while iteratively improving quality, security, and ROI. 

 

Related Sessions