Skip to main content
DASH NYC, June 9-10 | AI + Observability.

Back to Catalog

The Hidden Data Pipelines Behind Datadog: Lessons from Building Observability for Our Own Data Teams

About this Session

Behind many of the Datadog products customers use every day is a data world that looks a lot like yours: Spark jobs, Airflow DAGs, dbt models, Iceberg tables, and many teams depending on data arriving on time and in the right shape.

For years, our data teams operated the way most companies still do. Revenue engineering, internal analytics, product engineering, and platform teams each built their own freshness checks, quality rules, and troubleshooting runbooks. When a pipeline broke, no one knew which downstream dashboards or jobs were affected until someone complained. It worked, until the number of pipelines, tools, and consumers grew faster than the operating model around them.

In this talk, Jean-Mathieu Saponaro (Engineering Director) and Harel Shein (Engineering Manager) will share how Datadog's Analytics Data Platform org moved to unified tooling, partnering with the Data Observability product team to build a single pane of glass that works for our data practitioners and for yours. We will cover what we standardized, what we replaced, and how this changed the way our teams work: from alerting on job failures to troubleshooting end-to-end flows, from guessing about downstream impact to using lineage, and from reacting to broken pipelines to catching data issues in CI. Attendees will leave with ideas for their own data orgs, including a look at what comes next as trusted datasets start being consumed by AI agents.

 

Related Sessions