--- title: Monitor Databricks with Datadog | daily.dev description: Datadog's Databricks integration combines Agent-based Spark metrics, API polling of job/task runs, and system table queries to give end-to-end visibility into... canonical: https://daily.dev/posts/monitor-databricks-with-datadog-giqfytj4u twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Monitor Databricks with Datadog | daily.dev og:description: Datadog's Databricks integration combines Agent-based Spark metrics, API polling of job/task runs, and system table queries to give end-to-end visibility into... og:url: https://daily.dev/posts/monitor-databricks-with-datadog-giqfytj4u og:image: https://api.daily.dev/og/posts/giqfyTJ4U.png og:image:alt: Monitor Databricks with Datadog og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Monitor Databricks with Datadog **[Datadog](https://daily.dev/sources/datadog)** · 8 min read · 0 upvotes · 0 comments ## Summary Datadog's Databricks integration combines Agent-based Spark metrics, API polling of job/task runs, and system table queries to give end-to-end visibility into Databricks analytics, data engineering, and Model Serving workloads. It covers Jobs Monitoring for job/cluster health, cost, and rightsizing recommendations; Quality Monitoring and Data Streams Monitoring for data lineage and streaming pipeline issues; Model Serving endpoint monitoring with prebuilt dashboards and templates; Cloud Cost Management for DBU cost attribution and forecasting; and Reference Tables for enriching telemetry with business metadata. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How can I monitor Databricks job costs and performance alongside my other data pipelines? Datadog's Jobs Monitoring feature combines Agent-captured Spark metrics, Databricks API polling of job and task execution data, and system table queries on a dedicated SQL warehouse for cost and lineage data. It surfaces job failures, cost and performance trends, anomalies, and per-cluster rightsizing recommendations with projected savings, and integrates with Azure Data Factory and dbt for upstream/downstream visibility. _daily.dev surfaces practical guides for engineers comparing data pipeline observability tools like this._ ### How do I detect data quality issues in Databricks Delta and Unity Catalog tables? Datadog Quality Monitoring provides Data Observability monitors that continuously track data freshness, volume, column metrics, and custom rule violations using both anomaly detection and threshold-based rules. It collects lineage via the system.query.history table, and the OpenLineage Spark integration adds Spark-level lineage so quality alerts can be traced back to specific Spark jobs and transformations. _Engineers tracking down flaky data pipelines can find similar deep dives on daily.dev._ ### What metrics should I alert on for Databricks Model Serving endpoints? Key signals include server-side 5xx errors indicating model crashes or infrastructure failures, tail p99 latency pointing to autoscaling lag or endpoint pressure, and GPU memory saturation that can cause out-of-memory crashes. Endpoint health metrics such as latency, throughput, error counts, CPU/memory utilization, and GPU utilization flow into a prebuilt Model Serving dashboard with ready-made monitor templates. _daily.dev helps engineers stay current on best practices for alerting on AI/ML serving infrastructure._ ## Similar posts on daily.dev - [Databricks’ native monitoring resources](https://daily.dev/posts/databricks-native-monitoring-resources-rwbjvjawj) · Datadog · 0 upvotes · 0 comments - [Monitor Databricks with Grafana Cloud for instant visibility into your workloads](https://daily.dev/posts/monitor-databricks-with-grafana-cloud-for-instant-visibility-into-your-workloads-fmsak5z0j) · Grafana Labs · 0 upvotes · 0 comments - [Turn Databricks telemetry into actionable decisions with Dynatrace](https://daily.dev/posts/turn-databricks-telemetry-into-actionable-decisions-with-dynatrace-t6vr2eibt) · Dynatrace · 1 upvotes · 0 comments - [Key metrics for monitoring Databricks](https://daily.dev/posts/key-metrics-for-monitoring-databricks-hyjef9qqr) · Datadog · 1 upvotes · 0 comments --- Tags: [#devops](https://daily.dev/tags/devops), [#apache-spark](https://daily.dev/tags/apache-spark), [#databricks](https://daily.dev/tags/databricks), [#finops](https://daily.dev/tags/finops), [#data-observability](https://daily.dev/tags/data-observability) [View this post on daily.dev](https://daily.dev/posts/monitor-databricks-with-datadog-giqfytj4u) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Monitor Databricks with Datadog","url":"https://daily.dev/posts/monitor-databricks-with-datadog-giqfytj4u","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/monitor-databricks-with-datadog-giqfytj4u"},"datePublished":"2026-10-02T16:41:42.561Z","dateModified":"2026-10-02T17:01:39.353Z","description":"Datadog's Databricks integration combines Agent-based Spark metrics, API polling of job/task runs, and system table queries to give end-to-end visibility into...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/f6e296d01e824a4606340ca9303de9e0?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/f6e296d01e824a4606340ca9303de9e0?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Datadog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Datadog","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/8f9d7e3e4bd64bd7b9db9b2ea83e0c6a","url":"https://daily.dev/sources/datadog"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/monitor-databricks-with-datadog-giqfytj4u","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"devops,apache-spark,databricks,finops,data-observability","timeRequired":"PT8M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Datadog","item":"https://daily.dev/sources/datadog"},{"@type":"ListItem","position":3,"name":"Monitor Databricks with Datadog"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/monitor-databricks-with-datadog-giqfytj4u#faq","mainEntity":[{"@type":"Question","name":"How can I monitor Databricks job costs and performance alongside my other data pipelines?","acceptedAnswer":{"@type":"Answer","text":"Datadog's Jobs Monitoring feature combines Agent-captured Spark metrics, Databricks API polling of job and task execution data, and system table queries on a dedicated SQL warehouse for cost and lineage data. It surfaces job failures, cost and performance trends, anomalies, and per-cluster rightsizing recommendations with projected savings, and integrates with Azure Data Factory and dbt for upstream/downstream visibility. daily.dev surfaces practical guides for engineers comparing data pipeline observability tools like this."}},{"@type":"Question","name":"How do I detect data quality issues in Databricks Delta and Unity Catalog tables?","acceptedAnswer":{"@type":"Answer","text":"Datadog Quality Monitoring provides Data Observability monitors that continuously track data freshness, volume, column metrics, and custom rule violations using both anomaly detection and threshold-based rules. It collects lineage via the system.query.history table, and the OpenLineage Spark integration adds Spark-level lineage so quality alerts can be traced back to specific Spark jobs and transformations. Engineers tracking down flaky data pipelines can find similar deep dives on daily.dev."}},{"@type":"Question","name":"What metrics should I alert on for Databricks Model Serving endpoints?","acceptedAnswer":{"@type":"Answer","text":"Key signals include server-side 5xx errors indicating model crashes or infrastructure failures, tail p99 latency pointing to autoscaling lag or endpoint pressure, and GPU memory saturation that can cause out-of-memory crashes. Endpoint health metrics such as latency, throughput, error counts, CPU/memory utilization, and GPU utilization flow into a prebuilt Model Serving dashboard with ready-made monitor templates. daily.dev helps engineers stay current on best practices for alerting on AI/ML serving infrastructure."}}]} ```