The Databricks Workspace extension for Dynatrace unifies telemetry from across the Databricks platform into a single correlated view. It covers job and Spark reliability (distributed traces, task-level metrics, duration breakdowns), cost and capacity optimization (billing data enriched with workload metadata, right-sizing recommendations based on real utilization percentiles), AI-serving observability (request latency, token usage, requester breakdowns for chargeback), and audit log investigation (filtering by IP, service, user, and status code). Six bundled dashboards — workspace overview, job runs, cluster resource usage, cost management, AI gateway/model serving, and audit logs — provide immediate visibility without custom setup.
Table of contents
Operational reliability: a faster path to root causeConnect cost spikes to the workloads driving themGovern AI-serving costs and reliability as usage growsTurn audit logs into an active investigation toolReady-made value with bundled dashboardsPutting it all togetherQuestions this post answers
What metrics does the Dynatrace Databricks Workspace extension collect for job monitoring?
The extension collects job success rate, run count, queue/setup/execution/cleanup durations, active and completed run visibility, task-level distributed traces per job run, and Spark-level metrics including failed tasks, stage throughput, shuffle pressure, and executor health. The duration breakdown lets teams pinpoint whether a slowdown originates in the queue, during setup, or mid-execution without re-running the entire job. Teams troubleshooting Databricks job failures track these signals alongside the rest of their stack on daily.dev.
How can I identify which Databricks workloads are causing cost spikes?
The Dynatrace Databricks Workspace extension ingests billing system table data and enriches it with workload metadata, letting you trace a cost spike to a specific job, cluster, SQL warehouse, notebook, model serving endpoint, instance pool, app, or pipeline. The Cluster Resource Utilization dashboard flags over-provisioned clusters with concrete recommendations derived from actual CPU and memory utilization percentiles. Engineers making Databricks right-sizing decisions find cost optimization discussions like this on daily.dev.