Datadog's Jobs Monitoring now offers proactive optimization recommendations for Apache Spark and Databricks jobs, targeting cost and duration improvements. The feature surfaces actionable changes at the code, query, configuration, and infrastructure layers using real production execution data. Engineers can apply fixes via Bits Code (which can auto-generate pull requests), chat with Bits AI during reactive investigations, or use the Datadog MCP Server to bring Spark execution context directly into AI-assisted coding workflows. Two MCP tools — get_spark_health and get_spark_sql_plan — give coding agents focused runtime context without overwhelming them with full Spark History Server logs. Datadog reports internal results of 44% compute cost reduction and 60% run duration improvement using this workflow. Proactive Job Recommendations are in Public Preview; MCP and Bits integration is generally available.
