Datadog's Referential Data Platform team used an AI agent built on Claude, combined with Datadog Jobs Monitoring, to debug and optimize their ServiceQueryEdge Spark job. The job processed up to 27 TB and 16 billion records daily, costing ~$1.5k/day. By building a multi-agent system with a validator subagent to filter false positives, the team identified three key optimizations: removing redundant aggregation, replacing a SortMergeJoin with a broadcast join, and fixing a flawed salting implementation that caused data skew. The result was a 44% reduction in daily compute costs (~$830/day) and a 60% reduction in run duration in their largest datacenter, projecting ~$300k in annual savings. Key lessons include scoping data precisely for agents, using a validator agent to filter bad recommendations, and treating AI as a collaborative partner rather than an autonomous solver. Datadog has since released purpose-scoped Spark tools (get_spark_health, get_spark_sql_plan) in their MCP Server.
