---
title: "How we cut Spark compute costs by 44% with agentic AI and Datadog Jobs Monitoring"
url: https://daily.dev/posts/how-we-cut-spark-compute-costs-by-44-with-agentic-ai-and-datadog-jobs-monitoring-rxna5rcsp
source_url: https://www.datadoghq.com/blog/using-agentic-ai-with-jobs-monitoring
type: article
source: "Datadog"
published: 2026-06-02T11:56:30.213Z
updated: 2026-06-02T11:56:58.756Z
tags: ["devops", "mcp", "apache-spark", "agentic-ai"]
reading_time: 9
upvotes: 0
comments: 0
language: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# How we cut Spark compute costs by 44% with agentic AI and Datadog Jobs Monitoring

**[Datadog](https://daily.dev/sources/datadog)** · 9 min read · 0 upvotes · 0 comments

## Summary

Datadog's Referential Data Platform team used an AI agent built on Claude, combined with Datadog Jobs Monitoring, to debug and optimize their ServiceQueryEdge Spark job. The job processed up to 27 TB and 16 billion records daily, costing ~$1.5k/day. By building a multi-agent system with a validator subagent to filter false positives, the team identified three key optimizations: removing redundant aggregation, replacing a SortMergeJoin with a broadcast join, and fixing a flawed salting implementation that caused data skew. The result was a 44% reduction in daily compute costs (~$830/day) and a 60% reduction in run duration in their largest datacenter, projecting ~$300k in annual savings. Key lessons include scoping data precisely for agents, using a validator agent to filter bad recommendations, and treating AI as a collaborative partner rather than an autonomous solver. Datadog has since released purpose-scoped Spark tools (get_spark_health, get_spark_sql_plan) in their MCP Server.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://www.datadoghq.com/blog/using-agentic-ai-with-jobs-monitoring>

## Similar posts on daily.dev

- [Optimize Spark and Databricks jobs with Datadog](https://daily.dev/posts/optimize-spark-and-databricks-jobs-with-datadog-gp3lblhkw) · Datadog · 0 upvotes · 0 comments
- [How Datadog uses AI to build internal software delivery tools and improve system performance](https://daily.dev/posts/how-datadog-uses-ai-to-build-internal-software-delivery-tools-and-improve-system-performance-1mupjofm5) · Datadog · 3 upvotes · 0 comments
- [Reducing Agentforce Debugging to One Day with Query-Driven Observability](https://daily.dev/posts/reducing-agentforce-debugging-to-one-day-with-query-driven-observability-leacfrxu8) · Salesforce Engineering · 4 upvotes · 0 comments

---

Tags: [#devops](https://daily.dev/tags/devops), [#mcp](https://daily.dev/tags/mcp), [#apache-spark](https://daily.dev/tags/apache-spark), [#agentic-ai](https://daily.dev/tags/agentic-ai)

[View this post on daily.dev](https://daily.dev/posts/how-we-cut-spark-compute-costs-by-44-with-agentic-ai-and-datadog-jobs-monitoring-rxna5rcsp)