--- title: Apache Spark Resource Optimization: Lessons From Eight... description: Salesforce's Data 360 engineering team describes how they optimized Spark resource allocation across roughly eight million daily workload executions by... canonical: https://daily.dev/posts/apache-spark-resource-optimization-lessons-from-eight-million-jobs-a-day-56vchixir twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Apache Spark Resource Optimization: Lessons From Eight Million Jobs a Day | daily.dev og:description: Salesforce's Data 360 engineering team describes how they optimized Spark resource allocation across roughly eight million daily workload executions by... og:url: https://daily.dev/posts/apache-spark-resource-optimization-lessons-from-eight-million-jobs-a-day-56vchixir og:image: https://api.daily.dev/og/posts/56VChiXIr.png og:image:alt: Apache Spark Resource Optimization: Lessons From Eight Million Jobs a Day og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Apache Spark Resource Optimization: Lessons From Eight Million Jobs a Day **[Salesforce Engineering](https://daily.dev/sources/salesforceeng)** · 9 min read · 6 upvotes · 0 comments ## Summary Salesforce's Data 360 engineering team describes how they optimized Spark resource allocation across roughly eight million daily workload executions by building telemetry collection through their Data Processing Controller gateway, using an extended DataFlint plugin feeding Kafka into an Iceberg lakehouse. They explain how a curated Silver analytical layer prevents misleading conclusions from retries, spill double-counting, and partial runs, and how they used Trino plus an agentic SQL-generation skill to identify workloads with idle or zero-task executors. Applying this process, they reduced initial executor counts and lowered executor ceilings for specific workloads, validating changes over month-long production runs and achieving a 31% drop in weekly compute-cost run rate across targeted workloads while job starts increased over 7%. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How much did compute costs drop after optimizing Spark executor allocation based on execution telemetry? Weekly compute-cost run rate fell 31% in matched four-week windows before and after the changes, across targeted workloads at Salesforce's Data 360 platform. Separately, job starts through the Data Processing Controller increased by more than 7% in the same environments. This was a before-and-after comparison, not a controlled experiment, so other factors may have contributed. _See how teams validate Spark resource changes with production metrics by following cost optimization case studies on daily.dev._ ### How can I collect Spark execution telemetry without modifying application code? Instrument a shared submission gateway rather than individual applications. Salesforce extended an open-source DataFlint Spark plugin and injected it at their Data Processing Controller, the common gateway for managed Spark submissions, using Spark's listener hooks to capture per-driver telemetry and export it via Kafka into an Iceberg lakehouse, keeping collection isolated from customer code. _Engineers building Spark observability can track approaches like this on daily.dev before touching application code._ ### What is a safe way to reduce Spark executor resource allocation without hurting performance? Reduce initialExecutors while keeping Spark's dynamic allocation enabled, so startup capacity shrinks but can still grow under demand. Salesforce lowered initialExecutors to 4 for one workload and separately cut a maximum-executor ceiling by about 20% for another, validating each change over more than a month in production by checking p95 runtime, retries, OOMs, and spill before rolling out further. _Developers tuning Spark resource profiles can compare validated tactics like this on daily.dev._ ## Similar posts on daily.dev - [How Data 360 Optimized Kubernetes Scheduling Architecture](https://daily.dev/posts/how-data-360-optimized-kubernetes-scheduling-architecture-dlniw6hay) · Salesforce Engineering · 3 upvotes · 0 comments - [How Data 360 Segmentation Processes a Quadrillion Records](https://daily.dev/posts/how-data-360-segmentation-processes-a-quadrillion-records-2ajjiamao) · Salesforce Engineering · 0 upvotes · 0 comments - [How we cut Spark compute costs by 44% with agentic AI and Datadog Jobs Monitoring](https://daily.dev/posts/how-we-cut-spark-compute-costs-by-44-with-agentic-ai-and-datadog-jobs-monitoring-rxna5rcsp) · Datadog · 0 upvotes · 0 comments - [Rethinking Distributed Systems for Serverless Performance and Reliability](https://daily.dev/posts/rethinking-distributed-systems-for-serverless-performance-and-reliability-psqtmpomz) · databricks · 11 upvotes · 0 comments --- Tags: [#performance](https://daily.dev/tags/performance), [#kubernetes](https://daily.dev/tags/kubernetes), [#observability](https://daily.dev/tags/observability), [#apache-spark](https://daily.dev/tags/apache-spark), [#apache-iceberg](https://daily.dev/tags/apache-iceberg) [View this post on daily.dev](https://daily.dev/posts/apache-spark-resource-optimization-lessons-from-eight-million-jobs-a-day-56vchixir) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Apache Spark Resource Optimization: Lessons From Eight Million Jobs a Day","url":"https://daily.dev/posts/apache-spark-resource-optimization-lessons-from-eight-million-jobs-a-day-56vchixir","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/apache-spark-resource-optimization-lessons-from-eight-million-jobs-a-day-56vchixir"},"datePublished":"2026-10-08T15:19:25.347Z","dateModified":"2026-10-08T15:51:52.348Z","description":"Salesforce's Data 360 engineering team describes how they optimized Spark resource allocation across roughly eight million daily workload executions by...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/c3a9abcdf0568d164920e92f05eac8f4?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/c3a9abcdf0568d164920e92f05eac8f4?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Salesforce Engineering","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Salesforce Engineering","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/513375817ec24d97b9cdb5efd16f87f2","url":"https://daily.dev/sources/salesforceeng"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/apache-spark-resource-optimization-lessons-from-eight-million-jobs-a-day-56vchixir","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":6},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"performance,kubernetes,observability,apache-spark,apache-iceberg","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Salesforce Engineering","item":"https://daily.dev/sources/salesforceeng"},{"@type":"ListItem","position":3,"name":"Apache Spark Resource Optimization: Lessons From Eight Million Jobs a Day"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/apache-spark-resource-optimization-lessons-from-eight-million-jobs-a-day-56vchixir#faq","mainEntity":[{"@type":"Question","name":"How much did compute costs drop after optimizing Spark executor allocation based on execution telemetry?","acceptedAnswer":{"@type":"Answer","text":"Weekly compute-cost run rate fell 31% in matched four-week windows before and after the changes, across targeted workloads at Salesforce's Data 360 platform. Separately, job starts through the Data Processing Controller increased by more than 7% in the same environments. This was a before-and-after comparison, not a controlled experiment, so other factors may have contributed. See how teams validate Spark resource changes with production metrics by following cost optimization case studies on daily.dev."}},{"@type":"Question","name":"How can I collect Spark execution telemetry without modifying application code?","acceptedAnswer":{"@type":"Answer","text":"Instrument a shared submission gateway rather than individual applications. Salesforce extended an open-source DataFlint Spark plugin and injected it at their Data Processing Controller, the common gateway for managed Spark submissions, using Spark's listener hooks to capture per-driver telemetry and export it via Kafka into an Iceberg lakehouse, keeping collection isolated from customer code. Engineers building Spark observability can track approaches like this on daily.dev before touching application code."}},{"@type":"Question","name":"What is a safe way to reduce Spark executor resource allocation without hurting performance?","acceptedAnswer":{"@type":"Answer","text":"Reduce initialExecutors while keeping Spark's dynamic allocation enabled, so startup capacity shrinks but can still grow under demand. Salesforce lowered initialExecutors to 4 for one workload and separately cut a maximum-executor ceiling by about 20% for another, validating each change over more than a month in production by checking p95 runtime, retries, OOMs, and spill before rolling out further. Developers tuning Spark resource profiles can compare validated tactics like this on daily.dev."}}]} ```