--- title: Beyond Executor Slots: Building a Memory-Budget... description: Halodoc's engineering team built a memory-budget allocation layer on top of Jenkins to solve OOM kills on shared Mac CI workers. Jenkins only tracks executor... canonical: https://daily.dev/posts/beyond-executor-slots-building-a-memory-budget-allocator-for-jenkins-ci-zxyp8rzkk twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Beyond Executor Slots: Building a Memory-Budget Allocator for Jenkins CI | daily.dev og:description: Halodoc's engineering team built a memory-budget allocation layer on top of Jenkins to solve OOM kills on shared Mac CI workers. Jenkins only tracks executor... og:url: https://daily.dev/posts/beyond-executor-slots-building-a-memory-budget-allocator-for-jenkins-ci-zxyp8rzkk og:image: https://api.daily.dev/og/posts/ZXYp8RzKk.png og:image:alt: Beyond Executor Slots: Building a Memory-Budget Allocator for Jenkins CI og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Beyond Executor Slots: Building a Memory-Budget Allocator for Jenkins CI **[Halodoc](https://daily.dev/sources/halodoc)** · 12 min read · 2 upvotes · 0 comments ## Summary Halodoc's engineering team built a memory-budget allocation layer on top of Jenkins to solve OOM kills on shared Mac CI workers. Jenkins only tracks executor slots, not available memory, so multiple memory-heavy iOS/Android builds would land on the same machine and crash mid-run. The solution uses a DynamoDB-backed ledger to track each worker's allocatable memory budget and active reservations. Before a build starts, the allocator looks up the workload's memory tier, finds an eligible worker with sufficient budget, and performs an atomic conditional write to reserve capacity. Concurrent reservation races are handled via DynamoDB's ConditionalCheckFailedException. Four failure recovery paths ensure reservations are always released. After rollout, abort rate dropped 36% and pass rate improved from 52.5% to 71.4% despite a 47.9% increase in build volume, though confounding factors (new hardware, build mix changes) are acknowledged. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How can I prevent OOM kills when multiple Jenkins builds share the same Mac agent? Add a memory-budget allocation layer on top of Jenkins that tracks each worker's allocatable memory in a shared ledger (for example a DynamoDB table) and only lets a build start after atomically reserving enough budget on an eligible worker. Assign each workload a declared memory tier (e.g. 4GB for lightweight validation, 8GB for application builds, 12GB for regression suites), exclude workers below the required budget or at capacity, and release the reservation when the build finishes via a post-always step plus fallback cleanup jobs for disconnected agents. _daily.dev surfaces engineering deep dives like this for teams debugging flaky, memory-starved CI pipelines._ ### How do you prevent two Jenkins builds from double-booking the same CI worker's memory capacity? Use a conditional atomic write against a shared capacity ledger rather than a distributed lock. When two builds read the same ledger state and both attempt to reserve the same worker, the database (DynamoDB in this case) accepts one write and rejects the other with a ConditionalCheckFailedException; the rejected build simply moves to the next eligible candidate and retries instead of treating it as an error. _Engineers designing concurrent scheduling systems can find this kind of race-condition handling pattern on daily.dev._ ### What operational impact did adding memory-aware scheduling have on Jenkins CI reliability? After introducing a memory-budget allocator, abort rate dropped 36% (from 8.6% to 5.5%) and pass rate rose from 52.5% to 71.4%, while build volume grew 47.9% and average build duration increased by about 4.1 minutes because builds now wait upfront for sufficient memory rather than failing mid-run. The team notes these are directional results, not controlled measurements, since fleet capacity and build mix also changed during the comparison period. _daily.dev helps engineers weigh real-world CI reliability trade-offs before changing their own build scheduling._ ## Similar posts on daily.dev - [Enhancing Kubernetes Stability on AWS EKS by Leveraging the "Node Swap" feature in EKS](https://daily.dev/posts/enhancing-kubernetes-stability-on-aws-eks-by-leveraging-the-node-swap-feature-in-eks-kmz5mu5hs) · Halodoc · 2 upvotes · 0 comments - [Pinterest Reduces Spark OOM Failures by 96% Through Auto Memory Retries](https://daily.dev/posts/pinterest-reduces-spark-oom-failures-by-96-through-auto-memory-retries-xfchaut53) · InfoQ · 0 upvotes · 0 comments - [How Data 360 Optimized Kubernetes Scheduling Architecture](https://daily.dev/posts/how-data-360-optimized-kubernetes-scheduling-architecture-dlniw6hay) · Salesforce Engineering · 3 upvotes · 0 comments - [Elastic Build Machines now protect against out of memory builds](https://daily.dev/posts/elastic-build-machines-now-protect-against-out-of-memory-builds-7swq5onuq) · Vercel · 0 upvotes · 0 comments - [Drastically Reducing Out-of-Memory Errors in Apache Spark at Pinterest](https://daily.dev/posts/drastically-reducing-out-of-memory-errors-in-apache-spark-at-pinterest-mgvdvkmyr) · Pinterest Engineering · 1 upvotes · 0 comments --- Tags: [#cicd](https://daily.dev/tags/cicd), [#distributed-systems](https://daily.dev/tags/distributed-systems), [#jenkins](https://daily.dev/tags/jenkins), [#aws-dynamodb](https://daily.dev/tags/aws-dynamodb) [View this post on daily.dev](https://daily.dev/posts/beyond-executor-slots-building-a-memory-budget-allocator-for-jenkins-ci-zxyp8rzkk) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Beyond Executor Slots: Building a Memory-Budget Allocator for Jenkins CI","url":"https://daily.dev/posts/beyond-executor-slots-building-a-memory-budget-allocator-for-jenkins-ci-zxyp8rzkk","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/beyond-executor-slots-building-a-memory-budget-allocator-for-jenkins-ci-zxyp8rzkk"},"datePublished":"2026-07-24T13:27:41.397Z","dateModified":"2026-09-14T06:01:58.477Z","description":"Halodoc's engineering team built a memory-budget allocation layer on top of Jenkins to solve OOM kills on shared Mac CI workers. Jenkins only tracks executor...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/2e2e1d4eb96814a06e37cbac237f8725?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/2e2e1d4eb96814a06e37cbac237f8725?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Halodoc","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Halodoc","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/b3d554a1a7b44df48789c7ca6e50c4c2","url":"https://daily.dev/sources/halodoc"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/beyond-executor-slots-building-a-memory-budget-allocator-for-jenkins-ci-zxyp8rzkk","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"cicd,distributed-systems,jenkins,aws-dynamodb","timeRequired":"PT12M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Halodoc","item":"https://daily.dev/sources/halodoc"},{"@type":"ListItem","position":3,"name":"Beyond Executor Slots: Building a Memory-Budget Allocator for Jenkins CI"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/beyond-executor-slots-building-a-memory-budget-allocator-for-jenkins-ci-zxyp8rzkk#faq","mainEntity":[{"@type":"Question","name":"How can I prevent OOM kills when multiple Jenkins builds share the same Mac agent?","acceptedAnswer":{"@type":"Answer","text":"Add a memory-budget allocation layer on top of Jenkins that tracks each worker's allocatable memory in a shared ledger (for example a DynamoDB table) and only lets a build start after atomically reserving enough budget on an eligible worker. Assign each workload a declared memory tier (e.g. 4GB for lightweight validation, 8GB for application builds, 12GB for regression suites), exclude workers below the required budget or at capacity, and release the reservation when the build finishes via a post-always step plus fallback cleanup jobs for disconnected agents. daily.dev surfaces engineering deep dives like this for teams debugging flaky, memory-starved CI pipelines."}},{"@type":"Question","name":"How do you prevent two Jenkins builds from double-booking the same CI worker's memory capacity?","acceptedAnswer":{"@type":"Answer","text":"Use a conditional atomic write against a shared capacity ledger rather than a distributed lock. When two builds read the same ledger state and both attempt to reserve the same worker, the database (DynamoDB in this case) accepts one write and rejects the other with a ConditionalCheckFailedException; the rejected build simply moves to the next eligible candidate and retries instead of treating it as an error. Engineers designing concurrent scheduling systems can find this kind of race-condition handling pattern on daily.dev."}},{"@type":"Question","name":"What operational impact did adding memory-aware scheduling have on Jenkins CI reliability?","acceptedAnswer":{"@type":"Answer","text":"After introducing a memory-budget allocator, abort rate dropped 36% (from 8.6% to 5.5%) and pass rate rose from 52.5% to 71.4%, while build volume grew 47.9% and average build duration increased by about 4.1 minutes because builds now wait upfront for sufficient memory rather than failing mid-run. The team notes these are directional results, not controlled measurements, since fleet capacity and build mix also changed during the comparison period. daily.dev helps engineers weigh real-world CI reliability trade-offs before changing their own build scheduling."}}]} ```