Halodoc
Read post

Beyond Executor Slots: Building a Memory-Budget Allocator for Jenkins CI

Halodoc's engineering team built a memory-budget allocation layer on top of Jenkins to solve OOM kills on shared Mac CI workers. Jenkins only tracks executor slots, not available memory, so multiple memory-heavy iOS/Android builds would land on the same machine and crash mid-run. The solution uses a DynamoDB-backed ledger to track each worker's allocatable memory budget and active reservations. Before a build starts, the allocator looks up the workload's memory tier, finds an eligible worker with sufficient budget, and performs an atomic conditional write to reserve capacity. Concurrent reservation races are handled via DynamoDB's ConditionalCheckFailedException. Four failure recovery paths ensure reservations are always released. After rollout, abort rate dropped 36% and pass rate improved from 52.5% to 71.4% despite a 47.9% increase in build volume, though confounding factors (new hardware, build mix changes) are acknowledged.

    #cicd#distributed-systems#jenkins#aws-dynamodb
Jul 24•12m read time•From blogs.halodoc.io
Post cover image
Table of contents
A Free Executor Did Not Mean a Healthy WorkerWhy the Executor Model Broke DownA Shared Ledger for Memory-Budget ReservationsCalibrating Workload BudgetsThe Allocation LifecyclePreventing Double BookingFailure RecoveryHeath Checks, Quarantine and ObservabilityOperational Impact and What We Can AttributeTrade-offs and Current LimitationsConclusionReferencesAbout Halodoc
3.6K Impressions
Halodoc's image
Halodoc

HaloDoc is a healthcare technology platform that offers telemedicine services, online pharmacy, and ...

61 Followers

•

678 Upvotes

Would you recommend this post?

Copy link
WhatsApp
Facebook
X
New Squad
  • © 2026 Daily Dev Ltd.
  • Guidelines
  • Explore
  • Tags
  • Sources
  • Squads
  • Leaderboard