--- title: Day 18/30 AWS System Design Patterns | daily.dev description: A system design scenario presents a genomics data pipeline running nightly batch jobs on EC2 Spot Instances (c5.4xlarge, us-east-1a). When AWS reclaims 18... canonical: https://daily.dev/posts/day-18-30-aws-system-design-patterns-had3njxtb twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Day 18/30 AWS System Design Patterns | daily.dev og:description: A system design scenario presents a genomics data pipeline running nightly batch jobs on EC2 Spot Instances (c5.4xlarge, us-east-1a). When AWS reclaims 18... og:url: https://daily.dev/posts/day-18-30-aws-system-design-patterns-had3njxtb og:image: https://api.daily.dev/og/posts/Had3NJxtb.png og:image:alt: Day 18/30 AWS System Design Patterns og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Day 18/30 AWS System Design Patterns **[Joud Awad](https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh)** · [@joudawad](https://daily.dev/joudawad) · 2 min read · 67 upvotes · 20 comments ## Summary A system design scenario presents a genomics data pipeline running nightly batch jobs on EC2 Spot Instances (c5.4xlarge, us-east-1a). When AWS reclaims 18 instances simultaneously at 2:17 AM, all running jobs are lost with no recovery, adding 7 hours to the pipeline. The scenario highlights three root causes: no checkpointing, single instance type, and single Availability Zone. Four answer options are evaluated, covering switching to On-Demand, implementing checkpointing with diversification, using max price bids, and switching to ECS Fargate Spot. ## Content A data pipeline team runs nightly batch jobs on a fleet of EC2 Spot Instances _(spare AWS compute capacity sold at discount — can be reclaimed by AWS with 2-minute notice when capacity is needed)_ — all c5.4xlarge in us-east-1a — processing genomics sequencing data. Each job runs for 3–5 hours. The fleet typically runs 40 instances overnight. Cost compared to On-Demand is about 70% lower. Four weeks into production, EC2 reclaims Spot capacity in us-east-1a and takes 18 instances simultaneously at 2:17 AM. The jobs running on those instances are lost with no recovery. Rerunning the failed jobs adds 7 hours to the pipeline, causing results to miss the morning delivery window. The biology team escalates. No checkpointing is implemented. Jobs restart from zero on failure. The fleet is configured with a single instance type in a single Availability Zone. What combination of changes correctly addresses this? A) Switch to On-Demand instances _(fixed-price compute, no interruption risk)_ for the nightly pipeline — Spot is unsuitable for long-running batch workloads B) Implement job checkpointing to S3 _(object storage, durable across instance loss)_, diversify across multiple instance types and AZs _(Availability Zones — isolated failure domains)_, and use the EC2 Spot interruption notice _(2-minute warning sent before reclamation)_ to trigger a final checkpoint before termination C) Use Spot with a maximum price bid set to the On-Demand price — this prevents interruption by ensuring you always outbid other buyers; but Spot interruptions are driven by capacity demand, not purely by price — AWS reclaims capacity regardless of your bid when it needs it back D) Switch to ECS Fargate Spot _(Fargate containers on Spot capacity — handles lifecycle events but does not implement checkpointing)_ — Fargate manages the Spot lifecycle automatically so interruptions are handled without custom checkpointing logic Answer in the comments. #SystemDesign #AWS #DataEngineering #DataPipeline #Scalability ## Community discussion Top comments from developers on daily.dev. **@joudawad** · 6 upvotes > The answer is B. > > Spot interruptions are not avoidable — they are a property of the pricing model. The correct response is making jobs resilient to interruption, not eliminating Spot. > > Three changes together close the gap: > > First, checkpointing: jobs write their progress state to S3 _(object storage, durable across instance loss)_ every N minutes. When an instance is reclaimed, the job restarts from the last checkpoint rather than from zero. A 4-hour job with 15-minute checkpoints loses at most 15 minutes of compute, not 4 hours. > > Second, the EC2 Spot interruption notice _(2-minute warning... **@joudawad** · 2 upvotes > A — Switching to On-Demand *(fixed-price compute)* eliminates interruption risk but costs 3–4× more per job. The correct answer preserves the Spot economics through resilience, not avoidance. **@mdshafinahmed** · 2 upvotes > Good things brother. > Thank you for sharing. **@joudawad** · 1 upvotes > Also, it would mean a lot to me if you could support my content and stay in touch 🙏 > > - YouTube: [https://www.youtube.com/@system-design-lab](https://www.youtube.com/@system-design-lab) > - LinkedIn: [https://www.linkedin.com/in/joud-awad/](https://www.linkedin.com/in/joud-awad/) > - Medium Blog: [https://joudwawad.medium.com/](https://joudwawad.medium.com/) > - Substack: [https://joudawad.substack.com/](https://joudawad.substack.com/) **@joudawad** · 1 upvotes > C — Setting a maximum Spot bid to the On-Demand price does not prevent interruption. Spot interruptions are driven by EC2 capacity demand. When AWS needs the capacity back, your instance is reclaimed regardless of your bid — up to and including the On-Demand price ceiling. ## Similar posts on daily.dev - [A successful 1,000 TPS checkout game day: pilot light to full scale with AWS ARC Region switch](https://daily.dev/posts/a-successful-1-000-tps-checkout-game-day-pilot-light-to-full-scale-with-aws-arc-region-switch-of9balhwp) · Medium · 0 upvotes · 0 comments - [Kubernetes Spot Instances: How to Cut Compute Costs Without Gambling on Reliability](https://daily.dev/posts/kubernetes-spot-instances-how-to-cut-compute-costs-without-gambling-on-reliability-izpc2uk55) · Cast AI · 0 upvotes · 0 comments - [Intelligent Spot Instance Availability: How Machine Learning Reduces Interruptions by up to 94%](https://daily.dev/posts/intelligent-spot-instance-availability-how-machine-learning-reduces-interruptions-by-up-to-94--jvmxlkdvd) · Cast AI · 1 upvotes · 0 comments --- Tags: [#aws](https://daily.dev/tags/aws), [#career](https://daily.dev/tags/career), [#backend](https://daily.dev/tags/backend) [View this post on daily.dev](https://daily.dev/posts/day-18-30-aws-system-design-patterns-had3njxtb) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/day-18-30-aws-system-design-patterns-had3njxtb","headline":"Day 18/30 AWS System Design Patterns","text":"A system design scenario presents a genomics data pipeline running nightly batch jobs on EC2 Spot Instances (c5.4xlarge, us-east-1a). When AWS reclaims 18 instances simultaneously at 2:17 AM, all running jobs are lost with no recovery, adding 7 hours to the pipeline. The scenario highlights three root causes: no checkpointing, single instance type, and single Availability Zone. Four answer options are evaluated, covering switching to On-Demand, implementing checkpointing with diversification, using max price bids, and switching to ECS Fargate Spot.","url":"https://daily.dev/posts/day-18-30-aws-system-design-patterns-had3njxtb","datePublished":"2026-08-07T18:04:04.981Z","dateModified":"2026-09-13T21:05:13.514Z","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0","description":"Principal Solution Architecture ","worksFor":{"@type":"Organization","name":"Metalab","logo":"https://www.google.com/s2/favicons?domain=metalab.com&sz=128"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":85140}},"image":"https://media.daily.dev/image/upload/s--BiB__Tlh--/f_auto/v1786125864/posts/Had3NJxtb?_a=BAMAMicg0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":67},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":20}],"comment":[{"@type":"Comment","text":"The answer is B.\nSpot interruptions are not avoidable — they are a property of the pricing model. The correct response is making jobs resilient to interruption, not eliminating Spot.\nThree changes together close the gap:\nFirst, checkpointing: jobs write their progress state to S3 (object storage, durable across instance loss) every N minutes. When an instance is reclaimed, the job restarts from the last checkpoint rather than from zero. A 4-hour job with 15-minute checkpoints loses at most 15 minutes of compute, not 4 hours.\nSecond, the EC2 Spot interruption notice (2-minute warning before reclamation): a lightweight agent polling the instance metadata endpoint can trigger one final checkpoint write when the notice arrives, reducing data loss to near zero.\nThird, diversification: configuring the fleet across c5.4xlarge, c5a.4xlarge, m5.4xlarge, and multiple AZs (Availability Zones — isolated failure domains) with capacity-optimized allocation strategy means a price spike in us-east-1a for c5.4xlarge does not reclaim the entire fleet. Different instance types and AZs have independent Spot capacity pools.","datePublished":"2026-08-07T18:05:19.242Z","url":"https://daily.dev/posts/Had3NJxtb#c-5wzxHmQ5U","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":6}},{"@type":"Comment","text":"A — Switching to On-Demand (fixed-price compute) eliminates interruption risk but costs 3–4× more per job. The correct answer preserves the Spot economics through resilience, not avoidance.","datePublished":"2026-08-07T18:05:26.071Z","url":"https://daily.dev/posts/Had3NJxtb#c-j2HDkLGWK","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2}},{"@type":"Comment","text":"Good things brother.\nThank you for sharing.","datePublished":"2026-08-08T02:55:37.268Z","url":"https://daily.dev/posts/Had3NJxtb#c-6IMw3kcq1","author":{"@type":"Person","name":"Md. Shafin Ahmed","url":"https://daily.dev/mdshafinahmed","image":"https://media.daily.dev/image/upload/s---a0uwu1i--/f_auto/v1783046504/avatars/avatar_u8iR8DacSYjsphyDqQTQ9?_a=BAMAMicg0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2}},{"@type":"Comment","text":"Also, it would mean a lot to me if you could support my content and stay in touch 🙏\n\nYouTube: https://www.youtube.com/@system-design-lab\nLinkedIn: https://www.linkedin.com/in/joud-awad/\nMedium Blog: https://joudwawad.medium.com/\nSubstack: https://joudawad.substack.com/","datePublished":"2026-08-07T18:04:32.962Z","url":"https://daily.dev/posts/Had3NJxtb#c-hE767T7TI","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}},{"@type":"Comment","text":"C — Setting a maximum Spot bid to the On-Demand price does not prevent interruption. Spot interruptions are driven by EC2 capacity demand. When AWS needs the capacity back, your instance is reclaimed regardless of your bid — up to and including the On-Demand price ceiling.","datePublished":"2026-08-07T18:05:31.845Z","url":"https://daily.dev/posts/Had3NJxtb#c-fUCGkcMGY","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh","name":"Joud Awad"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Joud Awad","item":"https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh"},{"@type":"ListItem","position":3,"name":"Day 18/30 AWS System Design Patterns"}]} ```