A data pipeline team runs nightly batch jobs on a fleet of EC2 Spot Instances (spare AWS compute capacity sold at discount — can be reclaimed by AWS with 2-minute notice when capacity is needed) — all c5.4xlarge in us-east-1a — processing genomics sequencing data. Each job runs for 3–5 hours. The fleet typically runs 40 instances overnight. Cost compared to On-Demand is about 70% lower.
Four weeks into production, EC2 reclaims Spot capacity in us-east-1a and takes 18 instances simultaneously at 2:17 AM. The jobs running on those instances are lost with no recovery. Rerunning the failed jobs adds 7 hours to the pipeline, causing results to miss the morning delivery window. The biology team escalates.
No checkpointing is implemented. Jobs restart from zero on failure. The fleet is configured with a single instance type in a single Availability Zone.
What combination of changes correctly addresses this?
A) Switch to On-Demand instances (fixed-price compute, no interruption risk) for the nightly pipeline — Spot is unsuitable for long-running batch workloads
B) Implement job checkpointing to S3 (object storage, durable across instance loss), diversify across multiple instance types and AZs (Availability Zones — isolated failure domains), and use the EC2 Spot interruption notice (2-minute warning sent before reclamation) to trigger a final checkpoint before termination
C) Use Spot with a maximum price bid set to the On-Demand price — this prevents interruption by ensuring you always outbid other buyers; but Spot interruptions are driven by capacity demand, not purely by price — AWS reclaims capacity regardless of your bid when it needs it back
D) Switch to ECS Fargate Spot (Fargate containers on Spot capacity — handles lifecycle events but does not implement checkpointing) — Fargate manages the Spot lifecycle automatically so interruptions are handled without custom checkpointing logic
Answer in the comments.
#SystemDesign #AWS #DataEngineering #DataPipeline #Scalability

Smiley Face

Joud Awad

A system design scenario presents a genomics data pipeline running nightly batch jobs on EC2 Spot Instances (c5.4xlarge, us-east-1a). When AWS reclaims 18 instances simultaneously at 2:17 AM, all running jobs are lost with no recovery, adding 7 hours to the pipeline. The scenario highlights three root causes: no checkpointing, single instance type, and single Availability Zone. Four answer options are evaluated, covering switching to On-Demand, implementing checkpointing with diversification, using max price bids, and switching to ECS Fargate Spot.

Day 18/30 AWS System Design Patterns

The answer is B.
Spot interruptions are not avoidable — they are a property of the pricing model. The correct response is making jobs resilient to interruption, not eliminating Spot.
Three changes together close the gap:
First, checkpointing: jobs write their progress state to S3 (object storage, durable across instance loss) every N minutes. When an instance is reclaimed, the job restarts from the last checkpoint rather than from zero. A 4-hour job with 15-minute checkpoints loses at most 15 minutes of compute, not 4 hours.
Second, the EC2 Spot interruption notice (2-minute warning before reclamation): a lightweight agent polling the instance metadata endpoint can trigger one final checkpoint write when the notice arrives, reducing data loss to near zero.
Third, diversification: configuring the fleet across c5.4xlarge, c5a.4xlarge, m5.4xlarge, and multiple AZs (Availability Zones — isolated failure domains) with <code>capacity-optimized</code> allocation strategy means a price spike in us-east-1a for c5.4xlarge does not reclaim the entire fleet. Different instance types and AZs have independent Spot capacity pools.
<img src="https://media.daily.dev/image/upload/s--cXsaYQqS--/f_auto/v1786125914/ugc/content_cf4fc131-5f0b-4741-9e11-4e6fb28da209?_a=BAMAMicg0" alt="ChatGPT Image Aug 7, 2026, 09_02_42 PM.png">

A — Switching to On-Demand (fixed-price compute) eliminates interruption risk but costs 3–4× more per job. The correct answer preserves the Spot economics through resilience, not avoidance.

Good things brother.
Thank you for sharing.

Also, it would mean a lot to me if you could support my content and stay in touch 🙏
<ul>
<li>YouTube: <a href="https://www.youtube.com/@system-design-lab" target="_blank" rel="noopener nofollow">https://www.youtube.com/@system-design-lab</a></li>
<li>LinkedIn: <a href="https://www.linkedin.com/in/joud-awad/" target="_blank" rel="noopener nofollow">https://www.linkedin.com/in/joud-awad/</a></li>
<li>Medium Blog: <a href="https://joudwawad.medium.com/" target="_blank" rel="noopener nofollow">https://joudwawad.medium.com/</a></li>
<li>Substack: <a href="https://joudawad.substack.com/" target="_blank" rel="noopener nofollow">https://joudawad.substack.com/</a></li>
</ul>

C — Setting a maximum Spot bid to the On-Demand price does not prevent interruption. Spot interruptions are driven by EC2 capacity demand. When AWS needs the capacity back, your instance is reclaimed regardless of your bid — up to and including the On-Demand price ceiling.