Kubernetes Spot Instances offer 60-90% compute cost savings but require careful configuration to avoid reliability issues. Teams running Spot-heavy setups save 77% on average vs 59% for mixed fleets, per the 2025 Cast AI Kubernetes Cost Benchmark. Key safety mechanisms include PodDisruptionBudgets (critical to prevent simultaneous replica eviction), correct terminationGracePeriodSeconds tuned per cloud (90s for AWS, 25s for GCP/Azure), and AWS Node Termination Handler configured for your node group type. Karpenter users get native Spot interruption handling without NTH, plus SpotToSpotConsolidation (v0.34.0+) for continuous cost optimization. Workload suitability matters: stateless services, batch jobs, CI/CD runners, and checkpointed ML training are safe; stateful databases, control-plane components, and payment processors are not. The post also covers cloud-specific Spot market behavior, GCP Preemptible vs Spot VM differences, and how Cast AI automates predictive rebalancing 1-3 hours ahead of interruptions.
Table of contents
Key takeawaysWhy Spot cuts cost so sharplyWhich workloads suit SpotInterruption handling and fallbackSpot with KarpenterHow Cast AI handles Spot at scaleConclusionFrequently Asked QuestionsQuestions this post answers
Do I need AWS Node Termination Handler if I use Karpenter on EKS?
No, Karpenter replaces NTH entirely. Karpenter watches for Spot interruption notices via EventBridge and IMDS, cordons the affected node, and provisions a replacement natively without a separate DaemonSet. NTH is only needed for clusters using Cluster Autoscaler or self-managed node groups — use IMDS polling mode for self-managed groups, or Queue Processor mode (requiring EventBridge and SQS) for EKS managed node groups. Teams migrating from Cluster Autoscaler to Karpenter track these operational differences on daily.dev before cutting over.
What terminationGracePeriodSeconds should I set for pods running on GCP Spot VMs vs AWS Spot?
For AWS Spot, set terminationGracePeriodSeconds to 90 seconds — this fits within the 2-minute interruption notice and leaves room for a 5-second preStop hook plus 85 seconds of request draining. For GCP Spot VMs, use 25 seconds with a 2-second preStop hook, staying within the 30-second best-effort notice. GCP Preemptible VMs have an effective hard shutdown at ~15 seconds, so minimize shutdown logic there. Engineers tuning Kubernetes pod shutdown for Spot workloads across clouds find the latest provider behavior changes on daily.dev.
What happens if I set maxUnavailable: 0 in a PodDisruptionBudget for a Spot workload?
Setting maxUnavailable: 0 blocks drain operations entirely. When AWS Node Termination Handler cordons a node and issues eviction requests, Kubernetes checks the PDB first — if evicting any pod would violate the budget, it waits indefinitely. On a Spot node about to be reclaimed, this means pods will not move before the instance terminates, causing an uncontrolled outage. Use minAvailable: 1 instead, which allows drain to proceed while keeping one replica running. Kubernetes platform teams avoiding PDB misconfigurations in production Spot setups share patterns like this on daily.dev.