GPU utilization in production Kubernetes clusters averages just 5%, yet GPU costs are rising. This deep-dive covers the four main GPU cost leaks (idle nodes, oversized allocation, serial workloads, on-demand-only usage) and maps specific tools to each fix. The GPU Cost Optimization Loop (Measure → Allocate → Share → Automate) requires NVIDIA DCGM Exporter for observability, GPU Operator with MIG and time-slicing for partitioning and sharing, Karpenter for node lifecycle automation, and Cast AI as an orchestration layer tying everything together. Detailed YAML configs are provided for Karpenter NodePools, time-slicing ConfigMaps, and GPU pod specs. The post also compares optimization strategies for inference vs. training workloads, covering Spot instance savings (60–91% on AWS), MIG partition profiles for A100/H100, and network fabric requirements for multi-node distributed training.
