Kubernetes' default treatment of GPUs as opaque, indivisible integer resources leads to massive underutilization in AI inference fleets—cards sitting at 12-30% usage despite being fully 'scheduled'. The fix involves sharing physical GPUs via time-slicing, MPS, or MIG hardware partitioning depending on isolation needs, and scaling inference pods on queue depth and GPU utilization (via KEDA and DCGM/Prometheus) rather than CPU, since CPU is a decorrelated proxy for GPU saturation. Dynamic Resource Allocation (DRA) graduated to GA in Kubernetes v1.34 (September 2025) as a more expressive alternative to the device-plugin model, though fine-grained sharing features remain alpha/beta. Scale-to-zero and lazy-pull snapshotters introduce painful cold-start penalties because LLM weights must be streamed into VRAM; the recommended fix is keeping weights out of container images, streaming them from object storage, and maintaining a warm baseline of replicas for latency-sensitive paths.
Questions this post answers
Why does my Kubernetes GPU utilization stay so low even though every pod shows as scheduled and healthy?
Kubernetes typically assigns GPUs as an opaque integer resource (nvidia.com/gpu: 1), giving each pod an entire physical accelerator exclusively even when the workload only uses a fraction of it. Since the scheduler cannot see VRAM needs or compute fraction, cards commonly run at 12-30% utilization despite appearing fully allocated. Teams tuning GPU scheduling on kubernetes track patterns like this on daily.dev before rearchitecting inference infrastructure.
Should I use HPA or KEDA to autoscale GPU inference pods in Kubernetes?
Use KEDA rather than default HPA, because vanilla HPA scales on CPU utilization, which is a decorrelated proxy for GPU-bound inference workloads and can read healthy while the GPU is pinned at 100% and requests queue up. KEDA can read queue depth and GPU utilization (via DCGM exporter into Prometheus) and also supports scale-to-zero, which HPA cannot do. Engineers deciding between HPA and KEDA for GPU workloads follow comparisons like this on daily.dev.
What are the differences between time-slicing, MPS, and MIG for sharing an NVIDIA GPU in Kubernetes?
Time-slicing is pure software round-robin scheduling that works on any NVIDIA GPU but offers no memory or fault isolation, suiting dev and low-criticality work. MPS runs kernels concurrently across SMs for better throughput but still lacks memory isolation. MIG hardware-partitions Ampere-or-newer GPUs (A100, H100) into up to seven isolated instances with dedicated memory and compute, ideal for untrusted tenants needing predictable QoS, though profiles are static and planned in advance. Platform teams weighing GPU-sharing strategies reference breakdowns like this on daily.dev when planning capacity.