--- title: "Stop Treating GPUs Like Web Pods" url: https://daily.dev/posts/stop-treating-gpus-like-web-pods-qjygu2zrf source_url: https://cloudnativenow.com/contributed-content/stop-treating-gpus-like-web-pods type: article source: "Container Journal" published: 2026-08-12T11:42:59.953Z updated: 2026-08-12T13:22:02.989Z tags: ["kubernetes", "ai-inference"] reading_time: 8 upvotes: 1 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Stop Treating GPUs Like Web Pods **[Container Journal](https://daily.dev/sources/container_journal)** · 8 min read · 1 upvotes · 0 comments ## Summary Kubernetes' default treatment of GPUs as opaque, indivisible integer resources leads to massive underutilization in AI inference fleets—cards sitting at 12-30% usage despite being fully 'scheduled'. The fix involves sharing physical GPUs via time-slicing, MPS, or MIG hardware partitioning depending on isolation needs, and scaling inference pods on queue depth and GPU utilization (via KEDA and DCGM/Prometheus) rather than CPU, since CPU is a decorrelated proxy for GPU saturation. Dynamic Resource Allocation (DRA) graduated to GA in Kubernetes v1.34 (September 2025) as a more expressive alternative to the device-plugin model, though fine-grained sharing features remain alpha/beta. Scale-to-zero and lazy-pull snapshotters introduce painful cold-start penalties because LLM weights must be streamed into VRAM; the recommended fix is keeping weights out of container images, streaming them from object storage, and maintaining a warm baseline of replicas for latency-sensitive paths. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does my Kubernetes GPU utilization stay so low even though every pod shows as scheduled and healthy? Kubernetes typically assigns GPUs as an opaque integer resource (nvidia.com/gpu: 1), giving each pod an entire physical accelerator exclusively even when the workload only uses a fraction of it. Since the scheduler cannot see VRAM needs or compute fraction, cards commonly run at 12-30% utilization despite appearing fully allocated. _Teams tuning GPU scheduling on kubernetes track patterns like this on daily.dev before rearchitecting inference infrastructure._ ### Should I use HPA or KEDA to autoscale GPU inference pods in Kubernetes? Use KEDA rather than default HPA, because vanilla HPA scales on CPU utilization, which is a decorrelated proxy for GPU-bound inference workloads and can read healthy while the GPU is pinned at 100% and requests queue up. KEDA can read queue depth and GPU utilization (via DCGM exporter into Prometheus) and also supports scale-to-zero, which HPA cannot do. _Engineers deciding between HPA and KEDA for GPU workloads follow comparisons like this on daily.dev._ ### What are the differences between time-slicing, MPS, and MIG for sharing an NVIDIA GPU in Kubernetes? Time-slicing is pure software round-robin scheduling that works on any NVIDIA GPU but offers no memory or fault isolation, suiting dev and low-criticality work. MPS runs kernels concurrently across SMs for better throughput but still lacks memory isolation. MIG hardware-partitions Ampere-or-newer GPUs (A100, H100) into up to seven isolated instances with dedicated memory and compute, ideal for untrusted tenants needing predictable QoS, though profiles are static and planned in advance. _Platform teams weighing GPU-sharing strategies reference breakdowns like this on daily.dev when planning capacity._ ## Similar posts on daily.dev - [GPU Scheduling and Bin-Packing in Kubernetes: Pack More AI onto Every GPU](https://daily.dev/posts/gpu-scheduling-and-bin-packing-in-kubernetes-pack-more-ai-onto-every-gpu-6nuaafnff) · Cast AI · 0 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#ai-inference](https://daily.dev/tags/ai-inference) [View this post on daily.dev](https://daily.dev/posts/stop-treating-gpus-like-web-pods-qjygu2zrf)