Say goodbye to K8s GPU pain: How DRA changes everything
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
Kubernetes historically treated all GPUs as identical units, causing OOM errors when jobs landed on wrong hardware and MIG slice waste when small slices exhausted while large ones sat idle. Kubernetes 1.34 introduces Dynamic Resource Allocation (DRA), which lets workloads express hardware requirements using Common Expression Language (CEL) selectors. Three concrete YAML examples show how to request an H100-or-better with 40GB+ memory, implement flexible MIG fallback logic (small → medium → full GPU), and request four NVLink-connected GPUs for distributed training. DRA eliminates fragile node selectors, hardcoded hardware assumptions in Helm charts, and the need to maintain separate node pools per GPU generation.
Table of contents
The root of the problemThe MIG illusionDynamic Resource Allocation (DRA)The engineering takeawayQuestions this post answers
How do I request a specific GPU type with minimum memory in Kubernetes using DRA?
With DRA in Kubernetes 1.34+, you define a ResourceClaimTemplate using CEL selectors. For example, to request an H100 or newer GPU with at least 40GB memory, set the selector expression to check device.attributes["gpu.nvidia.com"].memory >= quantity("40Gi") and device.attributes["gpu.nvidia.com"].generation in ["H100","B200","B300"]. The Job then references the template via resourceClaimTemplateName, and the scheduler matches it to a qualifying device automatically. Platform engineers managing mixed GPU fleets track DRA patterns and Kubernetes scheduling changes on daily.dev.
How can I configure MIG fallback logic in Kubernetes so a job uses a small slice but falls back to a larger one if unavailable?
DRA ResourceClaimTemplates support a CEL selector that lists multiple acceptable MIG profiles in a single expression. Set the expression to check device.attributes["gpu.nvidia.com"].profile in ["mig-1g.10gb","mig-2g.20gb","mig-3g.40gb","full-gpu"]. The scheduler picks the first available match, eliminating the rigid per-profile resource types that previously left large slices idle when small ones were exhausted. Teams optimizing GPU utilization across inference workloads find relevant scheduling techniques on daily.dev.
How do I schedule a distributed training job on 4 NVLink-connected GPUs in Kubernetes?
Using DRA in Kubernetes 1.34+, define a ResourceClaimTemplate requesting count: 4 GPUs with a CEL selector filtering for device.attributes["gpu.nvidia.com"].fabric == "nvlink" and a constraints block with matchAttribute: "gpu.nvidia.com/nvlinkDomain" to ensure all four GPUs share the same NVLink domain. The Job references this template, and the scheduler places the pod only where topology constraints are satisfied. Developers building distributed training infrastructure on Kubernetes follow GPU topology patterns on daily.dev.