--- title: GPU Operator: the quiet hero behind Kubernetes GPUs description: A detailed walkthrough of how the NVIDIA GPU Operator enables Kubernetes to schedule GPU workloads. Covers the full setup flow: Node Feature Discovery (NFD)... canonical: https://daily.dev/posts/gpu-operator-the-quiet-hero-behind-kubernetes-gpus-psp0kummz twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: GPU Operator: the quiet hero behind Kubernetes GPUs | daily.dev og:description: A detailed walkthrough of how the NVIDIA GPU Operator enables Kubernetes to schedule GPU workloads. Covers the full setup flow: Node Feature Discovery (NFD)... og:url: https://daily.dev/posts/gpu-operator-the-quiet-hero-behind-kubernetes-gpus-psp0kummz og:image: https://api.daily.dev/og/posts/pSP0kumMZ.png og:image:alt: GPU Operator: the quiet hero behind Kubernetes GPUs og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # GPU Operator: the quiet hero behind Kubernetes GPUs **[Medium](https://daily.dev/sources/medium_js)** · 7 min read · 2 upvotes · 0 comments ## Summary A detailed walkthrough of how the NVIDIA GPU Operator enables Kubernetes to schedule GPU workloads. Covers the full setup flow: Node Feature Discovery (NFD) scanning hardware and labeling GPU nodes, GPU Operator deploying DaemonSets (nvidia-container-toolkit, nvidia-device-plugin, dcgm-exporter, cuda-validator), the container toolkit modifying containerd config to inject the NVIDIA runtime, the CUDA validator catching driver/library version mismatches before workloads run, and the device plugin advertising GPU resources to kubelet so the scheduler can treat GPUs like standard cluster resources. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What causes the 'Failed to initialize NVML: Driver/library version mismatch' error on an NVIDIA GPU node? It happens when the NVIDIA display driver version in the host kernel does not match the libnvidia-ml.so userspace library version, often because unattended OS updates bump the library independently of the driver. One observed case had driver version 580.126.09 paired with library version 580.159.03. Installing the driver via apt without apt-mark hold to pin the version is a common cause; using the GPU Operator's driver container avoids the mismatch entirely. _Anyone debugging GPU driver mismatches on Kubernetes nodes can find deeper explainers like this through daily.dev._ ### Why does the nvidia-device-plugin pod stay pending and the node show nvidia.com/gpu 0 in a Kubernetes GPU cluster? The device plugin pod is configured to wait until the nvidia-cuda-validator pod passes its driver/library compatibility check, so if the validator crashes with a version mismatch, the device plugin never registers with the local kubelet. As a result the node capacity stays at nvidia.com/gpu: 0, and any AI workload pods requesting a GPU remain in Pending state instead of being scheduled onto a broken node. _Teams troubleshooting stuck GPU scheduling in Kubernetes can track explanations like this via daily.dev._ ### What Kubernetes DaemonSets does the NVIDIA GPU Operator deploy on a GPU worker node? It deploys nvidia-container-toolkit, nvidia-device-plugin, dcgm-exporter, and cuda-validator as DaemonSets, triggered once Node Feature Discovery labels the node with a PCI vendor tag like feature.node.kubernetes.io/pci-10de.present=true for an NVIDIA card. The container toolkit then modifies containerd's config.toml to register an NVIDIA runtime handler so the node can run GPU workloads. _Engineers wiring up GPU nodes in Kubernetes can follow breakdowns like this on daily.dev._ ## Similar posts on daily.dev - [Best GPU Optimization Tools for Kubernetes and AI Workloads \(2026\)](https://daily.dev/posts/best-gpu-optimization-tools-for-kubernetes-and-ai-workloads-2026--xv0s5w89i) · Cast AI · 0 upvotes · 0 comments - [How to Run Isolated Tenant Kubernetes Clusters on Shared GPU Infrastructure](https://daily.dev/posts/how-to-run-isolated-tenant-kubernetes-clusters-on-shared-gpu-infrastructure-g9yhk1w1z) · NVIDIA Developer · 0 upvotes · 0 comments - [Stop wasting GPU allocation in Kubernetes with GPU-pruner](https://daily.dev/posts/stop-wasting-gpu-allocation-in-kubernetes-with-gpu-pruner-llycyb0ay) · Red Hat Developer · 0 upvotes · 0 comments - [Kubernetes GPU Challenges: What Platform Teams Face](https://daily.dev/posts/kubernetes-gpu-challenges-what-platform-teams-face-qil5lzjmq) · Devtron · 0 upvotes · 0 comments - [GPU autoscaling on Kubernetes with KEDA: Building an external scaler](https://daily.dev/posts/gpu-autoscaling-on-kubernetes-with-keda-building-an-external-scaler-a2w673l3g) · CNCF · 2 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#containers](https://daily.dev/tags/containers), [#nvidia](https://daily.dev/tags/nvidia) [View this post on daily.dev](https://daily.dev/posts/gpu-operator-the-quiet-hero-behind-kubernetes-gpus-psp0kummz) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"GPU Operator: the quiet hero behind Kubernetes GPUs","url":"https://daily.dev/posts/gpu-operator-the-quiet-hero-behind-kubernetes-gpus-psp0kummz","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/gpu-operator-the-quiet-hero-behind-kubernetes-gpus-psp0kummz"},"datePublished":"2026-07-23T19:33:43.416Z","dateModified":"2026-09-14T08:03:06.456Z","description":"A detailed walkthrough of how the NVIDIA GPU Operator enables Kubernetes to schedule GPU workloads. Covers the full setup flow: Node Feature Discovery (NFD)...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/c42b7f9d2d9eedd94057a562086e3278?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/c42b7f9d2d9eedd94057a562086e3278?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Medium","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Medium","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/medium","url":"https://daily.dev/sources/medium_js"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/gpu-operator-the-quiet-hero-behind-kubernetes-gpus-psp0kummz","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,containers,nvidia","timeRequired":"PT7M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Medium","item":"https://daily.dev/sources/medium_js"},{"@type":"ListItem","position":3,"name":"GPU Operator: the quiet hero behind Kubernetes GPUs"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/gpu-operator-the-quiet-hero-behind-kubernetes-gpus-psp0kummz#faq","mainEntity":[{"@type":"Question","name":"What causes the 'Failed to initialize NVML: Driver/library version mismatch' error on an NVIDIA GPU node?","acceptedAnswer":{"@type":"Answer","text":"It happens when the NVIDIA display driver version in the host kernel does not match the libnvidia-ml.so userspace library version, often because unattended OS updates bump the library independently of the driver. One observed case had driver version 580.126.09 paired with library version 580.159.03. Installing the driver via apt without apt-mark hold to pin the version is a common cause; using the GPU Operator's driver container avoids the mismatch entirely. Anyone debugging GPU driver mismatches on Kubernetes nodes can find deeper explainers like this through daily.dev."}},{"@type":"Question","name":"Why does the nvidia-device-plugin pod stay pending and the node show nvidia.com/gpu 0 in a Kubernetes GPU cluster?","acceptedAnswer":{"@type":"Answer","text":"The device plugin pod is configured to wait until the nvidia-cuda-validator pod passes its driver/library compatibility check, so if the validator crashes with a version mismatch, the device plugin never registers with the local kubelet. As a result the node capacity stays at nvidia.com/gpu: 0, and any AI workload pods requesting a GPU remain in Pending state instead of being scheduled onto a broken node. Teams troubleshooting stuck GPU scheduling in Kubernetes can track explanations like this via daily.dev."}},{"@type":"Question","name":"What Kubernetes DaemonSets does the NVIDIA GPU Operator deploy on a GPU worker node?","acceptedAnswer":{"@type":"Answer","text":"It deploys nvidia-container-toolkit, nvidia-device-plugin, dcgm-exporter, and cuda-validator as DaemonSets, triggered once Node Feature Discovery labels the node with a PCI vendor tag like feature.node.kubernetes.io/pci-10de.present=true for an NVIDIA card. The container toolkit then modifies containerd's config.toml to register an NVIDIA runtime handler so the node can run GPU workloads. Engineers wiring up GPU nodes in Kubernetes can follow breakdowns like this on daily.dev."}}]} ```