--- title: Stop wasting GPU allocation in Kubernetes with GPU-pruner description: GPU-pruner is an open source Kubernetes controller that queries NVIDIA Data Center GPU Manager (DCGM) metrics through Prometheus to detect truly idle GPU... canonical: https://daily.dev/posts/stop-wasting-gpu-allocation-in-kubernetes-with-gpu-pruner-llycyb0ay twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Stop wasting GPU allocation in Kubernetes with GPU-pruner | daily.dev og:description: GPU-pruner is an open source Kubernetes controller that queries NVIDIA Data Center GPU Manager (DCGM) metrics through Prometheus to detect truly idle GPU... og:url: https://daily.dev/posts/stop-wasting-gpu-allocation-in-kubernetes-with-gpu-pruner-llycyb0ay og:image: https://api.daily.dev/og/posts/LLycyb0Ay.png og:image:alt: Stop wasting GPU allocation in Kubernetes with GPU-pruner og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Stop wasting GPU allocation in Kubernetes with GPU-pruner **[Red Hat Developer](https://daily.dev/sources/rhdev)** · 12 min read · 0 upvotes · 0 comments ## Summary GPU-pruner is an open source Kubernetes controller that queries NVIDIA Data Center GPU Manager (DCGM) metrics through Prometheus to detect truly idle GPU workloads and safely scales their parent resources to zero, rather than deleting them, to reclaim wasted GPU capacity. Unlike typical notebook cullers that watch browser sessions or API traffic, it monitors actual GPU engine utilization (defaulting to a 35-minute window and a 1% activity threshold), walking up ownerReferences to find the correct Deployment, StatefulSet, Kubeflow Notebook, or KServe InferenceService to pause. It includes Slack notifications with a grace period before scale-down, a dry-run mode, a web dashboard, and guidance for correctly configuring Prometheus ServiceMonitors with honorLabels to avoid label collisions. The tool integrates with Red Hat OpenShift AI's Kubeflow Notebooks and KServe endpoints, though the author's team has not yet deployed it in production. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How can I detect idle GPU workloads in Kubernetes that are still allocated but not doing any work? Query NVIDIA Data Center GPU Manager (DCGM) metrics through Prometheus rather than relying on session or browser activity. GPU-pruner, an open source Kubernetes controller, checks whether DCGM_FI_PROF_GR_ENGINE_ACTIVE and DCGM_FI_DEV_GPU_UTIL stay below a 1% threshold over a configurable observation window (35 minutes by default), then flags the workload as idle and scales its parent resource to zero. _Track tools like this on daily.dev if reclaiming wasted GPU capacity on your Kubernetes cluster is an ongoing headache._ ### Why does my Prometheus query for namespace="ml-workloads" return no results for DCGM GPU metrics? By default Prometheus sets honor_labels: false, which causes it to overwrite the namespace and pod labels from the DCGM exporter's raw metrics with its own service discovery values (typically the gpu-operator namespace and the exporter pod name), pushing the real values into exported_namespace and exported_pod instead. Setting honorLabels: true on the ServiceMonitor preserves the original telemetry labels so queries against the actual workload namespace work correctly. _Developers wiring up GPU or Prometheus monitoring can follow configuration gotchas like this one on daily.dev._ ### How does GPU-pruner decide which Kubernetes resource to scale down after finding an idle pod? It walks up the pod's Kubernetes ownerReferences hierarchy, or parses KServe-specific labeling schemes, to trace the pod back to its root controller, such as a Deployment, StatefulSet, Kubeflow Notebook, or KServe InferenceService. It then scales that parent resource's replica count to zero instead of deleting the pod, preserving the workload's configuration so it can be restored later. _Anyone automating GPU capacity reclamation on Kubernetes can follow tools like this via daily.dev._ ## Similar posts on daily.dev - [Get Real-Time Visibility into GPU Usage Across Kubernetes Clusters](https://daily.dev/posts/get-real-time-visibility-into-gpu-usage-across-kubernetes-clusters-ccg90khzv) · NVIDIA Developer · 3 upvotes · 0 comments - [Making GPU Clusters More Efficient with NVIDIA Data Center Monitoring](https://daily.dev/posts/making-gpu-clusters-more-efficient-with-nvidia-data-center-monitoring-hxuekxyge) · NVIDIA Developer · 1 upvotes · 0 comments - [Reclaiming underutilized GPUs in Kubernetes using scheduler plugins](https://daily.dev/posts/reclaiming-underutilized-gpus-in-kubernetes-using-scheduler-plugins-upxqnb5wj) · CNCF · 0 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#prometheus](https://daily.dev/tags/prometheus), [#mlops](https://daily.dev/tags/mlops) [View this post on daily.dev](https://daily.dev/posts/stop-wasting-gpu-allocation-in-kubernetes-with-gpu-pruner-llycyb0ay) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Stop wasting GPU allocation in Kubernetes with GPU-pruner","url":"https://daily.dev/posts/stop-wasting-gpu-allocation-in-kubernetes-with-gpu-pruner-llycyb0ay","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/stop-wasting-gpu-allocation-in-kubernetes-with-gpu-pruner-llycyb0ay"},"datePublished":"2026-08-31T03:27:08.917Z","dateModified":"2026-08-31T03:27:36.570Z","description":"GPU-pruner is an open source Kubernetes controller that queries NVIDIA Data Center GPU Manager (DCGM) metrics through Prometheus to detect truly idle GPU...","image":"https://media.daily.dev/image/upload/s--qPvKM23u--/f_auto/v1722860399/public/Placeholder%2009","thumbnailUrl":"https://media.daily.dev/image/upload/s--qPvKM23u--/f_auto/v1722860399/public/Placeholder%2009","isAccessibleForFree":true,"articleSection":"Red Hat Developer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Red Hat Developer","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/2a8895f3216c4900bdc455585a4ad3e4","url":"https://daily.dev/sources/rhdev"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/stop-wasting-gpu-allocation-in-kubernetes-with-gpu-pruner-llycyb0ay","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,prometheus,mlops","timeRequired":"PT12M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Red Hat Developer","item":"https://daily.dev/sources/rhdev"},{"@type":"ListItem","position":3,"name":"Stop wasting GPU allocation in Kubernetes with GPU-pruner"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/stop-wasting-gpu-allocation-in-kubernetes-with-gpu-pruner-llycyb0ay#faq","mainEntity":[{"@type":"Question","name":"How can I detect idle GPU workloads in Kubernetes that are still allocated but not doing any work?","acceptedAnswer":{"@type":"Answer","text":"Query NVIDIA Data Center GPU Manager (DCGM) metrics through Prometheus rather than relying on session or browser activity. GPU-pruner, an open source Kubernetes controller, checks whether DCGM_FI_PROF_GR_ENGINE_ACTIVE and DCGM_FI_DEV_GPU_UTIL stay below a 1% threshold over a configurable observation window (35 minutes by default), then flags the workload as idle and scales its parent resource to zero. Track tools like this on daily.dev if reclaiming wasted GPU capacity on your Kubernetes cluster is an ongoing headache."}},{"@type":"Question","name":"Why does my Prometheus query for namespace=\"ml-workloads\" return no results for DCGM GPU metrics?","acceptedAnswer":{"@type":"Answer","text":"By default Prometheus sets honor_labels: false, which causes it to overwrite the namespace and pod labels from the DCGM exporter's raw metrics with its own service discovery values (typically the gpu-operator namespace and the exporter pod name), pushing the real values into exported_namespace and exported_pod instead. Setting honorLabels: true on the ServiceMonitor preserves the original telemetry labels so queries against the actual workload namespace work correctly. Developers wiring up GPU or Prometheus monitoring can follow configuration gotchas like this one on daily.dev."}},{"@type":"Question","name":"How does GPU-pruner decide which Kubernetes resource to scale down after finding an idle pod?","acceptedAnswer":{"@type":"Answer","text":"It walks up the pod's Kubernetes ownerReferences hierarchy, or parses KServe-specific labeling schemes, to trace the pod back to its root controller, such as a Deployment, StatefulSet, Kubeflow Notebook, or KServe InferenceService. It then scales that parent resource's replica count to zero instead of deleting the pod, preserving the workload's configuration so it can be restored later. Anyone automating GPU capacity reclamation on Kubernetes can follow tools like this via daily.dev."}}]} ```