--- title: When Kubeflow meets Cilium: Debugging 60% idle GPUs in... description: A distributed training job on Kubernetes showed 60%+ GPU idle rates despite all pods running healthy. The root cause was a mismatch between Kubernetes'... canonical: https://daily.dev/posts/when-kubeflow-meets-cilium-debugging-60-idle-gpus-in-kubernetes-h0d49snrn twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: When Kubeflow meets Cilium: Debugging 60% idle GPUs in Kubernetes | daily.dev og:description: A distributed training job on Kubernetes showed 60%+ GPU idle rates despite all pods running healthy. The root cause was a mismatch between Kubernetes'... og:url: https://daily.dev/posts/when-kubeflow-meets-cilium-debugging-60-idle-gpus-in-kubernetes-h0d49snrn og:image: https://api.daily.dev/og/posts/h0D49SNRn.png og:image:alt: When Kubeflow meets Cilium: Debugging 60% idle GPUs in Kubernetes og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # When Kubeflow meets Cilium: Debugging 60% idle GPUs in Kubernetes **[CNCF](https://daily.dev/sources/cncf)** · 5 min read · 3 upvotes · 0 comments ## Summary A distributed training job on Kubernetes showed 60%+ GPU idle rates despite all pods running healthy. The root cause was a mismatch between Kubernetes' topology-agnostic scheduling and Cilium's zone-boundary network policies: the training coordinator landed in a different availability zone than the GPU workers, and the network policy silently blocked communication. The fix required no changes to Cilium or Kubeflow — just adding nodeAffinity, topologySpreadConstraints, and tolerations to the workload spec to co-locate all pods in the same zone. This resolved all three failure modes: hard network blocks, cross-zone latency degrading throughput 30–60%, and unexpected inter-AZ egress costs. GPU utilization jumped from ~40% to ~85%. The pattern generalizes to any topology-aware CNI combined with any distributed ML framework. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why are my GPUs sitting idle even though all Kubeflow training pods show healthy and running? The training coordinator and GPU workers likely landed in different availability zones, and a Cilium zone-boundary network policy is silently blocking the connection between them because Kubernetes scheduling is topology-agnostic while Cilium enforces topology-aware isolation. This can appear as a hard connection block, a 30-60% throughput drop from cross-zone latency, or unexpected cross-AZ egress costs, with no errors logged. _daily.dev surfaces practical fixes like this for engineers chasing silent GPU utilization problems._ ### How do I fix Kubernetes pods being scheduled across zones when using Cilium network policies with Kubeflow? Add nodeAffinity to pin the training group to one GPU zone, topologySpreadConstraints to co-locate the coordinator and workers, and a toleration so the coordinator can land on GPU-tainted nodes; alternatively use podAffinity with a zone topology key to place workers wherever the coordinator lands, without hard-coding a zone name. No changes to Cilium, Kubernetes, or Kubeflow are needed. _For teams tuning GPU cluster topology, daily.dev tracks fixes like this affinity-based scheduling approach._ ### How much can co-locating training pods in the same zone improve GPU utilization? GPU utilization increased from around 40% to around 85% after the coordinator and GPU workers were co-located in the same availability zone, eliminating the cross-zone network policy block, the latency from cross-zone gradient synchronization, and the associated inter-AZ egress costs simultaneously with a single fix. _Engineers optimizing GPU spend can follow real-world results like this utilization jump on daily.dev._ ## Similar posts on daily.dev - [GPU Scheduling and Bin-Packing in Kubernetes: Pack More AI onto Every GPU](https://daily.dev/posts/gpu-scheduling-and-bin-packing-in-kubernetes-pack-more-ai-onto-every-gpu-6nuaafnff) · Cast AI · 0 upvotes · 0 comments - [Multi AZ by Default, Local by Design, Cutting Cross Zone Traffic With Cilium](https://daily.dev/posts/multi-az-by-default-local-by-design-cutting-cross-zone-traffic-with-cilium-can44cqgu) · Isovalent · 1 upvotes · 0 comments - [Day 2 with Cilium: Small configurations that keep large clusters boring](https://daily.dev/posts/day-2-with-cilium-small-configurations-that-keep-large-clusters-boring-qzugybgcw) · cilium · 4 upvotes · 0 comments - [The cluster that runs on our own metal: rebuilding survey processing on hybrid Kubernetes](https://daily.dev/posts/the-cluster-that-runs-on-our-own-metal-rebuilding-survey-processing-on-hybrid-kubernetes-4inmn4eto) · Medium · 0 upvotes · 0 comments - [GPU Time-Slicing for Concurrent LLM Agents on Kubernetes](https://daily.dev/posts/gpu-time-slicing-for-concurrent-llm-agents-on-kubernetes-guzcjf1ji) · Towards Data Science · 0 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#gpu](https://daily.dev/tags/gpu), [#cilium](https://daily.dev/tags/cilium) [View this post on daily.dev](https://daily.dev/posts/when-kubeflow-meets-cilium-debugging-60-idle-gpus-in-kubernetes-h0d49snrn) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"When Kubeflow meets Cilium: Debugging 60% idle GPUs in Kubernetes","url":"https://daily.dev/posts/when-kubeflow-meets-cilium-debugging-60-idle-gpus-in-kubernetes-h0d49snrn","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/when-kubeflow-meets-cilium-debugging-60-idle-gpus-in-kubernetes-h0d49snrn"},"datePublished":"2026-07-23T11:33:55.992Z","dateModified":"2026-09-13T19:41:41.771Z","description":"A distributed training job on Kubernetes showed 60%+ GPU idle rates despite all pods running healthy. The root cause was a mismatch between Kubernetes'...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/be8e10060f47df68d9f379f20df66b38?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/be8e10060f47df68d9f379f20df66b38?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"CNCF","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"CNCF","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/55536a03390741eebd96d2f9b74d2d8d","url":"https://daily.dev/sources/cncf"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/when-kubeflow-meets-cilium-debugging-60-idle-gpus-in-kubernetes-h0d49snrn","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,gpu,cilium","timeRequired":"PT5M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"CNCF","item":"https://daily.dev/sources/cncf"},{"@type":"ListItem","position":3,"name":"When Kubeflow meets Cilium: Debugging 60% idle GPUs in Kubernetes"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/when-kubeflow-meets-cilium-debugging-60-idle-gpus-in-kubernetes-h0d49snrn#faq","mainEntity":[{"@type":"Question","name":"Why are my GPUs sitting idle even though all Kubeflow training pods show healthy and running?","acceptedAnswer":{"@type":"Answer","text":"The training coordinator and GPU workers likely landed in different availability zones, and a Cilium zone-boundary network policy is silently blocking the connection between them because Kubernetes scheduling is topology-agnostic while Cilium enforces topology-aware isolation. This can appear as a hard connection block, a 30-60% throughput drop from cross-zone latency, or unexpected cross-AZ egress costs, with no errors logged. daily.dev surfaces practical fixes like this for engineers chasing silent GPU utilization problems."}},{"@type":"Question","name":"How do I fix Kubernetes pods being scheduled across zones when using Cilium network policies with Kubeflow?","acceptedAnswer":{"@type":"Answer","text":"Add nodeAffinity to pin the training group to one GPU zone, topologySpreadConstraints to co-locate the coordinator and workers, and a toleration so the coordinator can land on GPU-tainted nodes; alternatively use podAffinity with a zone topology key to place workers wherever the coordinator lands, without hard-coding a zone name. No changes to Cilium, Kubernetes, or Kubeflow are needed. For teams tuning GPU cluster topology, daily.dev tracks fixes like this affinity-based scheduling approach."}},{"@type":"Question","name":"How much can co-locating training pods in the same zone improve GPU utilization?","acceptedAnswer":{"@type":"Answer","text":"GPU utilization increased from around 40% to around 85% after the coordinator and GPU workers were co-located in the same availability zone, eliminating the cross-zone network policy block, the latency from cross-zone gradient synchronization, and the associated inter-AZ egress costs simultaneously with a single fix. Engineers optimizing GPU spend can follow real-world results like this utilization jump on daily.dev."}}]} ```