--- title: Self-healing GPU nodes in Kubernetes: What we learned... description: AWS engineers share six hard-won lessons from building the EKS Node Monitoring Agent, an open-source tool that enables self-healing GPU nodes in Kubernetes.... canonical: https://daily.dev/posts/self-healing-gpu-nodes-in-kubernetes-what-we-learned-building-the-eks-node-monitoring-agent-nivfgvnnn twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Self-healing GPU nodes in Kubernetes: What we learned building the EKS node monitoring agent | daily.dev og:description: AWS engineers share six hard-won lessons from building the EKS Node Monitoring Agent, an open-source tool that enables self-healing GPU nodes in Kubernetes.... og:url: https://daily.dev/posts/self-healing-gpu-nodes-in-kubernetes-what-we-learned-building-the-eks-node-monitoring-agent-nivfgvnnn og:image: https://api.daily.dev/og/posts/NiVfGvNnn.png og:image:alt: Self-healing GPU nodes in Kubernetes: What we learned building the EKS node monitoring agent og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Self-healing GPU nodes in Kubernetes: What we learned building the EKS node monitoring agent **[The New Stack](https://daily.dev/sources/newstack)** · 16 min read · 2 upvotes · 0 comments ## Summary AWS engineers share six hard-won lessons from building the EKS Node Monitoring Agent, an open-source tool that enables self-healing GPU nodes in Kubernetes. The agent detects hardware failures (GPU PCIe drops, XID errors, NVLink failures, container runtime wedges) and writes Kubernetes NodeConditions that Karpenter uses to automatically replace degraded nodes. Key lessons include treating reason codes as versioned API contracts, distinguishing absent from unknown monitor states, separating detection from diagnosis, and avoiding goroutine burst patterns that caused measurable throughput loss in distributed GPU training workloads. The repair cycle runs end-to-end in under 12 minutes with safety gates preventing mass replacement during correlated fleet events. On EKS Auto Mode, the entire system runs out of the box with no operator intervention required. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is the difference between condition severity and event severity in the EKS Node Monitoring Agent? Condition severity marks a terminal, non-recoverable fault (like GPU device-count mismatch or NVLink failure), flips the NodeCondition to False, and makes the node eligible for automatic replacement by Karpenter. Event severity is informational only, posting a Kubernetes event while the NodeCondition stays True, covering signals like EBS IOPS throttling or GPU thermal warnings that may be transient or application-induced rather than terminal hardware faults. _Teams tuning auto-repair thresholds can track these architecture patterns as they evolve on daily.dev._ ### How long does it take for Karpenter to replace a failed GPU node in EKS Auto Mode after a fault is detected? The full cycle from fault injection to a replacement node running workloads took under 12 minutes in testing. Detection of critical GPU faults via DCGM's push-based policy channel landed in under a second, followed by a 10-minute toleration window for accelerated hardware faults (versus 30 minutes for other fault types), then roughly 90 seconds for the replacement instance to launch and register. _Engineers designing GPU fleet resilience can follow real-world repair timing benchmarks like this on daily.dev._ ### Why did a node health monitoring agent slow down distributed GPU training jobs on EKS? The EKS Node Monitoring Agent's independent goroutines had polling intervals that periodically aligned, causing dozens of monitors to wake simultaneously and burst CPU usage across many cores at once. In NCCL-based distributed training, where every node in a communication group must complete its step before others proceed, this microsecond-scale jitter cascaded into measurable throughput loss, prompting the customer to disable the agent entirely until a startup jitter fix staggered polling times. _Anyone debugging mysterious GPU training slowdowns can compare notes on host-level interference via daily.dev._ ## Similar posts on daily.dev - [Amazon ECS now auto-repairs failing GPUs and instances. Here’s why it matters for SREs.](https://daily.dev/posts/amazon-ecs-now-auto-repairs-failing-gpus-and-instances-here-s-why-it-matters-for-sres--l6e6gjyt6) · The New Stack · 3 upvotes · 0 comments - [Automate Kubernetes AI Cluster Health with NVSentinel](https://daily.dev/posts/automate-kubernetes-ai-cluster-health-with-nvsentinel-kfavzk0il) · NVIDIA Developer · 1 upvotes · 0 comments - [How Does Kubernetes Self-Healing Work? Understand Self-Healing By Breaking a Real Cluster](https://daily.dev/posts/how-does-kubernetes-self-healing-work-understand-self-healing-by-breaking-a-real-cluster-hsytfe0nv) · freeCodeCamp · 1 upvotes · 0 comments - [Self-Healing Kubernetes Gets Real — and Risky: Running AI Agents on Amazon EKS](https://daily.dev/posts/self-healing-kubernetes-gets-real-and-risky-running-ai-agents-on-amazon-eks-0cqww64he) · Cloud Native Now · 0 upvotes · 0 comments - [Optimize EKS operations with agents: Reduce MTTR with AWS DevOps Agent and a Kubernetes Operator](https://daily.dev/posts/optimize-eks-operations-with-agents-reduce-mttr-with-aws-devops-agent-and-a-kubernetes-operator-98hxetg2q) · AWS · 0 upvotes · 0 comments --- Tags: [#aws](https://daily.dev/tags/aws), [#kubernetes](https://daily.dev/tags/kubernetes), [#gpu](https://daily.dev/tags/gpu) [View this post on daily.dev](https://daily.dev/posts/self-healing-gpu-nodes-in-kubernetes-what-we-learned-building-the-eks-node-monitoring-agent-nivfgvnnn) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Self-healing GPU nodes in Kubernetes: What we learned building the EKS node monitoring agent","url":"https://daily.dev/posts/self-healing-gpu-nodes-in-kubernetes-what-we-learned-building-the-eks-node-monitoring-agent-nivfgvnnn","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/self-healing-gpu-nodes-in-kubernetes-what-we-learned-building-the-eks-node-monitoring-agent-nivfgvnnn"},"datePublished":"2026-07-19T13:06:48.993Z","dateModified":"2026-09-14T06:26:28.328Z","description":"AWS engineers share six hard-won lessons from building the EKS Node Monitoring Agent, an open-source tool that enables self-healing GPU nodes in Kubernetes....","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/717a928ef8d9303672bd1de1d86376ea?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/717a928ef8d9303672bd1de1d86376ea?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"The New Stack","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"The New Stack","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/newstack","url":"https://daily.dev/sources/newstack"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/self-healing-gpu-nodes-in-kubernetes-what-we-learned-building-the-eks-node-monitoring-agent-nivfgvnnn","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"aws,kubernetes,gpu","timeRequired":"PT16M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"The New Stack","item":"https://daily.dev/sources/newstack"},{"@type":"ListItem","position":3,"name":"Self-healing GPU nodes in Kubernetes: What we learned building the EKS node monitoring agent"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/self-healing-gpu-nodes-in-kubernetes-what-we-learned-building-the-eks-node-monitoring-agent-nivfgvnnn#faq","mainEntity":[{"@type":"Question","name":"What is the difference between condition severity and event severity in the EKS Node Monitoring Agent?","acceptedAnswer":{"@type":"Answer","text":"Condition severity marks a terminal, non-recoverable fault (like GPU device-count mismatch or NVLink failure), flips the NodeCondition to False, and makes the node eligible for automatic replacement by Karpenter. Event severity is informational only, posting a Kubernetes event while the NodeCondition stays True, covering signals like EBS IOPS throttling or GPU thermal warnings that may be transient or application-induced rather than terminal hardware faults. Teams tuning auto-repair thresholds can track these architecture patterns as they evolve on daily.dev."}},{"@type":"Question","name":"How long does it take for Karpenter to replace a failed GPU node in EKS Auto Mode after a fault is detected?","acceptedAnswer":{"@type":"Answer","text":"The full cycle from fault injection to a replacement node running workloads took under 12 minutes in testing. Detection of critical GPU faults via DCGM's push-based policy channel landed in under a second, followed by a 10-minute toleration window for accelerated hardware faults (versus 30 minutes for other fault types), then roughly 90 seconds for the replacement instance to launch and register. Engineers designing GPU fleet resilience can follow real-world repair timing benchmarks like this on daily.dev."}},{"@type":"Question","name":"Why did a node health monitoring agent slow down distributed GPU training jobs on EKS?","acceptedAnswer":{"@type":"Answer","text":"The EKS Node Monitoring Agent's independent goroutines had polling intervals that periodically aligned, causing dozens of monitors to wake simultaneously and burst CPU usage across many cores at once. In NCCL-based distributed training, where every node in a communication group must complete its step before others proceed, this microsecond-scale jitter cascaded into measurable throughput loss, prompting the customer to disable the agent entirely until a startup jitter fix staggered polling times. Anyone debugging mysterious GPU training slowdowns can compare notes on host-level interference via daily.dev."}}]} ```