--- title: Auto repair for Together GPU Clusters: from GPU fault to... description: Together AI's auto repair system for GPU clusters moves a node from a detected fault to a healthy state with minimal manual diagnosis. It watches both inside... canonical: https://daily.dev/posts/auto-repair-for-together-gpu-clusters-from-gpu-fault-to-healthy-node-mgrm2ccnw twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Auto repair for Together GPU Clusters: from GPU fault to healthy node | daily.dev og:description: Together AI's auto repair system for GPU clusters moves a node from a detected fault to a healthy state with minimal manual diagnosis. It watches both inside... og:url: https://daily.dev/posts/auto-repair-for-together-gpu-clusters-from-gpu-fault-to-healthy-node-mgrm2ccnw og:image: https://api.daily.dev/og/posts/mgRM2CCNW.png og:image:alt: Auto repair for Together GPU Clusters: from GPU fault to healthy node og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Auto repair for Together GPU Clusters: from GPU fault to healthy node **[Together AI](https://daily.dev/sources/togetherai)** · 11 min read · 0 upvotes · 0 comments ## Summary Together AI's auto repair system for GPU clusters moves a node from a detected fault to a healthy state with minimal manual diagnosis. It watches both inside the VM (Xid errors, DCGM metrics, Node Problem Detector) and on the host (InfiniBand rail state, host reachability), then maps each fault to the lightest fix: reboot, quick reprovision, migrate, or remove. Users set per-fault, per-Xid-code consent for automatic repair and a wait-for-idle window, with a default max wait of 2 hours (cap 24). Repeated faults on the same machine escalate up the ladder, and a roughly 30-minute cooldown plus circuit breaker limit repairs during correlated outages. Vacated machines auto-revalidate with GPU, NCCL, InfiniBand and storage checks before rejoining the pool. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What does an Xid 79 error mean and how is it fixed on a GPU cluster? Xid 79 indicates a fault where the GPU needs to lose power to recover, something a VM reboot alone cannot achieve even if the hardware is otherwise fine. Together GPU Clusters addresses this with a preview host power cycle action that power-cycles the physical host without needing BMC credentials, guarded by a boot-ID check that prevents retrying a host that already came back online. _Teams tracking tricky GPU fault codes like this can follow infrastructure reliability coverage on daily.dev._ ### How long does automated node repair wait before forcibly interrupting a running GPU job? The wait-for-idle setting holds a repair until the node is actually idle, defaulting to a maximum wait of 2 hours with a cap of 24 hours, after which the repair proceeds regardless and the timeout is recorded. A 'do not interrupt running jobs' option removes the bound entirely, and operators are advised to set the wait to at least their checkpoint interval. _Engineers tuning GPU cluster resiliency policies can track practices like this through daily.dev._ ### How does automated infrastructure repair avoid mass node replacement during a correlated outage like a bad driver push or network partition? A circuit breaker trips when one cluster generates a burst of correlated automatic repairs, since many nodes failing at once signals a cluster-level cause that node-by-node repair cannot fix. Additional rails include a persistence requirement before a fault becomes a recommendation, a roughly 30-minute cooldown after any repair, and a concurrency budget sized from cluster node count. _Anyone designing safeguards against alert storms in cluster automation can follow related patterns on daily.dev._ ## Similar posts on daily.dev - [New in Together GPU Clusters: Reliability and control for production GPU clusters](https://daily.dev/posts/new-in-together-gpu-clusters-reliability-and-control-for-production-gpu-clusters-azg0okgzb) · Together AI · 0 upvotes · 0 comments - [Self-healing GPU nodes in Kubernetes: What we learned building the EKS node monitoring agent](https://daily.dev/posts/self-healing-gpu-nodes-in-kubernetes-what-we-learned-building-the-eks-node-monitoring-agent-nivfgvnnn) · The New Stack · 2 upvotes · 0 comments - [How we keep GPUs reliable across Databricks AI](https://daily.dev/posts/how-we-keep-gpus-reliable-across-databricks-ai-um3yzqfzg) · databricks · 0 upvotes · 0 comments - [Reliable GPU compute for large training & inference jobs](https://daily.dev/posts/reliable-gpu-compute-for-large-training-inference-jobs-pcelksbh8) · Netflix TechBlog · 0 upvotes · 0 comments - [Automate Kubernetes AI Cluster Health with NVSentinel](https://daily.dev/posts/automate-kubernetes-ai-cluster-health-with-nvsentinel-kfavzk0il) · NVIDIA Developer · 1 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#together-ai](https://daily.dev/tags/together-ai) [View this post on daily.dev](https://daily.dev/posts/auto-repair-for-together-gpu-clusters-from-gpu-fault-to-healthy-node-mgrm2ccnw) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Auto repair for Together GPU Clusters: from GPU fault to healthy node","url":"https://daily.dev/posts/auto-repair-for-together-gpu-clusters-from-gpu-fault-to-healthy-node-mgrm2ccnw","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/auto-repair-for-together-gpu-clusters-from-gpu-fault-to-healthy-node-mgrm2ccnw"},"datePublished":"2026-10-27T20:54:03.542Z","dateModified":"2026-10-27T21:14:59.800Z","description":"Together AI's auto repair system for GPU clusters moves a node from a detected fault to a healthy state with minimal manual diagnosis. It watches both inside...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/f078afa08c9c295a32c7c0a4634218ad?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/f078afa08c9c295a32c7c0a4634218ad?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Together AI","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Together AI","logo":"https://media.daily.dev/image/upload/s--tCjWcJfJ--/f_auto,q_auto/v1780213200/logos/togetherai?_a=BAMAMiWQ0","url":"https://daily.dev/sources/togetherai"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/auto-repair-for-together-gpu-clusters-from-gpu-fault-to-healthy-node-mgrm2ccnw","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,together-ai","timeRequired":"PT11M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Together AI","item":"https://daily.dev/sources/togetherai"},{"@type":"ListItem","position":3,"name":"Auto repair for Together GPU Clusters: from GPU fault to healthy node"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/auto-repair-for-together-gpu-clusters-from-gpu-fault-to-healthy-node-mgrm2ccnw#faq","mainEntity":[{"@type":"Question","name":"What does an Xid 79 error mean and how is it fixed on a GPU cluster?","acceptedAnswer":{"@type":"Answer","text":"Xid 79 indicates a fault where the GPU needs to lose power to recover, something a VM reboot alone cannot achieve even if the hardware is otherwise fine. Together GPU Clusters addresses this with a preview host power cycle action that power-cycles the physical host without needing BMC credentials, guarded by a boot-ID check that prevents retrying a host that already came back online. Teams tracking tricky GPU fault codes like this can follow infrastructure reliability coverage on daily.dev."}},{"@type":"Question","name":"How long does automated node repair wait before forcibly interrupting a running GPU job?","acceptedAnswer":{"@type":"Answer","text":"The wait-for-idle setting holds a repair until the node is actually idle, defaulting to a maximum wait of 2 hours with a cap of 24 hours, after which the repair proceeds regardless and the timeout is recorded. A 'do not interrupt running jobs' option removes the bound entirely, and operators are advised to set the wait to at least their checkpoint interval. Engineers tuning GPU cluster resiliency policies can track practices like this through daily.dev."}},{"@type":"Question","name":"How does automated infrastructure repair avoid mass node replacement during a correlated outage like a bad driver push or network partition?","acceptedAnswer":{"@type":"Answer","text":"A circuit breaker trips when one cluster generates a burst of correlated automatic repairs, since many nodes failing at once signals a cluster-level cause that node-by-node repair cannot fix. Additional rails include a persistence requirement before a fault becomes a recommendation, a roughly 30-minute cooldown after any repair, and a concurrency budget sized from cluster node count. Anyone designing safeguards against alert storms in cluster automation can follow related patterns on daily.dev."}}]} ```