--- title: Keep Distributed Training Running with Automated... description: NVIDIA Recovery is a new open-source service that automates recovery for distributed AI training jobs across a cluster, released under DSX OS. It detects... canonical: https://daily.dev/posts/keep-distributed-training-running-with-automated-workload-recovery-m9tvrppp1 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Keep Distributed Training Running with Automated Workload Recovery | daily.dev og:description: NVIDIA Recovery is a new open-source service that automates recovery for distributed AI training jobs across a cluster, released under DSX OS. It detects... og:url: https://daily.dev/posts/keep-distributed-training-running-with-automated-workload-recovery-m9tvrppp1 og:image: https://api.daily.dev/og/posts/m9tVrppp1.png og:image:alt: Keep Distributed Training Running with Automated Workload Recovery og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Keep Distributed Training Running with Automated Workload Recovery **[NVIDIA Developer](https://daily.dev/sources/nvidiadev)** · 11 min read · 0 upvotes · 0 comments ## Summary NVIDIA Recovery is a new open-source service that automates recovery for distributed AI training jobs across a cluster, released under DSX OS. It detects crashes, hangs, and stragglers using application logs, GPU metrics, and scheduler signals, then isolates faulty nodes and triggers checkpoint restarts. On one production cluster, it covered nearly 200 jobs a day over three months and logged about 8,550 anomaly detections, with mean detection and recovery-action latencies of 69 and 94 seconds. It currently supports Slurm, with Kubernetes support on the roadmap. IBM Research validated it in monitor mode before enabling autonomous action, and Verda became the first NVIDIA Cloud Partner to deploy it. It's available under the Apache 2.0 license on GitHub. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is NVIDIA Recovery and how does it work with Slurm? NVIDIA Recovery is an open-source, workload-level service, part of NVIDIA DSX OS, that automates recovery across a cluster's training jobs. It detects interruptions via application logs, GPU metrics, and scheduler signals, attributes failures, isolates implicated nodes, and coordinates checkpoint restarts. It currently supports Slurm, including Slurm on Kubernetes via slurm-operator, with native Kubernetes support planned next, and is licensed under Apache 2.0. _Teams building resilient GPU training pipelines can track tools like this as they mature on daily.dev._ ### How much GPU runtime do hardware failures actually cost in large-scale distributed training? In an 11-month study of 24,000 A100 GPUs at Meta, hardware failures interrupted fewer than 1% of training jobs, but those interrupted jobs accounted for 19% of total GPU runtime. This happens because the largest, longest-running jobs are the most exposed to failure: at 99.9% daily per-GPU reliability, a 10,000-GPU job has only about a 0.005% chance of running a full day uninterrupted. _Anyone estimating GPU cluster reliability budgets can follow this kind of failure math on daily.dev._ ### How fast can automated anomaly detection and recovery actions run on a production GPU training cluster? On one NVIDIA production training cluster monitored from July 9 to October 9, NVIDIA Recovery covered nearly 200 jobs per day and recorded about 8,550 anomaly detections, with mean detection latency of 69 seconds and mean recovery-action latency of 94 seconds. Recovery-action latency ends at the automated action itself; scheduling and checkpoint restoration determine when productive training actually resumes. _Infra engineers benchmarking recovery tooling for GPU clusters can compare numbers like these on daily.dev._ ## Similar posts on daily.dev - [How NVIDIA Mission Control 2.5 Brings Autonomous Recovery to Kubernetes](https://daily.dev/posts/how-nvidia-mission-control-2-5-brings-autonomous-recovery-to-kubernetes-xyjutgwim) · NVIDIA Developer · 0 upvotes · 0 comments - [Reliable GPU compute for large training & inference jobs](https://daily.dev/posts/reliable-gpu-compute-for-large-training-inference-jobs-pcelksbh8) · Netflix TechBlog · 0 upvotes · 0 comments - [Fault-Tolerant Training on Spot GPU Droplets: Checkpoint, Resume, and Keep Your Run Alive](https://daily.dev/posts/fault-tolerant-training-on-spot-gpu-droplets-checkpoint-resume-and-keep-your-run-alive-kehrdtpmr) · DigitalOcean Community · 0 upvotes · 0 comments - [Auto repair for Together GPU Clusters: from GPU fault to healthy node](https://daily.dev/posts/auto-repair-for-together-gpu-clusters-from-gpu-fault-to-healthy-node-mgrm2ccnw) · Together AI · 0 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#nvidia](https://daily.dev/tags/nvidia), [#gpu](https://daily.dev/tags/gpu), [#mlops](https://daily.dev/tags/mlops) [View this post on daily.dev](https://daily.dev/posts/keep-distributed-training-running-with-automated-workload-recovery-m9tvrppp1) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Keep Distributed Training Running with Automated Workload Recovery","url":"https://daily.dev/posts/keep-distributed-training-running-with-automated-workload-recovery-m9tvrppp1","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/keep-distributed-training-running-with-automated-workload-recovery-m9tvrppp1"},"datePublished":"2026-11-02T22:46:17.102Z","dateModified":"2026-11-02T22:46:49.073Z","description":"NVIDIA Recovery is a new open-source service that automates recovery for distributed AI training jobs across a cluster, released under DSX OS. It detects...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/d0db23ce91e2b2dbd875eebad63fffa4?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/d0db23ce91e2b2dbd875eebad63fffa4?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"NVIDIA Developer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"NVIDIA Developer","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/86e45aab42ba48ce83103d01b1119910","url":"https://daily.dev/sources/nvidiadev"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/keep-distributed-training-running-with-automated-workload-recovery-m9tvrppp1","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning,nvidia,gpu,mlops","timeRequired":"PT11M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"NVIDIA Developer","item":"https://daily.dev/sources/nvidiadev"},{"@type":"ListItem","position":3,"name":"Keep Distributed Training Running with Automated Workload Recovery"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/keep-distributed-training-running-with-automated-workload-recovery-m9tvrppp1#faq","mainEntity":[{"@type":"Question","name":"What is NVIDIA Recovery and how does it work with Slurm?","acceptedAnswer":{"@type":"Answer","text":"NVIDIA Recovery is an open-source, workload-level service, part of NVIDIA DSX OS, that automates recovery across a cluster's training jobs. It detects interruptions via application logs, GPU metrics, and scheduler signals, attributes failures, isolates implicated nodes, and coordinates checkpoint restarts. It currently supports Slurm, including Slurm on Kubernetes via slurm-operator, with native Kubernetes support planned next, and is licensed under Apache 2.0. Teams building resilient GPU training pipelines can track tools like this as they mature on daily.dev."}},{"@type":"Question","name":"How much GPU runtime do hardware failures actually cost in large-scale distributed training?","acceptedAnswer":{"@type":"Answer","text":"In an 11-month study of 24,000 A100 GPUs at Meta, hardware failures interrupted fewer than 1% of training jobs, but those interrupted jobs accounted for 19% of total GPU runtime. This happens because the largest, longest-running jobs are the most exposed to failure: at 99.9% daily per-GPU reliability, a 10,000-GPU job has only about a 0.005% chance of running a full day uninterrupted. Anyone estimating GPU cluster reliability budgets can follow this kind of failure math on daily.dev."}},{"@type":"Question","name":"How fast can automated anomaly detection and recovery actions run on a production GPU training cluster?","acceptedAnswer":{"@type":"Answer","text":"On one NVIDIA production training cluster monitored from July 9 to October 9, NVIDIA Recovery covered nearly 200 jobs per day and recorded about 8,550 anomaly detections, with mean detection latency of 69 seconds and mean recovery-action latency of 94 seconds. Recovery-action latency ends at the automated action itself; scheduling and checkpoint restoration determine when productive training actually resumes. Infra engineers benchmarking recovery tooling for GPU clusters can compare numbers like these on daily.dev."}}]} ```