--- title: How NVIDIA Mission Control 2.5 Brings Autonomous... description: NVIDIA Mission Control 2.5 extends its autonomous recovery engine from Slurm-based HPC workloads to Kubernetes workloads managed by NVIDIA Run:ai. The engine... canonical: https://daily.dev/posts/how-nvidia-mission-control-2-5-brings-autonomous-recovery-to-kubernetes-xyjutgwim twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How NVIDIA Mission Control 2.5 Brings Autonomous Recovery to Kubernetes | daily.dev og:description: NVIDIA Mission Control 2.5 extends its autonomous recovery engine from Slurm-based HPC workloads to Kubernetes workloads managed by NVIDIA Run:ai. The engine... og:url: https://daily.dev/posts/how-nvidia-mission-control-2-5-brings-autonomous-recovery-to-kubernetes-xyjutgwim og:image: https://api.daily.dev/og/posts/XyjUTGWIm.png og:image:alt: How NVIDIA Mission Control 2.5 Brings Autonomous Recovery to Kubernetes og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How NVIDIA Mission Control 2.5 Brings Autonomous Recovery to Kubernetes **[NVIDIA Developer](https://daily.dev/sources/nvidiadev)** · 7 min read · 0 upvotes · 0 comments ## Summary NVIDIA Mission Control 2.5 extends its autonomous recovery engine from Slurm-based HPC workloads to Kubernetes workloads managed by NVIDIA Run:ai. The engine combines autonomous job recovery and autonomous hardware recovery to detect, diagnose, and recover from GPU and node failures during distributed training. Scenarios described include restarting jobs around a failing GPU, retrying transient software faults without quarantining hardware, and removing degrading nodes from scheduling before they cause failures. Mission Control 2.5 is available now, with additional features including a beta AI-enabled network anomaly detection service, centralized Keycloak authentication, Kubernetes hardening updates, and early access for NVIDIA Vera Rubin NVL72. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What new capability does NVIDIA Mission Control 2.5 add for Kubernetes workloads? Mission Control 2.5 extends its autonomous recovery engine, previously limited to Slurm-based HPC workloads, to Kubernetes-based workloads managed by NVIDIA Run:ai. The engine includes autonomous job recovery, which watches running jobs and determines restart or requeue actions, and autonomous hardware recovery, which isolates faulty GPUs or nodes and runs diagnostics without blocking workload recovery. _Teams rolling out Kubernetes-based AI training can track resiliency features like this on daily.dev._ ### How does NVIDIA Run:ai handle a GPU hardware fault during distributed training compared to a transient software error? For a hardware fault such as ECC errors or a hung worker, the autonomous job recovery service restarts or reschedules the job while the hardware recovery service isolates the suspect node and runs diagnostics in parallel. For a transient software fault on a healthy node, the system simply retries per scheduler policy without quarantining any hardware, avoiding unnecessary remediation. _Operators comparing failure-handling strategies across schedulers can follow updates like this on daily.dev._ ### What other features ship in NVIDIA Mission Control 2.5 besides Kubernetes autonomous recovery? Mission Control 2.5 also adds a beta AI-enabled network anomaly detection service, improved authentication user experience via central Keycloak across services, additional security and Kubernetes hardening updates, and early access support for NVIDIA Vera Rubin NVL72. The release is available now, with full release notes published separately. _Infrastructure teams planning an upgrade can keep tabs on release details like these via daily.dev._ ## Similar posts on daily.dev - [Accelerate Token Production in AI Factories Using Unified Services and Real-Time AI](https://daily.dev/posts/accelerate-token-production-in-ai-factories-using-unified-services-and-real-time-ai-bzw1g44qi) · NVIDIA Developer · 1 upvotes · 0 comments - [New in Together GPU Clusters: Reliability and control for production GPU clusters](https://daily.dev/posts/new-in-together-gpu-clusters-reliability-and-control-for-production-gpu-clusters-azg0okgzb) · Together AI · 0 upvotes · 0 comments - [Automate Kubernetes AI Cluster Health with NVSentinel](https://daily.dev/posts/automate-kubernetes-ai-cluster-health-with-nvsentinel-kfavzk0il) · NVIDIA Developer · 1 upvotes · 0 comments - [Kubernetes as AI’s operating system: 1.35 release signals](https://daily.dev/posts/kubernetes-as-ai-s-operating-system-1-35-release-signals-maydhbopw) · CNCF · 8 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes) [View this post on daily.dev](https://daily.dev/posts/how-nvidia-mission-control-2-5-brings-autonomous-recovery-to-kubernetes-xyjutgwim) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How NVIDIA Mission Control 2.5 Brings Autonomous Recovery to Kubernetes","url":"https://daily.dev/posts/how-nvidia-mission-control-2-5-brings-autonomous-recovery-to-kubernetes-xyjutgwim","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-nvidia-mission-control-2-5-brings-autonomous-recovery-to-kubernetes-xyjutgwim"},"datePublished":"2026-10-26T15:02:42.435Z","dateModified":"2026-10-26T15:03:10.105Z","description":"NVIDIA Mission Control 2.5 extends its autonomous recovery engine from Slurm-based HPC workloads to Kubernetes workloads managed by NVIDIA Run:ai. The engine...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/13eb8b8223ffbb5ed2cd24e4fbf8fa24?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/13eb8b8223ffbb5ed2cd24e4fbf8fa24?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"NVIDIA Developer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"NVIDIA Developer","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/86e45aab42ba48ce83103d01b1119910","url":"https://daily.dev/sources/nvidiadev"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-nvidia-mission-control-2-5-brings-autonomous-recovery-to-kubernetes-xyjutgwim","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes","timeRequired":"PT7M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"NVIDIA Developer","item":"https://daily.dev/sources/nvidiadev"},{"@type":"ListItem","position":3,"name":"How NVIDIA Mission Control 2.5 Brings Autonomous Recovery to Kubernetes"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-nvidia-mission-control-2-5-brings-autonomous-recovery-to-kubernetes-xyjutgwim#faq","mainEntity":[{"@type":"Question","name":"What new capability does NVIDIA Mission Control 2.5 add for Kubernetes workloads?","acceptedAnswer":{"@type":"Answer","text":"Mission Control 2.5 extends its autonomous recovery engine, previously limited to Slurm-based HPC workloads, to Kubernetes-based workloads managed by NVIDIA Run:ai. The engine includes autonomous job recovery, which watches running jobs and determines restart or requeue actions, and autonomous hardware recovery, which isolates faulty GPUs or nodes and runs diagnostics without blocking workload recovery. Teams rolling out Kubernetes-based AI training can track resiliency features like this on daily.dev."}},{"@type":"Question","name":"How does NVIDIA Run:ai handle a GPU hardware fault during distributed training compared to a transient software error?","acceptedAnswer":{"@type":"Answer","text":"For a hardware fault such as ECC errors or a hung worker, the autonomous job recovery service restarts or reschedules the job while the hardware recovery service isolates the suspect node and runs diagnostics in parallel. For a transient software fault on a healthy node, the system simply retries per scheduler policy without quarantining any hardware, avoiding unnecessary remediation. Operators comparing failure-handling strategies across schedulers can follow updates like this on daily.dev."}},{"@type":"Question","name":"What other features ship in NVIDIA Mission Control 2.5 besides Kubernetes autonomous recovery?","acceptedAnswer":{"@type":"Answer","text":"Mission Control 2.5 also adds a beta AI-enabled network anomaly detection service, improved authentication user experience via central Keycloak across services, additional security and Kubernetes hardening updates, and early access support for NVIDIA Vera Rubin NVL72. The release is available now, with full release notes published separately. Infrastructure teams planning an upgrade can keep tabs on release details like these via daily.dev."}}]} ```