<!-- mobian-agent-page publisher="dailydev" canonical="https://daily.dev/posts/pytorch-2-15-tackles-the-one-bad-gpu-problem-ocr51ovqs" -->

---
title: PyTorch 2.15 tackles the 'one bad GPU' problem | daily.dev
description: PyTorch 2.15 ships a rewritten NCCL backend that stops a single failing GPU from forcing a full training restart. The new ProcessGroupNCCL allows only one...
canonical: https://daily.dev/posts/pytorch-2-15-tackles-the-one-bad-gpu-problem-ocr51ovqs
twitter:card: summary_large_image
twitter:site: @dailydotdev
og:type: website
og:site_name: daily.dev
og:title: PyTorch 2.15 tackles the 'one bad GPU' problem | daily.dev
og:description: PyTorch 2.15 ships a rewritten NCCL backend that stops a single failing GPU from forcing a full training restart. The new ProcessGroupNCCL allows only one...
og:url: https://daily.dev/posts/pytorch-2-15-tackles-the-one-bad-gpu-problem-ocr51ovqs
og:image: https://api.daily.dev/og/posts/oCr51ovQs.png
og:image:alt: PyTorch 2.15 tackles the 'one bad GPU' problem
og:image:width: 1200
og:image:height: 630
og:locale: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# PyTorch 2.15 tackles the 'one bad GPU' problem

**[DEVCLASS](https://daily.dev/sources/devclass)** · 4 min read · 1 upvotes · 0 comments

## Summary

PyTorch 2.15 ships a rewritten NCCL backend that stops a single failing GPU from forcing a full training restart. The new ProcessGroupNCCL allows only one communicator per process group during initialization, which by default eliminates the crosstalk that the old version's many communicators caused. A third-party orchestrator can now exclude a crashed or slow GPU and assign a replacement worker. A new 'flight recorder' captures operation snapshots for debugging stalls. The legacy backend remains as a rollback option but will be removed in 2.16, planned for early next year. The release also totals 3,117 commits from 583 contributors and includes parallel processing improvements for Apple gear and support for Nvidia's next-generation Blackwell GPUs.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://www.devclass.com/devops/2026/11/07/pytorch-215-tackles-the-one-bad-gpu-problem/5308234>

## Questions this post answers

### What changed in PyTorch 2.15's NCCL backend for handling GPU failures during training?

PyTorch 2.15 includes a rewritten ProcessGroupNCCL backend that allows a third-party orchestrator to exclude a crashed or slow GPU and assign a replacement worker instead of restarting the whole job. The backend now enforces one communicator per process group during initialization, removing the crosstalk and deadlocks caused by the old design spawning dozens of communicators per group.

_daily.dev helps engineers tuning distributed training pipelines keep up with changes like PyTorch's NCCL rewrite._

### What is the flight recorder feature added in PyTorch 2.15?

The flight recorder is a new diagnostic feature in PyTorch 2.15's NCCL backend that captures all GPU communication operations as a series of snapshots, which admins can inspect after a cluster stall or deadlock instead of receiving only a generic timeout message. It is part of the rewritten ProcessGroupNCCL interface.

_Anyone debugging mysterious GPU cluster stalls can track tooling updates like this one on daily.dev._

### Will the old PyTorch NCCL backend still work after upgrading to 2.15?

Yes, the legacy ProcessGroupNCCL backend ships in PyTorch 2.15 as a rollback option alongside the new rewritten backend. It is scheduled for removal in PyTorch 2.16, planned for early next year, once the new backend is confirmed to work as expected.

_Teams planning a PyTorch upgrade path can follow release and deprecation timelines like this on daily.dev._

## Similar posts on daily.dev

- [PyTorch 2.15 Release Blog](https://daily.dev/posts/pytorch-2-15-release-blog-zkriurwl4) · PyTorch · 0 upvotes · 0 comments
- [Fusing Communication and Compute with New Device API and Copy Engine Collectives in NVIDIA NCCL 2.28](https://daily.dev/posts/fusing-communication-and-compute-with-new-device-api-and-copy-engine-collectives-in-nvidia-nccl-2-28-oek1lxll5) · NVIDIA Developer · 0 upvotes · 0 comments
- [PyTorch distributed is changing and TorchComms is why](https://daily.dev/posts/pytorch-distributed-is-changing-and-torchcomms-is-why-bbyvls17k) · Red Hat Developer · 0 upvotes · 0 comments
- [How we keep GPUs reliable across Databricks AI](https://daily.dev/posts/how-we-keep-gpus-reliable-across-databricks-ai-um3yzqfzg) · databricks · 0 upvotes · 0 comments

---

Tags: [#nvidia](https://daily.dev/tags/nvidia), [#distributed-systems](https://daily.dev/tags/distributed-systems), [#pytorch](https://daily.dev/tags/pytorch)

[View this post on daily.dev](https://daily.dev/posts/pytorch-2-15-tackles-the-one-bad-gpu-problem-ocr51ovqs)

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]}
{"@context":"https://schema.org","@type":"TechArticle","headline":"PyTorch 2.15 tackles the 'one bad GPU' problem","url":"https://daily.dev/posts/pytorch-2-15-tackles-the-one-bad-gpu-problem-ocr51ovqs","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/pytorch-2-15-tackles-the-one-bad-gpu-problem-ocr51ovqs"},"datePublished":"2026-11-07T18:41:03.407Z","dateModified":"2026-11-07T18:41:30.400Z","description":"PyTorch 2.15 ships a rewritten NCCL backend that stops a single failing GPU from forcing a full training restart. The new ProcessGroupNCCL allows only one...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/328f29448e1956c3e1aab36d956ddfd5?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/328f29448e1956c3e1aab36d956ddfd5?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"DEVCLASS","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"DEVCLASS","logo":"https://media.daily.dev/image/upload/s--quq5_zKP--/f_auto/v1732552064/logos/devclass","url":"https://daily.dev/sources/devclass"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/pytorch-2-15-tackles-the-one-bad-gpu-problem-ocr51ovqs","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"nvidia,distributed-systems,pytorch","timeRequired":"PT4M"}
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"DEVCLASS","item":"https://daily.dev/sources/devclass"},{"@type":"ListItem","position":3,"name":"PyTorch 2.15 tackles the 'one bad GPU' problem"}]}
{"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/pytorch-2-15-tackles-the-one-bad-gpu-problem-ocr51ovqs#faq","mainEntity":[{"@type":"Question","name":"What changed in PyTorch 2.15's NCCL backend for handling GPU failures during training?","acceptedAnswer":{"@type":"Answer","text":"PyTorch 2.15 includes a rewritten ProcessGroupNCCL backend that allows a third-party orchestrator to exclude a crashed or slow GPU and assign a replacement worker instead of restarting the whole job. The backend now enforces one communicator per process group during initialization, removing the crosstalk and deadlocks caused by the old design spawning dozens of communicators per group. daily.dev helps engineers tuning distributed training pipelines keep up with changes like PyTorch's NCCL rewrite."}},{"@type":"Question","name":"What is the flight recorder feature added in PyTorch 2.15?","acceptedAnswer":{"@type":"Answer","text":"The flight recorder is a new diagnostic feature in PyTorch 2.15's NCCL backend that captures all GPU communication operations as a series of snapshots, which admins can inspect after a cluster stall or deadlock instead of receiving only a generic timeout message. It is part of the rewritten ProcessGroupNCCL interface. Anyone debugging mysterious GPU cluster stalls can track tooling updates like this one on daily.dev."}},{"@type":"Question","name":"Will the old PyTorch NCCL backend still work after upgrading to 2.15?","acceptedAnswer":{"@type":"Answer","text":"Yes, the legacy ProcessGroupNCCL backend ships in PyTorch 2.15 as a rollback option alongside the new rewritten backend. It is scheduled for removal in PyTorch 2.16, planned for early next year, once the new backend is confirmed to work as expected. Teams planning a PyTorch upgrade path can follow release and deprecation timelines like this on daily.dev."}}]}
```