--- title: Flight Recorder: A New Lens for Understanding NCCL... description: NCCL watchdog timeouts are a common and notoriously hard-to-debug failure mode in distributed PyTorch training. The error is a catch-all triggered when a GPU... canonical: https://daily.dev/posts/flight-recorder-a-new-lens-for-understanding-nccl-watchdog-timeouts-pytorch-wpkypnmil twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Flight Recorder: A New Lens for Understanding NCCL Watchdog Timeouts – PyTorch | daily.dev og:description: NCCL watchdog timeouts are a common and notoriously hard-to-debug failure mode in distributed PyTorch training. The error is a catch-all triggered when a GPU... og:url: https://daily.dev/posts/flight-recorder-a-new-lens-for-understanding-nccl-watchdog-timeouts-pytorch-wpkypnmil og:image: https://api.daily.dev/og/posts/wpKypnMIL.png og:image:alt: Flight Recorder: A New Lens for Understanding NCCL Watchdog Timeouts – PyTorch og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Flight Recorder: A New Lens for Understanding NCCL Watchdog Timeouts – PyTorch **[PyTorch](https://daily.dev/sources/pytorch)** · 25 min read · 0 upvotes · 0 comments ## Summary NCCL watchdog timeouts are a common and notoriously hard-to-debug failure mode in distributed PyTorch training. The error is a catch-all triggered when a GPU collective operation exceeds a timeout, but the root cause is almost always collective desync rather than slowness. Based on Meta's fleet experience, over 60% of timeouts stem from CPU-side issues like execution divergence, PT2 compilation asymmetry, or improper error handling. Other causes include GPU compute kernel hangs, misconfigured collective arguments (especially all_to_all splits), and network/hardware failures. PyTorch's Flight Recorder (FR) addresses this by maintaining a per-rank ring buffer of collective metadata, call stacks, and state. On timeout, FR dumps records from all ranks via a TCP side-channel, enabling post-hoc cross-rank analysis. The fr_trace tool aligns and aggregates these records to identify mismatches, and Meta has built a visualization layer on top for rapid diagnosis. Two case studies illustrate how FR pinpointed CPU execution divergence in metric aggregation and misconfigured all_to_all splits in a RecSys workload. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [How we keep GPUs reliable across Databricks AI](https://daily.dev/posts/how-we-keep-gpus-reliable-across-databricks-ai-um3yzqfzg) · databricks · 0 upvotes · 0 comments - [Real-Time Performance Monitoring and Faster Debugging with NCCL Inspector and Prometheus](https://daily.dev/posts/real-time-performance-monitoring-and-faster-debugging-with-nccl-inspector-and-prometheus-ijip07tjm) · NVIDIA Developer · 0 upvotes · 0 comments - [Keeping GPUs Ticking Like Clockwork](https://daily.dev/posts/keeping-gpus-ticking-like-clockwork-xhvxds2o1) · The New Stack · 2 upvotes · 0 comments - [Introducing dial9: a flight recorder for Tokio](https://daily.dev/posts/introducing-dial9-a-flight-recorder-for-tokio-txlhqqz42) · Tokio · 33 upvotes · 2 comments --- Tags: [#gpu](https://daily.dev/tags/gpu), [#machine-learning](https://daily.dev/tags/machine-learning), [#pytorch](https://daily.dev/tags/pytorch) [View this post on daily.dev](https://daily.dev/posts/flight-recorder-a-new-lens-for-understanding-nccl-watchdog-timeouts-pytorch-wpkypnmil) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Flight Recorder: A New Lens for Understanding NCCL Watchdog Timeouts – PyTorch","url":"https://daily.dev/posts/flight-recorder-a-new-lens-for-understanding-nccl-watchdog-timeouts-pytorch-wpkypnmil","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/flight-recorder-a-new-lens-for-understanding-nccl-watchdog-timeouts-pytorch-wpkypnmil"},"datePublished":"2026-04-07T18:00:38.058Z","dateModified":"2026-10-14T02:34:56.598Z","description":"NCCL watchdog timeouts are a common and notoriously hard-to-debug failure mode in distributed PyTorch training. The error is a catch-all triggered when a GPU...","image":"https://media.daily.dev/image/upload/s--HRgLpUt6--/f_auto/v1722860399/public/Placeholder%2003","thumbnailUrl":"https://media.daily.dev/image/upload/s--HRgLpUt6--/f_auto/v1722860399/public/Placeholder%2003","isAccessibleForFree":true,"articleSection":"PyTorch","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"PyTorch","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/5fdab3d75a834a97a52d433d9c7e5ff9","url":"https://daily.dev/sources/pytorch"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/flight-recorder-a-new-lens-for-understanding-nccl-watchdog-timeouts-pytorch-wpkypnmil","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"gpu,machine-learning,pytorch","timeRequired":"PT25M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"PyTorch","item":"https://daily.dev/sources/pytorch"},{"@type":"ListItem","position":3,"name":"Flight Recorder: A New Lens for Understanding NCCL Watchdog Timeouts – PyTorch"}]} ```