--- title: "How we keep GPUs reliable across Databricks AI" url: https://daily.dev/posts/how-we-keep-gpus-reliable-across-databricks-ai-um3yzqfzg source_url: https://www.databricks.com/blog/how-we-keep-gpus-reliable-across-databricks-ai type: article source: "databricks" published: 2026-07-01T23:02:03.543Z updated: 2026-07-01T23:02:31.836Z tags: ["machine-learning", "databricks"] reading_time: 9 upvotes: 0 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How we keep GPUs reliable across Databricks AI **[databricks](https://daily.dev/sources/databricks)** · 9 min read · 0 upvotes · 0 comments ## Summary Databricks AI shares how they maintain GPU reliability at scale across distributed training workloads. The post covers three main failure categories: crashed jobs (often surfacing as NCCL watchdog timeouts), silent slowdowns from thermal or interconnect degradation, and numerical corruption from ECC-uncorrectable faults. At large GPU counts, failures during a run are statistically expected — a 1,024-GPU job running 30 days has a 57% chance of encountering one. To address this, Databricks built a multi-stage health check service called gpu-monitor with three layers: active bootstrap checks run at node provisioning, passive continuous checks monitor nodes under live workloads, and periodic multi-node checks validate inter-node fabric health using NCCL collective bandwidth probes across payload sizes from 8 bytes to 2 GiB. A real incident is detailed where a single InfiniBand port flap crashed a 7-hour training run due to NCCL_IB_TIMEOUT firing before the PyTorch watchdog — highlighting that cumulative downtime matters more than flap count. The post is the first in a series on GPU reliability engineering at Databricks scale. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [Keeping 20,000 GPUs healthy](https://daily.dev/posts/keeping-20-000-gpus-healthy-if9axbund) · Modal · 40 upvotes · 3 comments - [Keeping GPUs Ticking Like Clockwork](https://daily.dev/posts/keeping-gpus-ticking-like-clockwork-xhvxds2o1) · The New Stack · 2 upvotes · 0 comments - [Making GPU Clusters More Efficient with NVIDIA Data Center Monitoring](https://daily.dev/posts/making-gpu-clusters-more-efficient-with-nvidia-data-center-monitoring-hxuekxyge) · NVIDIA Developer · 1 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#databricks](https://daily.dev/tags/databricks) [View this post on daily.dev](https://daily.dev/posts/how-we-keep-gpus-reliable-across-databricks-ai-um3yzqfzg)