Databricks AI shares how they maintain GPU reliability at scale across distributed training workloads. The post covers three main failure categories: crashed jobs (often surfacing as NCCL watchdog timeouts), silent slowdowns from thermal or interconnect degradation, and numerical corruption from ECC-uncorrectable faults. At large GPU counts, failures during a run are statistically expected — a 1,024-GPU job running 30 days has a 57% chance of encountering one. To address this, Databricks built a multi-stage health check service called gpu-monitor with three layers: active bootstrap checks run at node provisioning, passive continuous checks monitor nodes under live workloads, and periodic multi-node checks validate inter-node fabric health using NCCL collective bandwidth probes across payload sizes from 8 bytes to 2 GiB. A real incident is detailed where a single InfiniBand port flap crashed a 7-hour training run due to NCCL_IB_TIMEOUT firing before the PyTorch watchdog — highlighting that cumulative downtime matters more than flap count. The post is the first in a series on GPU reliability engineering at Databricks scale.
