Gradient synchronization is the primary bottleneck in large-scale distributed GPU training. This deep dive covers how MPI collective operations like All-Reduce, Allgather, and Send/Recv power the four main parallelism strategies: data parallelism, tensor parallelism, pipeline parallelism, and sharded data parallelism. It explains how GPU-aware MPI with GPUDirect RDMA eliminates costly PCIe round-trips by letting NICs read and write GPU VRAM directly. The post then walks through building PyTorch from source with MPI support, configuring the torch.distributed MPI backend, and running a working MNIST distributed training example across two GPUs. Cross-vendor GPU communication limitations and the experimental multi-backend support in PyTorch are also addressed.
