Nonuniform Tensor Parallelism (NTP) is an experimental framework for maintaining high Goodput during large-scale LLM training when GPUs fail or become temporarily unavailable. When a GPU drops out of a tensor parallelism group, NTP dynamically reduces the TP degree to use only remaining functional GPUs, applies power-boosting to compensate for the reduced replica's slower throughput, and performs efficient resharding overlapped with backward computation — adding less than 1% overhead to healthy replicas. The approach prevents transient hardware issues from stalling entire training jobs across thousands of GPUs. NTP has been added to the developer branch of NVIDIA Megatron Core, with related work extending the concept to Nonuniform Expert Parallelism for MoE models.
