---
title: "Enhancing Goodput in Large-Scale LLM Training with Nonuniform Tensor Parallelism"
url: https://daily.dev/posts/enhancing-goodput-in-large-scale-llm-training-with-nonuniform-tensor-parallelism-f68cy3jxh
source_url: https://developer.nvidia.com/blog/enhancing-goodput-in-large-scale-llm-training-with-nonuniform-tensor-parallelism
type: article
source: "NVIDIA Developer"
published: 2026-07-06T21:47:16.036Z
updated: 2026-07-06T21:47:36.710Z
tags: ["machine-learning", "nvidia"]
reading_time: 6
upvotes: 0
comments: 0
language: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Enhancing Goodput in Large-Scale LLM Training with Nonuniform Tensor Parallelism

**[NVIDIA Developer](https://daily.dev/sources/nvidiadev)** · 6 min read · 0 upvotes · 0 comments

## Summary

Nonuniform Tensor Parallelism (NTP) is an experimental framework for maintaining high Goodput during large-scale LLM training when GPUs fail or become temporarily unavailable. When a GPU drops out of a tensor parallelism group, NTP dynamically reduces the TP degree to use only remaining functional GPUs, applies power-boosting to compensate for the reduced replica's slower throughput, and performs efficient resharding overlapped with backward computation — adding less than 1% overhead to healthy replicas. The approach prevents transient hardware issues from stalling entire training jobs across thousands of GPUs. NTP has been added to the developer branch of NVIDIA Megatron Core, with related work extending the concept to Nonuniform Expert Parallelism for MoE models.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://developer.nvidia.com/blog/enhancing-goodput-in-large-scale-llm-training-with-nonuniform-tensor-parallelism>

## Similar posts on daily.dev

- [A Guide to Multi-Node Tensor and Pipeline Parallelism](https://daily.dev/posts/a-guide-to-multi-node-tensor-and-pipeline-parallelism-66v1ghuqr) · DigitalOcean Community · 0 upvotes · 0 comments
- [Train Your Large Model on Multiple GPUs with Tensor Parallelism](https://daily.dev/posts/train-your-large-model-on-multiple-gpus-with-tensor-parallelism-htkcsji9o) · Machine Learning Mastery · 3 upvotes · 0 comments
- [A Guide to Multi-Node Tensor and Pipeline Parallelism](https://daily.dev/posts/a-guide-to-multi-node-tensor-and-pipeline-parallelism-w8cxdynel) · DigitalOcean Community · 0 upvotes · 0 comments
- [Scaling Large MoE Models with Wide Expert Parallelism on NVL72 Rack Scale Systems](https://daily.dev/posts/scaling-large-moe-models-with-wide-expert-parallelism-on-nvl72-rack-scale-systems-h6ejybkws) · NVIDIA Developer · 1 upvotes · 0 comments
- [Scaling AI Inference Across Multiple GPUs Using NVIDIA TensorRT with Multi-Device Inference Support](https://daily.dev/posts/scaling-ai-inference-across-multiple-gpus-using-nvidia-tensorrt-with-multi-device-inference-support-j43290twv) · NVIDIA Developer · 0 upvotes · 0 comments

---

Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#nvidia](https://daily.dev/tags/nvidia)

[View this post on daily.dev](https://daily.dev/posts/enhancing-goodput-in-large-scale-llm-training-with-nonuniform-tensor-parallelism-f68cy3jxh)