--- title: "FP8 Training on AMD GPUs with TorchTitan and TorchAO: Upstreaming Performance Improvements – PyTorch" url: https://daily.dev/posts/fp8-training-on-amd-gpus-with-torchtitan-and-torchao-upstreaming-performance-improvements-pytorch-drcnnsflp source_url: https://pytorch.org/blog/fp8-training-on-amd-gpus-with-torchtitan-and-torchao-upstreaming-performance-improvements type: article source: "PyTorch" published: 2026-08-13T16:04:22.998Z updated: 2026-08-13T16:04:52.689Z tags: ["machine-learning", "gpu", "pytorch", "amd"] reading_time: 9 upvotes: 0 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # FP8 Training on AMD GPUs with TorchTitan and TorchAO: Upstreaming Performance Improvements – PyTorch **[PyTorch](https://daily.dev/sources/pytorch)** · 9 min read · 0 upvotes · 0 comments ## Summary AMD and Meta/PyTorch engineers upstreamed FP8 training optimizations for AMD Instinct GPUs directly into TorchAO and TorchTitan, giving competitive FP8 performance out of the box without any AMD-specific installs. On dense models, rowwise FP8 delivers a 13.4% throughput gain over BF16 on Llama3-8B. For MoE architectures like DeepSeek-V3 671B, FP8 quantization initially added heavy overhead due to memory-bound kernel launches and HBM round-trips; fused Triton kernels recovered 89% of that overhead, with the colwise scales fix alone delivering a 6.2x speedup per MoE layer. Key work included adding hardware auto-detection for AMD's e4m3fnuz FP8 format (fixing silent numerical corruption from mismatched max values), enabling FP8 grouped GEMM on ROCm via Composable Kernel, and a three-level Triton fusion pipeline reducing kernel launches, improving memory coalescing, and relaxing unnecessary atomic memory fences on AMD hardware. An autotuning search-space expansion was tried and reverted after showing no benefit. Work continues on MXFP8 grouped GEMM for MI355X GPUs. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Does TorchTitan support FP8 training on AMD Instinct GPUs now? Yes, AMD's FP8 optimizations from Primus-Turbo have been upstreamed directly into pytorch/ao and pytorch/torchtitan, giving competitive FP8 performance out of the box with nothing AMD-specific to install. Teams get the gains by upgrading TorchAO and TorchTitan. Rowwise FP8 delivers a 13.4% throughput gain over BF16 on Llama3-8B on 8xMI300X GPUs. _daily.dev surfaces upstream framework changes like this for teams tuning GPU training throughput._ ### Why does FP8 training on AMD GPUs give silently wrong results if the wrong number format is used? AMD Instinct GPUs use the e4m3fnuz FP8 format, which has a max representable value of 240 and no NaN/Inf encodings. If a library computes scales against a different max value (as TorchAO initially did, assuming NVIDIA's e4m3fn), tensors get scaled beyond the hardware's representable range, clipping activations and corrupting gradients silently rather than raising an error. TorchAO now auto-detects the platform and selects the correct FP8 dtype automatically. _developers debugging GPU numerics issues follow fixes like this one on daily.dev._ ### How much do fused Triton kernels reduce FP8 quantization overhead for MoE models like DeepSeek-V3? Fused Triton kernels recovered 89% of the FP8 quantization overhead on DeepSeek-V3 671B MoE shapes. Replacing a five-kernel eager chain with a single fused kernel gave a 17% end-to-end throughput improvement (5,996 to 7,027 tok/s) on 8xMI325X GPUs, and fixing non-coalesced memory writes in the colwise scales kernel produced a 6.2x speedup per MoE layer on MI300X. _engineers optimizing MoE training performance track kernel-level wins like these on daily.dev._ ## Similar posts on daily.dev - [Efficient MoE Pre-training at Scale on 1K AMD GPUs with TorchTitan – PyTorch](https://daily.dev/posts/efficient-moe-pre-training-at-scale-on-1k-amd-gpus-with-torchtitan-pytorch-xscr4vica) · PyTorch · 0 upvotes · 0 comments - [MXFP8 Training for MoEs: 1.3x training speedup vs BF16 for Llama4 Scout on GB200 cluster using TorchAO and TorchTitan – PyTorch](https://daily.dev/posts/mxfp8-training-for-moes-1-3x-training-speedup-vs-bf16-for-llama4-scout-on-gb200-cluster-using-torch-agasmbqcx) · PyTorch · 0 upvotes · 0 comments - [Enabling Up to 41% Faster Pre-training: MXFP8 and DeepEP for DeepSeek-V3 on B200 with TorchTitan – PyTorch](https://daily.dev/posts/enabling-up-to-41-faster-pre-training-mxfp8-and-deepep-for-deepseek-v3-on-b200-with-torchtitan-p-xgwnngegk) · PyTorch · 0 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#gpu](https://daily.dev/tags/gpu), [#pytorch](https://daily.dev/tags/pytorch), [#amd](https://daily.dev/tags/amd) [View this post on daily.dev](https://daily.dev/posts/fp8-training-on-amd-gpus-with-torchtitan-and-torchao-upstreaming-performance-improvements-pytorch-drcnnsflp)