--- title: Enabling Up to 41% Faster Pre-training: MXFP8 and DeepEP... description: A joint PyTorch and Nebius experiment demonstrates up to 41% faster pre-training of DeepSeek-V3 MoE models on a 256-GPU NVIDIA B200 cluster using TorchTitan.... canonical: https://daily.dev/posts/enabling-up-to-41-faster-pre-training-mxfp8-and-deepep-for-deepseek-v3-on-b200-with-torchtitan-p-xgwnngegk twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Enabling Up to 41% Faster Pre-training: MXFP8 and DeepEP for DeepSeek-V3 on B200 with TorchTitan – PyTorch | daily.dev og:description: A joint PyTorch and Nebius experiment demonstrates up to 41% faster pre-training of DeepSeek-V3 MoE models on a 256-GPU NVIDIA B200 cluster using TorchTitan.... og:url: https://daily.dev/posts/enabling-up-to-41-faster-pre-training-mxfp8-and-deepep-for-deepseek-v3-on-b200-with-torchtitan-p-xgwnngegk og:image: https://api.daily.dev/og/posts/XgWNNgeGk.png og:image:alt: Enabling Up to 41% Faster Pre-training: MXFP8 and DeepEP for DeepSeek-V3 on B200 with TorchTitan – PyTorch og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Enabling Up to 41% Faster Pre-training: MXFP8 and DeepEP for DeepSeek-V3 on B200 with TorchTitan – PyTorch **[PyTorch](https://daily.dev/sources/pytorch)** · 10 min read · 0 upvotes · 0 comments ## Summary A joint PyTorch and Nebius experiment demonstrates up to 41% faster pre-training of DeepSeek-V3 MoE models on a 256-GPU NVIDIA B200 cluster using TorchTitan. Two orthogonal optimizations were evaluated: MXFP8 mixed-precision training (via TorchAO) and DeepEP communication acceleration. For the 671B model, DeepEP alone yielded +32% throughput over BF16 baseline by replacing standard all-to-all with GPU-initiated RDMA+NVLink kernels. Combining MXFP8 on grouped GEMMs with DeepEP pushed total throughput to 918 tokens/sec, a +41% gain. Convergence experiments on the 16B model confirmed MXFP8 training matches BF16 quality with no degradation over 1,500 steps. Future work includes enabling MXFP8 on linear layers once torch.compile supports MXFP8 with tensor parallelism. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [MXFP8 Training for MoEs: 1.3x training speedup vs BF16 for Llama4 Scout on GB200 cluster using TorchAO and TorchTitan – PyTorch](https://daily.dev/posts/mxfp8-training-for-moes-1-3x-training-speedup-vs-bf16-for-llama4-scout-on-gb200-cluster-using-torch-agasmbqcx) · PyTorch · 0 upvotes · 0 comments - [Accelerating Dropless MoE Training in JAX with NVIDIA Transformer Engine](https://daily.dev/posts/accelerating-dropless-moe-training-in-jax-with-nvidia-transformer-engine-iqd8saow0) · NVIDIA Developer · 0 upvotes · 0 comments - [DeepSeek-V3.2 on GB300: Performance Breakthrough](https://daily.dev/posts/deepseek-v3-2-on-gb300-performance-breakthrough-3ccxkuvow) · vLLM · 1 upvotes · 0 comments - [FP8 Training on AMD GPUs with TorchTitan and TorchAO: Upstreaming Performance Improvements – PyTorch](https://daily.dev/posts/fp8-training-on-amd-gpus-with-torchtitan-and-torchao-upstreaming-performance-improvements-pytorch-drcnnsflp) · PyTorch · 0 upvotes · 0 comments - [Using NVFP4 Low-Precision Model Training for Higher Throughput Without Losing Accuracy](https://daily.dev/posts/using-nvfp4-low-precision-model-training-for-higher-throughput-without-losing-accuracy-mburtmoay) · NVIDIA Developer · 1 upvotes · 0 comments --- Tags: [#pytorch](https://daily.dev/tags/pytorch), [#deepseek](https://daily.dev/tags/deepseek), [#mixture-of-experts](https://daily.dev/tags/mixture-of-experts) [View this post on daily.dev](https://daily.dev/posts/enabling-up-to-41-faster-pre-training-mxfp8-and-deepep-for-deepseek-v3-on-b200-with-torchtitan-p-xgwnngegk) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Enabling Up to 41% Faster Pre-training: MXFP8 and DeepEP for DeepSeek-V3 on B200 with TorchTitan – PyTorch","url":"https://daily.dev/posts/enabling-up-to-41-faster-pre-training-mxfp8-and-deepep-for-deepseek-v3-on-b200-with-torchtitan-p-xgwnngegk","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/enabling-up-to-41-faster-pre-training-mxfp8-and-deepep-for-deepseek-v3-on-b200-with-torchtitan-p-xgwnngegk"},"datePublished":"2026-04-07T18:00:58.969Z","dateModified":"2026-04-07T18:04:16.244Z","description":"A joint PyTorch and Nebius experiment demonstrates up to 41% faster pre-training of DeepSeek-V3 MoE models on a 256-GPU NVIDIA B200 cluster using TorchTitan....","image":"https://media.daily.dev/image/upload/s--2-1xRawN--/f_auto/v1722860399/public/Placeholder%2011","thumbnailUrl":"https://media.daily.dev/image/upload/s--2-1xRawN--/f_auto/v1722860399/public/Placeholder%2011","isAccessibleForFree":true,"articleSection":"PyTorch","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"PyTorch","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/5fdab3d75a834a97a52d433d9c7e5ff9","url":"https://daily.dev/sources/pytorch"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/enabling-up-to-41-faster-pre-training-mxfp8-and-deepep-for-deepseek-v3-on-b200-with-torchtitan-p-xgwnngegk","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"pytorch,deepseek,mixture-of-experts","timeRequired":"PT10M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"PyTorch","item":"https://daily.dev/sources/pytorch"},{"@type":"ListItem","position":3,"name":"Enabling Up to 41% Faster Pre-training: MXFP8 and DeepEP for DeepSeek-V3 on B200 with TorchTitan – PyTorch"}]} ```