--- title: Setting a World Record for MoE Pre-Training on NVIDIA... description: NVIDIA's GB300 NVL72 rack-scale system has set a world record for MoE pre-training, achieving 1,648 TFLOPs per GPU on DeepSeek-V3 671B using 256 GPUs — roughly... canonical: https://daily.dev/posts/setting-a-world-record-for-moe-pre-training-on-nvidia-gb300-nvl72-so5qhcp58 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Setting a World Record for MoE Pre-Training on NVIDIA GB300 NVL72 | daily.dev og:description: NVIDIA's GB300 NVL72 rack-scale system has set a world record for MoE pre-training, achieving 1,648 TFLOPs per GPU on DeepSeek-V3 671B using 256 GPUs — roughly... og:url: https://daily.dev/posts/setting-a-world-record-for-moe-pre-training-on-nvidia-gb300-nvl72-so5qhcp58 og:image: https://api.daily.dev/og/posts/So5qhCP58.png og:image:alt: Setting a World Record for MoE Pre-Training on NVIDIA GB300 NVL72 og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Setting a World Record for MoE Pre-Training on NVIDIA GB300 NVL72 **[NVIDIA Developer](https://daily.dev/sources/nvidiadev)** · 7 min read · 0 upvotes · 0 comments ## Summary NVIDIA's GB300 NVL72 rack-scale system has set a world record for MoE pre-training, achieving 1,648 TFLOPs per GPU on DeepSeek-V3 671B using 256 GPUs — roughly 3x the throughput of the previous GB200 NVL72 generation. The post explains why MoE architectures shift the bottleneck from compute to all-to-all communication, and how GB300 NVL72 addresses this with fifth-generation NVLink (1.8 TB/s per GPU, 130 TB/s rack-wide non-blocking bandwidth), 800 Gbps scale-out networking via ConnectX-8 SuperNICs, and co-designed software. Megatron Core, TorchTitan, and JAX all benefit from NVIDIA optimizations, with JAX showing nearly 10x performance improvement over six months purely from software gains. Scaling from 256 to 1,024 GPUs, Megatron Core retains 98.5% per-GPU efficiency, demonstrating near-linear scaling. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [Accelerating Dropless MoE Training in JAX with NVIDIA Transformer Engine](https://daily.dev/posts/accelerating-dropless-moe-training-in-jax-with-nvidia-transformer-engine-iqd8saow0) · NVIDIA Developer · 0 upvotes · 0 comments - [Mixture of Experts Powers the Most Intelligent Frontier AI Models, Runs 10x Faster on NVIDIA Blackwell NVL72](https://daily.dev/posts/mixture-of-experts-powers-the-most-intelligent-frontier-ai-models-runs-10x-faster-on-nvidia-blackwe-mlfwpdcyw) · NVIDIA · 0 upvotes · 0 comments - [MoE models changed what "fits on my GPU" means, and most hardware advice hasn't caught up](https://daily.dev/posts/moe-models-changed-what-fits-on-my-gpu-means-and-most-hardware-advice-hasn-t-caught-up-qcmkcp9b6) · XDA Developers · 1 upvotes · 1 comments - [Scaling Large MoE Models with Wide Expert Parallelism on NVL72 Rack Scale Systems](https://daily.dev/posts/scaling-large-moe-models-with-wide-expert-parallelism-on-nvl72-rack-scale-systems-h6ejybkws) · NVIDIA Developer · 1 upvotes · 0 comments - [Efficient MoE Pre-training at Scale on 1K AMD GPUs with TorchTitan – PyTorch](https://daily.dev/posts/efficient-moe-pre-training-at-scale-on-1k-amd-gpus-with-torchtitan-pytorch-xscr4vica) · PyTorch · 0 upvotes · 0 comments --- Tags: [#deepseek](https://daily.dev/tags/deepseek), [#machine-learning](https://daily.dev/tags/machine-learning), [#mixture-of-experts](https://daily.dev/tags/mixture-of-experts) [View this post on daily.dev](https://daily.dev/posts/setting-a-world-record-for-moe-pre-training-on-nvidia-gb300-nvl72-so5qhcp58) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Setting a World Record for MoE Pre-Training on NVIDIA GB300 NVL72","url":"https://daily.dev/posts/setting-a-world-record-for-moe-pre-training-on-nvidia-gb300-nvl72-so5qhcp58","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/setting-a-world-record-for-moe-pre-training-on-nvidia-gb300-nvl72-so5qhcp58"},"datePublished":"2026-07-23T18:57:08.356Z","dateModified":"2026-10-14T02:26:34.093Z","description":"NVIDIA's GB300 NVL72 rack-scale system has set a world record for MoE pre-training, achieving 1,648 TFLOPs per GPU on DeepSeek-V3 671B using 256 GPUs — roughly...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/78ab27cab80a42552bcefd15ddb7d332?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/78ab27cab80a42552bcefd15ddb7d332?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"NVIDIA Developer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"NVIDIA Developer","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/86e45aab42ba48ce83103d01b1119910","url":"https://daily.dev/sources/nvidiadev"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/setting-a-world-record-for-moe-pre-training-on-nvidia-gb300-nvl72-so5qhcp58","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"deepseek,machine-learning,mixture-of-experts","timeRequired":"PT7M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"NVIDIA Developer","item":"https://daily.dev/sources/nvidiadev"},{"@type":"ListItem","position":3,"name":"Setting a World Record for MoE Pre-Training on NVIDIA GB300 NVL72"}]} ```