--- title: Hugging Face releases a free MoE training handbook after... description: Hugging Face released a free, open handbook on training Mixture-of-Experts models, built from 800k GPU hours of its own experiments. Thom Wolf argues dense... canonical: https://daily.dev/posts/hugging-face-releases-a-free-moe-training-handbook-after-800k-gpu-hours-of-experiments-bkvgav0yz twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Hugging Face releases a free MoE training handbook after 800k GPU hours of experiments | daily.dev og:description: Hugging Face released a free, open handbook on training Mixture-of-Experts models, built from 800k GPU hours of its own experiments. Thom Wolf argues dense... og:url: https://daily.dev/posts/hugging-face-releases-a-free-moe-training-handbook-after-800k-gpu-hours-of-experiments-bkvgav0yz og:image: https://api.daily.dev/og/posts/BkVgAv0yZ.png og:image:alt: Hugging Face releases a free MoE training handbook after 800k GPU hours of experiments og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Hugging Face releases a free MoE training handbook after 800k GPU hours of experiments **[Collections](https://daily.dev/sources/collections)** · 2 min read · 8 upvotes · 0 comments ## Summary Hugging Face released a free, open handbook on training Mixture-of-Experts models, built from 800k GPU hours of its own experiments. Thom Wolf argues dense models are nearly finished, noting 18 of the 20 best open models today are MoE. The guide covers expert count and sizing, hiding all-to-all communication overhead, a mysterious 25x slowdown with no error logs, router choices like softmax versus sigmoid, aux loss versus DeepSeek's loss-free bias, and upcycling dense checkpoints. It runs about 70k words with 55 interactive figures and runnable code. ## Content Eighteen of the 20 best open models today are Mixture-of-Experts, according to Thom_Wolf. His take: dense is basically over. The catch is that the know-how to train a good MoE still sits mostly in the heads of a handful of teams. His team decided to fix that by running the experiments themselves. They spent 800k GPU hours and wrote up the results as "The Ultimate MoE Training Handbook", which is free and open. A related post from Nouamanetazi describes it as the most comprehensive writeup of its kind, built on roughly half a million GPU hours of work on optimizing MoE training. ## What it covers The handbook is organized around the questions training teams actually run into: - How many experts should you use, how big should they be, and should some be shared? - Why is half of each step spent in all-to-all communication, and how do you hide it? - Why is a run 25x slower with no error in the logs? - Should the router use softmax, sigmoid or sqrt(softplus), and why keep it in fp32? - Auxiliary loss or DeepSeek's loss-free bias? - Should you upcycle a dense checkpoint or train from scratch? - Can you serve with fewer experts per token than you trained with? ## Before you start It's long. Thom_Wolf puts it at about 70k words and says to budget a few days. In exchange you get 55 interactive figures, code you can run, and numbers from the team's own runs rather than borrowed benchmarks. The announcement was widely shared, including by @ClementDelangue, @yoheinakajima and @mervenoyann. ## Questions this post answers ### Why would training a mixture-of-experts model suddenly run 25 times slower with no errors in the logs? This is a known failure mode covered in Hugging Face's MoE training handbook, which treats it as one of the practical problems teams run into rather than a theoretical edge case. The handbook, built from 800k GPU hours of experiments, explains the cause and how to catch it, since silent slowdowns like this often only get passed along informally between teams. _Anyone debugging silent MoE slowdowns can track practical write-ups like this one on daily.dev._ ### Should I use aux loss or DeepSeek's loss-free balancing method for training a mixture-of-experts model? Hugging Face's MoE training handbook directly compares aux loss against DeepSeek's loss-free bias approach as one of its core topics, alongside router choice (softmax, sigmoid, or sqrt-softplus) and whether to keep routing computation in fp32. The guide is based on the team's own 800k GPU hours of training runs rather than theory alone. _Teams weighing MoE routing and balancing strategies can follow guides like this on daily.dev._ ### Is it possible to serve a mixture-of-experts model with fewer active experts per token than it was trained with? Hugging Face's MoE training handbook addresses this exact question as one of its practical topics, alongside upcycling a dense checkpoint into an MoE versus training from scratch. The handbook runs about 70,000 words with 55 interactive figures and runnable code, drawn from the team's own experimental runs. _daily.dev helps engineers planning MoE deployment tradeoffs keep up with hands-on guides like this._ ## Community take How the wider developer community reacted, aggregated from 5 discussions and 81 comments across x (as of 2026-11-06). **TL;DR:** The release is widely praised as a generous, well-produced open resource that finally writes down MoE training know-how, though some push back on the 'dense is over' framing and others ask practical follow-up questions about serving, scaling, and routing. **Sentiment:** 65% positive · 25% mixed · 10% skeptical **The case for** - Seen as a major gift to the field for documenting tribal MoE know-how that previously lived in only a few teams' heads. - The interactive diagrams/widgets are called out as especially well made and useful. - Concrete numbers from real runs (like load-balance TFLOP/s differences) are appreciated over theoretical claims. - Many say it lowers the barrier for smaller teams/builders to train MoE models competently. **The pushback** - Several push back on the claim that dense models are over, arguing dense still wins for small models/labs or specific use cases. - The training recipes reportedly assume 8 GPUs at ~80GB each, so they can't be run as-is on a single GPU. - Some question whether the findings generalize to smaller scale setups rather than just the large runs tested. - The handbook is said to focus mostly on the training loop, leaving serving-side concerns like expert paging less covered. **By community** - x (positive): Overwhelmingly enthusiastic and grateful reactions to the handbook's quality and openness, with a minority raising scaling, serving, and 'dense is over' pushback. **Hottest debate:** Whether dense models are really 'basically over' or still have a place for smaller models and labs. **Open questions** - Does the handbook's guidance hold up at smaller training scales, not just the 800k-GPU-hour frontier runs? - How does expert paging and serving-side expert placement work, as opposed to training-time placement? - What's the ideal total-experts-vs-active-experts-per-token ratio, given some models now ship extreme ratios like 16 of 896? - Will the open handbook meaningfully close the know-how gap within a year, or does MoE expertise remain a moat for a few teams? **Highlights** > @Thom_Wolf @Nouamanetazi @lvwerra @huggingface I'm building a NaN based Mesh LM. > — [black\_goo\_who on x](https://x.com/black_goo_who/status/2118442010550100144) > @Nouamanetazi @Thom_Wolf @lvwerra @huggingface Mostly the serve side on a Mac. I stream experts from SSD into a RAM pool so a full Q8 MoE fits on 128 GB. Cold first token and which experts stay hot is where I lose time. Does ch4 talk about that, or is it more about training-time expert placement? > — [lotar\_vk on x](https://x.com/lotar_vk/status/2118454750102401157) > @lotar_vk @Thom_Wolf @lvwerra @huggingface mostly the train loop but do check ch4: which aspects of expert paging are you wondering about? > — [Nouamanetazi on x · 1 comments](https://x.com/Nouamanetazi/status/2118442708192858496) > @Thom_Wolf @Nouamanetazi @lvwerra @huggingface For readers planning to run it: the handbook’s first dense and MoE recipes each assume eight GPUs with about 80 GB each. CPU tokenization can run on a single machine without Slurm; the training recipes cannot be run as written on one GPU. > — [burnrate\_ai on x](https://x.com/burnrate_ai/status/2118462819847028881) > @wleatherman9 @Nouamanetazi @lvwerra @huggingface short answer from the handbook: at fixed total and active params, finer experts help up to a point. in one ablation the loss kept dropping until 128 experts (each 1/4 width, top-8), then nothing more (most released models sit at 128 to 384 experts, top-4 to top-8) past that, > — [Thom\_Wolf on x](https://x.com/Thom_Wolf/status/2118443336256090248) **Source threads** - [x](https://x.com/yoheinakajima/status/2118443222183891133) · 0 points · 0 comments - [x](https://x.com/Thom_Wolf/status/2118420599089365114) · 0 points · 0 comments - [x](https://x.com/Thom_Wolf/status/2118432475730882627) · 0 points · 81 comments - [x](https://x.com/ClementDelangue/status/2118497125072093569) · 0 points · 0 comments - [x](https://x.com/mervenoyann/status/2118569558961496077) · 0 points · 0 comments ## Similar posts on daily.dev - [The MoE-ification of the Open Model Ecosystem, and What It Means for Your Inference Bill](https://daily.dev/posts/the-moe-ification-of-the-open-model-ecosystem-and-what-it-means-for-your-inference-bill-e4tw8ail5) · DigitalOcean Community · 1 upvotes · 0 comments - [Mixture of Experts \(MoEs\) in Transformers](https://daily.dev/posts/mixture-of-experts-moes-in-transformers-wclvbejkn) · Hugging Face · 1 upvotes · 0 comments - [MoE inference engineering, clearly explained](https://daily.dev/posts/moe-inference-engineering-clearly-explained-pkujcfe50) · Daily Dose of Data Science \| Avi Chawla \| Substack · 0 upvotes · 0 comments - [Efficient MoE Training for Biological Foundation Models](https://daily.dev/posts/efficient-moe-training-for-biological-foundation-models-cruwsoplo) · NVIDIA Developer · 0 upvotes · 0 comments - [Mixture-of-experts models quietly changed what hardware you need for local AI](https://daily.dev/posts/mixture-of-experts-models-quietly-changed-what-hardware-you-need-for-local-ai-j9dfxecun) · XDA Developers · 0 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#deepseek](https://daily.dev/tags/deepseek), [#mixture-of-experts](https://daily.dev/tags/mixture-of-experts) [View this post on daily.dev](https://daily.dev/posts/hugging-face-releases-a-free-moe-training-handbook-after-800k-gpu-hours-of-experiments-bkvgav0yz) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Hugging Face releases a free MoE training handbook after 800k GPU hours of experiments","url":"https://daily.dev/posts/hugging-face-releases-a-free-moe-training-handbook-after-800k-gpu-hours-of-experiments-bkvgav0yz","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/hugging-face-releases-a-free-moe-training-handbook-after-800k-gpu-hours-of-experiments-bkvgav0yz"},"datePublished":"2026-11-05T20:40:19.093Z","dateModified":"2026-11-06T05:03:02.620Z","description":"Hugging Face released a free, open handbook on training Mixture-of-Experts models, built from 800k GPU hours of its own experiments. Thom Wolf argues dense...","isAccessibleForFree":true,"articleSection":"Collections","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Collections","logo":"https://media.daily.dev/image/upload/s--fk_6ycEi--/f_auto,q_auto/v1780996001/logos/collections?_a=BAMAMiWQ0","url":"https://daily.dev/sources/collections"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/hugging-face-releases-a-free-moe-training-handbook-after-800k-gpu-hours-of-experiments-bkvgav0yz","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":8},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning,deepseek,mixture-of-experts","timeRequired":"PT2M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Collections","item":"https://daily.dev/sources/collections"},{"@type":"ListItem","position":3,"name":"Hugging Face releases a free MoE training handbook after 800k GPU hours of experiments"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/hugging-face-releases-a-free-moe-training-handbook-after-800k-gpu-hours-of-experiments-bkvgav0yz#faq","mainEntity":[{"@type":"Question","name":"Why would training a mixture-of-experts model suddenly run 25 times slower with no errors in the logs?","acceptedAnswer":{"@type":"Answer","text":"This is a known failure mode covered in Hugging Face's MoE training handbook, which treats it as one of the practical problems teams run into rather than a theoretical edge case. The handbook, built from 800k GPU hours of experiments, explains the cause and how to catch it, since silent slowdowns like this often only get passed along informally between teams. Anyone debugging silent MoE slowdowns can track practical write-ups like this one on daily.dev."}},{"@type":"Question","name":"Should I use aux loss or DeepSeek's loss-free balancing method for training a mixture-of-experts model?","acceptedAnswer":{"@type":"Answer","text":"Hugging Face's MoE training handbook directly compares aux loss against DeepSeek's loss-free bias approach as one of its core topics, alongside router choice (softmax, sigmoid, or sqrt-softplus) and whether to keep routing computation in fp32. The guide is based on the team's own 800k GPU hours of training runs rather than theory alone. Teams weighing MoE routing and balancing strategies can follow guides like this on daily.dev."}},{"@type":"Question","name":"Is it possible to serve a mixture-of-experts model with fewer active experts per token than it was trained with?","acceptedAnswer":{"@type":"Answer","text":"Hugging Face's MoE training handbook addresses this exact question as one of its practical topics, alongside upcycling a dense checkpoint into an MoE versus training from scratch. The handbook runs about 70,000 words with 55 interactive figures and runnable code, drawn from the team's own experimental runs. daily.dev helps engineers planning MoE deployment tradeoffs keep up with hands-on guides like this."}}]} ```