--- title: Mixture of Experts: Explained & Implemented | daily.dev description: A conceptual and practical walkthrough of Mixture of Experts (MoE), covering how the architecture works within transformers, why it reduces compute (without... canonical: https://daily.dev/posts/mixture-of-experts-explained-implemented-wckplhf1m twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Mixture of Experts: Explained & Implemented | daily.dev og:description: A conceptual and practical walkthrough of Mixture of Experts (MoE), covering how the architecture works within transformers, why it reduces compute (without... og:url: https://daily.dev/posts/mixture-of-experts-explained-implemented-wckplhf1m og:image: https://api.daily.dev/og/posts/wckpLHF1m.png og:image:alt: Mixture of Experts: Explained & Implemented og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Mixture of Experts: Explained & Implemented **[NeuralNine](https://daily.dev/sources/neuralnine)** · 24 min read · 2 upvotes · 0 comments ## Summary A conceptual and practical walkthrough of Mixture of Experts (MoE), covering how the architecture works within transformers, why it reduces compute (without reducing VRAM), and how to implement it from scratch in PyTorch. The implementation includes an Expert class, a SparseMoE class with a linear router, top-k expert selection, weight normalization, and a load balancing loss to prevent routing collapse. A trivial training demo (learning to multiply by 2) illustrates expert selection and weighting in action. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do you implement a mixture of experts layer from scratch in PyTorch? Build an Expert class as a small feed-forward network (linear, GELU, linear), then a SparseMoE class containing a router (a single linear layer producing per-expert logits) and a ModuleList of experts. In the forward pass, apply softmax to router logits, take the top-k experts via torch.topk, renormalize their weights, route each token's subset to its assigned experts using torch.where, and combine outputs with index_add_ weighted by the normalized probabilities. _Explore more deep-learning architecture walkthroughs like this on daily.dev when building your own model layers._ ### What is a load balancing loss in mixture of experts models and why is it needed? A load balancing loss penalizes a router for consistently favoring only one or two experts instead of distributing tokens evenly. It is computed as the number of experts times the sum of the mean router probability multiplied by the fraction of tokens selecting each expert; a perfectly balanced router yields a loss of 1, while a fully unbalanced one yields a loss equal to the number of experts, discouraging degenerate routing. _Developers tuning expert routing can find more architecture deep dives like this on daily.dev._ ### Why does mixture of experts reduce compute compared to a standard feed-forward transformer layer? Mixture of experts reduces compute because only a small subset of experts (e.g., top 2 out of 8) is activated per token instead of running every token through one large feed-forward network. This lowers the number of active weight computations during training and inference, though it does not reduce VRAM usage since all expert weights must still be stored in memory. _Anyone weighing compute trade-offs in model architecture can track explainers like this on daily.dev._ ## Similar posts on daily.dev - [Mixture of Experts \(MoEs\) in Transformers](https://daily.dev/posts/mixture-of-experts-moes-in-transformers-wclvbejkn) · Hugging Face · 1 upvotes · 0 comments - [What Is Mixture-of-Experts \(MoE\) Architecture](https://daily.dev/posts/what-is-mixture-of-experts-moe-architecture-bhakdfzdn) · C\# Corner · 1 upvotes · 0 comments - [How the Mixture of Experts Architecture Works in AI Models](https://daily.dev/posts/how-the-mixture-of-experts-architecture-works-in-ai-models-im7k7iyrk) · freeCodeCamp · 2 upvotes · 0 comments - [DeepSeek-V3 from Scratch: Mixture of Experts \(MoE\)](https://daily.dev/posts/deepseek-v3-from-scratch-mixture-of-experts-moe--dexrfaga5) · PyImageSearch · 0 upvotes · 0 comments --- Tags: [#python](https://daily.dev/tags/python), [#llm](https://daily.dev/tags/llm), [#pytorch](https://daily.dev/tags/pytorch), [#mixture-of-experts](https://daily.dev/tags/mixture-of-experts) [View this post on daily.dev](https://daily.dev/posts/mixture-of-experts-explained-implemented-wckplhf1m) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Mixture of Experts: Explained & Implemented","url":"https://daily.dev/posts/mixture-of-experts-explained-implemented-wckplhf1m","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/mixture-of-experts-explained-implemented-wckplhf1m"},"datePublished":"2026-08-10T13:52:32.267Z","dateModified":"2026-09-14T06:17:26.457Z","description":"A conceptual and practical walkthrough of Mixture of Experts (MoE), covering how the architecture works within transformers, why it reduces compute (without...","image":"https://i.ytimg.com/vi/vHF7gGvAqu8/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/vHF7gGvAqu8/sddefault.jpg","isAccessibleForFree":true,"articleSection":"NeuralNine","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"NeuralNine","logo":"https://media.daily.dev/image/upload/s--bMYO07OB--/f_auto,q_auto/v1777800757/logos/neuralnine?_a=BAMAMiWQ0","url":"https://daily.dev/sources/neuralnine"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/mixture-of-experts-explained-implemented-wckplhf1m","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"python,llm,pytorch,mixture-of-experts","timeRequired":"PT24M","video":{"@type":"VideoObject","name":"Mixture of Experts: Explained & Implemented","description":"A conceptual and practical walkthrough of Mixture of Experts (MoE), covering how the architecture works within transformers, why it reduces compute (without...","thumbnailUrl":"https://i.ytimg.com/vi/vHF7gGvAqu8/sddefault.jpg","uploadDate":"2026-08-10T13:52:32.267Z","duration":"PT24M","url":"https://api.daily.dev/r/wckpLHF1m","embedUrl":"https://www.youtube.com/embed/vHF7gGvAqu8"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"NeuralNine","item":"https://daily.dev/sources/neuralnine"},{"@type":"ListItem","position":3,"name":"Mixture of Experts: Explained & Implemented"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/mixture-of-experts-explained-implemented-wckplhf1m#faq","mainEntity":[{"@type":"Question","name":"How do you implement a mixture of experts layer from scratch in PyTorch?","acceptedAnswer":{"@type":"Answer","text":"Build an Expert class as a small feed-forward network (linear, GELU, linear), then a SparseMoE class containing a router (a single linear layer producing per-expert logits) and a ModuleList of experts. In the forward pass, apply softmax to router logits, take the top-k experts via torch.topk, renormalize their weights, route each token's subset to its assigned experts using torch.where, and combine outputs with index_add_ weighted by the normalized probabilities. Explore more deep-learning architecture walkthroughs like this on daily.dev when building your own model layers."}},{"@type":"Question","name":"What is a load balancing loss in mixture of experts models and why is it needed?","acceptedAnswer":{"@type":"Answer","text":"A load balancing loss penalizes a router for consistently favoring only one or two experts instead of distributing tokens evenly. It is computed as the number of experts times the sum of the mean router probability multiplied by the fraction of tokens selecting each expert; a perfectly balanced router yields a loss of 1, while a fully unbalanced one yields a loss equal to the number of experts, discouraging degenerate routing. Developers tuning expert routing can find more architecture deep dives like this on daily.dev."}},{"@type":"Question","name":"Why does mixture of experts reduce compute compared to a standard feed-forward transformer layer?","acceptedAnswer":{"@type":"Answer","text":"Mixture of experts reduces compute because only a small subset of experts (e.g., top 2 out of 8) is activated per token instead of running every token through one large feed-forward network. This lowers the number of active weight computations during training and inference, though it does not reduce VRAM usage since all expert weights must still be stored in memory. Anyone weighing compute trade-offs in model architecture can track explainers like this on daily.dev."}}]} ```