--- title: How mHC Reinvents Residual Connections | daily.dev description: Residual connections have been a cornerstone of deep learning since ResNets, enabling gradient flow and stable training of deep models. Hyper Connections (mHC)... canonical: https://daily.dev/posts/how-mhc-reinvents-residual-connections-8dmpvwaxs twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How mHC Reinvents Residual Connections | daily.dev og:description: Residual connections have been a cornerstone of deep learning since ResNets, enabling gradient flow and stable training of deep models. Hyper Connections (mHC)... og:url: https://daily.dev/posts/how-mhc-reinvents-residual-connections-8dmpvwaxs og:image: https://api.daily.dev/og/posts/8DMPvWAXs.png og:image:alt: How mHC Reinvents Residual Connections og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How mHC Reinvents Residual Connections **[Jia-Bin Huang](https://daily.dev/sources/jbhuang0604)** · 13 min read · 0 upvotes · 0 comments ## Summary Residual connections have been a cornerstone of deep learning since ResNets, enabling gradient flow and stable training of deep models. Hyper Connections (mHC) extend this by expanding input features into multiple parallel residual streams, using learnable aggregation, expansion, and feature-mixing matrices to dynamically route information across streams. This yields up to 1.8x faster convergence over standard residuals. However, DeepSeek found that unconstrained feature-mixing matrices cause training instability — values slightly above or below 1 compound exponentially across layers. The fix is constraining the mixing matrix to be doubly stochastic (all elements positive, rows and columns summing to 1) via the Sinkhorn algorithm. DeepSeek also switched activation functions from tanh to sigmoid and added a scalar factor of 2 to ensure hyperconnections initialize identically to standard residuals. Three infrastructure optimizations (fused kernels, activation recomputation, and pipeline overlapping) keep the training overhead to just 6.7% with an expansion rate of 4. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [DeepSeek's mHC: When Residual Connections Explode](https://daily.dev/posts/deepseek-s-mhc-when-residual-connections-explode-46wypc9bp) · Hacker News · 4 upvotes · 0 comments - [Why Decade-Old Residual Connections Still Power All of AI \(And Why That’s a Problem\)](https://daily.dev/posts/why-decade-old-residual-connections-still-power-all-of-ai-and-why-that-s-a-problem--srohkbexi) · Towards Data Science · 1 upvotes · 0 comments - [Residual Connections: The Innovation That Made Deep Transformers Possible](https://daily.dev/posts/residual-connections-the-innovation-that-made-deep-transformers-possible-7livr4rhb) · Medium · 0 upvotes · 0 comments - [A New Way to Handle Residual Connections in Transformers](https://daily.dev/posts/a-new-way-to-handle-residual-connections-in-transformers-yo1k3snuy) · Daily Dose of Data Science \| Avi Chawla \| Substack · 5 upvotes · 0 comments --- Tags: [#deep-learning](https://daily.dev/tags/deep-learning), [#neural-networks](https://daily.dev/tags/neural-networks), [#deepseek](https://daily.dev/tags/deepseek) [View this post on daily.dev](https://daily.dev/posts/how-mhc-reinvents-residual-connections-8dmpvwaxs) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How mHC Reinvents Residual Connections","url":"https://daily.dev/posts/how-mhc-reinvents-residual-connections-8dmpvwaxs","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-mhc-reinvents-residual-connections-8dmpvwaxs"},"datePublished":"2026-04-19T08:05:22.729Z","dateModified":"2026-04-19T08:06:47.538Z","description":"Residual connections have been a cornerstone of deep learning since ResNets, enabling gradient flow and stable training of deep models. Hyper Connections (mHC)...","image":"https://i.ytimg.com/vi/jYn_1PpRzxI/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/jYn_1PpRzxI/sddefault.jpg","isAccessibleForFree":true,"articleSection":"Jia-Bin Huang","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Jia-Bin Huang","logo":"https://media.daily.dev/image/upload/s--8hwFrfWM--/f_auto,q_auto/v1776585907/logos/jbhuang0604","url":"https://daily.dev/sources/jbhuang0604"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-mhc-reinvents-residual-connections-8dmpvwaxs","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"deep-learning,neural-networks,deepseek","timeRequired":"PT13M","video":{"@type":"VideoObject","name":"How mHC Reinvents Residual Connections","description":"Residual connections have been a cornerstone of deep learning since ResNets, enabling gradient flow and stable training of deep models. Hyper Connections (mHC)...","thumbnailUrl":"https://i.ytimg.com/vi/jYn_1PpRzxI/sddefault.jpg","uploadDate":"2026-04-19T08:05:22.729Z","duration":"PT13M","url":"https://api.daily.dev/r/8DMPvWAXs","embedUrl":"https://www.youtube.com/embed/jYn_1PpRzxI"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Jia-Bin Huang","item":"https://daily.dev/sources/jbhuang0604"},{"@type":"ListItem","position":3,"name":"How mHC Reinvents Residual Connections"}]} ```