--- title: "Why Decade-Old Residual Connections Still Power All of AI (And Why That’s a Problem)" url: https://daily.dev/posts/why-decade-old-residual-connections-still-power-all-of-ai-and-why-that-s-a-problem--srohkbexi source_url: https://towardsdatascience.com/why-this-decade-old-idea-still-powers-all-of-ai-and-why-its-a-problem type: article source: "Towards Data Science" published: 2026-06-12T17:11:54.500Z updated: 2026-06-12T17:12:21.053Z tags: ["llm", "neural-networks", "deepseek"] reading_time: 14 upvotes: 1 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Why Decade-Old Residual Connections Still Power All of AI (And Why That’s a Problem) **[Towards Data Science](https://daily.dev/sources/tds)** · 14 min read · 1 upvotes · 0 comments ## Summary DeepSeek researchers propose Manifold-Constrained Hyper-Connections (mHC), a redesign of the residual connection mechanism that has powered neural networks since ResNets in 2015. Standard residual connections create an information bottleneck as models scale. ByteDance's earlier Hyper-Connections (HC) widened the residual stream but introduced signal explosion (up to 3,000x amplification) and GPU memory overhead. mHC fixes this by constraining the residual mapping matrix to a doubly stochastic matrix via the Sinkhorn-Knopp algorithm, capping spectral norm at 1 to prevent gradient instability. Hardware optimizations including kernel fusion via TileLang, selective recomputation, and overlapping communication reduce the training overhead to just 6.7%. Experiments on models up to 27B parameters show improved training stability, better benchmark performance (especially on reasoning tasks), and consistent scaling behavior. Trade-offs include the 6.7% compute cost premium, high implementation complexity requiring custom GPU kernels, and a mathematical approximation that may drift at 500B+ parameter scales. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [DeepSeek's mHC: When Residual Connections Explode](https://daily.dev/posts/deepseek-s-mhc-when-residual-connections-explode-46wypc9bp) · Hacker News · 4 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#neural-networks](https://daily.dev/tags/neural-networks), [#deepseek](https://daily.dev/tags/deepseek) [View this post on daily.dev](https://daily.dev/posts/why-decade-old-residual-connections-still-power-all-of-ai-and-why-that-s-a-problem--srohkbexi)