Why Decade-Old Residual Connections Still Power All of AI (And Why That’s a Problem)
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
DeepSeek researchers propose Manifold-Constrained Hyper-Connections (mHC), a redesign of the residual connection mechanism that has powered neural networks since ResNets in 2015. Standard residual connections create an information bottleneck as models scale. ByteDance's earlier Hyper-Connections (HC) widened the residual stream but introduced signal explosion (up to 3,000x amplification) and GPU memory overhead. mHC fixes this by constraining the residual mapping matrix to a doubly stochastic matrix via the Sinkhorn-Knopp algorithm, capping spectral norm at 1 to prevent gradient instability. Hardware optimizations including kernel fusion via TileLang, selective recomputation, and overlapping communication reduce the training overhead to just 6.7%. Experiments on models up to 27B parameters show improved training stability, better benchmark performance (especially on reasoning tasks), and consistent scaling behavior. Trade-offs include the 6.7% compute cost premium, high implementation complexity requiring custom GPU kernels, and a mathematical approximation that may drift at 500B+ parameter scales.
