mHC

For a decade, the residual connection has been the bedrock of deep learning. Every ResNet, every Transformer, every LLM relies on the same formula: This paradigm succeeded because of identity mapping the 𝑥 𝑙 x l ​ term passes unchanged through the network, stabilizing gradient flow. But this conservative design leaves performance on the table. The residual stream is narrow (C dimensions), and layers can only add to it, never reorganize it. Recent work on Hyper-Connections(HC) tried to break this ceiling by expanding the residual stream width and diversifying connectivity patterns. It worked, models got measurably better. But as DeepSeek discovered when scaling to 27B parameters, this diversification fundamentally compromises the identity mapping property. The result: severe training instability and restricted scalability, compounded by substantial memory access overhead from the wider stream. The paper proposes Manifold-Constrained Hyper-Connections (mHC) as a general framework t

saved by

• Yixiong Hao