[2409.19606] Hyper-Connections

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs. arXiv Operational Status Get status notifications via email or slack

View PDF HTML (experimental) Abstract:We present hyper-connections, a simple yet effective method that can serve as an alternative to residual connections. This approach specifically addresses common drawbacks observed in residual connection variants, such as the seesaw effect between gradient vanishing and representation collapse. Theoretically, hyper-connections allow the network to adjust the strength of connections between features at different depths and dynamically rearrange layers. We conduct experiments focusing on the pre-training of large language models, including dense and…

saved by

• Sarah Pan

related reading

• arxiv.org/pdf/2512.24880#page=3.56arxiv.org
• [2512.24880] mHC: Manifold-Constrained Hyper-Connectionsarxiv.org
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• mHC: Manifold-Constrained Hyper-Connectionsalphaxiv.org
• What is Residual Connection? | Towards Data Sciencetowardsdatascience.com
• [1609.09106] HyperNetworksarxiv.org
• The Decade of Deep Learning | Leo Gaobmk.sh
• Residual neural network - Wikipediaen.wikipedia.org
• DeepSeek and the Day Before New Year'swheremachinesthink.substack.com
• Ilya 30u30arc.net
• bachlechner21a.pdfproceedings.mlr.press
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io