[1609.09106] HyperNetworks

Abstract:This work explores hypernetworks: an approach of using a one network, also known as a hypernetwork, to generate the weights for another network. Hypernetworks provide an abstraction that is similar to what is found in nature: the relationship between a genotype - the hypernetwork - and a phenotype - the main network. Though they are also reminiscent of HyperNEAT in evolution, our hypernetworks are trained end-to-end with backpropagation and thus are usually faster. The focus of this work is to make hypernetworks useful for deep convolutional networks and long recurrent networks, where hypernetworks can be viewed as relaxed form of weight-sharing across layers. Our main result is that hypernetworks can generate non-shared weights for LSTM and achieve near state-of-the-art results on a variety of sequence modelling tasks including character-level language modelling, handwriting generation and neural machine translation, challenging the weight-sharing paradigm for recurrent networks. Our results also show that hypernetworks applied to convolutional networks still achieve respectable results for image recognition tasks compared to state-of-the-art baseline models while requiring fewer learnable parameters.

HyperNetworks This work explores hypernetworks: an approach of using a one network, also known as a hypernetwork, to generate the weights for another network. Hypernetworks provide an abstraction that is similar to what is found in nature: the relationship between a genotype - the hypernetwork - and a phenotype - the main network. Though they are also reminiscent of HyperNEAT in evolution, our hypernetworks are trained end-to-end with backpropagation and thus are usually faster. The focus of this work is to make hypernetworks useful for deep convolutional networks and long recurrent networks,

saved by

• Lev Chizhov

related reading

• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• arxiv.org/pdf/1805.08522arxiv.org
• The Unreasonable Effectiveness of Recurrent Neural Networkskarpathy.github.io
• The Decade of Deep Learning | Leo Gaobmk.sh
• The Scaling Hypothesis · Gwern.netgwern.net
• Researchers Build AI That Builds AI | Quanta Magazinequantamagazine.org
• H-Nets - the Past | Goomba Labgoombalab.github.io
• Neural networks and deep learningneuralnetworksanddeeplearning.com
• arxiv.org/pdf/2512.24880#page=3.56arxiv.org
• Neural Networks, Types, and Functional Programming -- colah's blogcolah.github.io
• CS231n Deep Learning for Computer Visioncs231n.github.io
• [2404.19756] KAN: Kolmogorov-Arnold Networksarxiv.org