✳flâneur — a map of the web's best reading

Hopfield Networks is All You Need | hopfield-layers

ml-jku.github.io · 5,041 words · saved by 1 readers

This blog post explains the paper Hopfield Networks is All You Need and the corresponding new PyTorch Hopfield layer. We introduce a new energy function and a corresponding new update rule which is guaranteed to converge to a local minimum of the energy function. The new energy function is a generalization (discrete states ⇒ ⇒ continuous states) of modern Hopfield Networks aka Dense Associative Memories introduced by Krotov and Hopfield and Demircigil et al. The new modern Hopfield Network with continuous states keeps the characteristics of its discrete counterparts: Due to its continuous states this new modern Hopfield Network is differentiable and can be integrated into deep learning architectures. Typically patterns are retrieved after one update which is compatible with activating the layers of deep networks. This enables an abundance of new deep learning architectures. Three useful types of Hopfield layers are provided. Surprisingly, the new update rule is the attention mechanis

Hopfield Networks is All You Need | hopfield-layers This blog post explains the paper Hopfield Networks is All You Need and the corresponding new PyTorch Hopfield layer . Table of Contents Main contributions What this blog post is about From classical Hopfield Networks to self-attention Hopfield Networks Modern Hopfield Networks New energy function for continuous-valued patterns and states The update of the new energy function is the self-attention of transformer networks Hopfield layers for Deep Learning architectures Layer Hopfield Layer HopfieldLayer Hopfield Lookup via HopfieldLayer Layer

Explore this link on the map →

related reading