[1606.08415] Gaussian Error Linear Units (GELUs)

Abstract:We propose the Gaussian Error Linear Unit (GELU), a high-performing neural network activation function. The GELU activation function is $x\Phi(x)$, where $\Phi(x)$ the standard Gaussian cumulative distribution function. The GELU nonlinearity weights inputs by their value, rather than gates inputs by their sign as in ReLUs ($x\mathbf{1}_{x>0}$). We perform an empirical evaluation of the GELU nonlinearity against the ReLU and ELU activations and find performance improvements across all considered computer vision, natural language processing, and speech tasks.

G AUSSIAN E RROR L INEAR U NITS (GELU S ) Dan Hendrycks∗ Kevin Gimpel University of California, Berkeley Toyota Technological Institute at Chicago hendrycks@berkeley.edu kgimpel@ttic.edu A BSTRACT We propose the Gaussian Error Linear Unit (GELU), a…

related reading

• [1606.08415] Gaussian Error Linear Units (GELUs)arxiv.org
• Activation function and GLU variants for Transformer models | by Tarique Anwar | Mediummedium.com
• ReLU Activation Function in Deep Learning - GeeksforGeeksgeeksforgeeks.org
• The Decade of Deep Learning | Leo Gaobmk.sh
• 2002.05202arxiv.org
• The Role of Neural Network Activation Functionsarxiv.org
• [2002.05202] GLU Variants Improve Transformerar5iv.labs.arxiv.org
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• [2002.05202] GLU Variants Improve Transformerarxiv.org
• Neural networks and deep learningneuralnetworksanddeeplearning.com
• interpreting GPT: the logit lens — LessWronglesswrong.com
• Convolutional Neural Networks, Explained | Towards Data Sciencetowardsdatascience.com