[1606.08415] Gaussian Error Linear Units (GELUs)

Abstract:We propose the Gaussian Error Linear Unit (GELU), a high-performing neural network activation function. The GELU activation function is $x\Phi(x)$, where $\Phi(x)$ the standard Gaussian cumulative distribution function. The GELU nonlinearity weights inputs by their value, rather than gates inputs by their sign as in ReLUs ($x\mathbf{1}_{x>0}$). We perform an empirical evaluation of the GELU nonlinearity against the ReLU and ELU activations and find performance improvements across all considered computer vision, natural language processing, and speech tasks.

G AUSSIAN E RROR L INEAR U NITS (GELU S ) Dan Hendrycks∗ Kevin Gimpel University of California, Berkeley Toyota Technological Institute at Chicago hendrycks@berkeley.edu kgimpel@ttic.edu A BSTRACT We propose the Gaussian Error Linear Unit (GELU), a…

related reading

• [1606.08415] Gaussian Error Linear Units (GELUs)arxiv.org
• Toy Models of Superpositiontransformer-circuits.pub
• [2002.05202] GLU Variants Improve Transformerar5iv.labs.arxiv.org
• Activation function and GLU variants for Transformer models | by Tarique Anwar | Mediummedium.com
• ReLU Activation Function in Deep Learning - GeeksforGeeksgeeksforgeeks.org
• 2002.05202arxiv.org
• The Role of Neural Network Activation Functionsarxiv.org
• [1706.04599] On Calibration of Modern Neural Networksarxiv.org
• Softmax Linear Unitstransformer-circuits.pub
• [1911.08731] Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationarxiv.org
• arxiv.org/pdf/1805.08522arxiv.org
• guo17a.pdfproceedings.mlr.press