[1606.08415] Gaussian Error Linear Units (GELUs)

Abstract:We propose the Gaussian Error Linear Unit (GELU), a high-performing neural network activation function. The GELU activation function is $x\Phi(x)$, where $\Phi(x)$ the standard Gaussian cumulative distribution function. The GELU nonlinearity weights inputs by their value, rather than gates inputs by their sign as in ReLUs ($x\mathbf{1}_{x>0}$). We perform an empirical evaluation of the GELU nonlinearity against the ReLU and ELU activations and find performance improvements across all considered computer vision, natural language processing, and speech tasks.

related reading

• [1606.08415] Gaussian Error Linear Units (GELUs)arxiv.org
• Activation function and GLU variants for Transformer models | by Tarique Anwar | Mediummedium.com
• 2002.05202arxiv.org
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• [2002.05202] GLU Variants Improve Transformerar5iv.labs.arxiv.org
• The Decade of Deep Learning | Leo Gaobmk.sh
• ReLU Activation Function in Deep Learning - GeeksforGeeksgeeksforgeeks.org
• [2002.05202] GLU Variants Improve Transformerarxiv.org
• Tiny TPUtinytpu.com
• gpt-4.pdfcdn.openai.com
• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activationstransformer-circuits.pub
• Capítulo 8 - Função de Ativação - Deep Learning Bookdeeplearningbook.com.br