GLU Variants Improve Transformer | HTML5
Gated Linear Units [Dauphin et al., 2016] consist of the component-wise product of two linear projections, one of which is first passed through a sigmoid function. Variations on GLU are possible, using different nonlinear (or even linear) functions in place of sigmoid. We test these variants in the feed-forward sublayers of the Transformer [Vaswani et al., 2017] sequence-to-sequence model, and find that some of them yield quality improvements over the typically-used ReLU or GELU activations. The Transformer [Vaswani et al., 2017] sequence-to-sequence model alternates between multi-head attention, and what it calls "position-wise feed-forward networks" (FFN). The FFN takes a vector 𝑥 (the hidden representation at a particular position in the sequence) and passes it through two learned linear transformations, (represented by the matrices 𝑊 1 and 𝑊 2 and bias vectors 𝑏 1 and 𝑏 2 ). A rectified-linear (ReLU) [Glorot et al., 2011] activation function applied between the two li
GLU Variants Improve Transformer Noam Shazeer Google noam@google.com Abstract Gated Linear Units [Dauphin et al., 2016 ] consist of the component-wise product of two linear projections, one of which is first passed through a sigmoid function. Variations on GLU are possible, using different nonlinear (or even linear) functions in place of sigmoid. We test these variants in the feed-forward sublayers of the Transformer [Vaswani et al., 2017 ] sequence-to-sequence model, and find that some of them yield quality improvements over the typically-used ReLU or GELU activations. 1 Introduction The Tran
Explore this link on the map →related reading
- 2002.05202arxiv.org
- [2002.05202] GLU Variants Improve Transformerarxiv.org
- Activation function and GLU variants for Transformer models | by Tarique Anwar | Mediummedium.com
- A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
- The Annotated Transformernlp.seas.harvard.edu
- The Annotated Transformernlp.seas.harvard.edu
- Transformers from Scratche2eml.school
- The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
- Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
- transformer_attention.pdfarxiv.org
- Transformer (deep learning) - Wikipediaen.wikipedia.org
- 1706.03762arxiv.org