Activation function and GLU variants for Transformer models | by Tarique Anwar | Medium

Characterizing the first week of April 2022 as happening in the field of AI and Deep Learning would be an understatement. Within the same…

Activation function and GLU variants for Transformer models Tarique Anwar 8 min read · Apr 18, 2022 -- 1 Listen Share Characterizing the first week of April 2022 as happening in the field of AI and Deep Learning would be an understatement. Within the same week, Google and OpenAI showcased their models PaLM⁶ and DALLE 2. PaLM is a 540 billion parameter transformer-based language model seemingly capable of a state-of-the-art performance on a multitude of tasks in natural language. In addition to those, breakthrough capabilities are also demonstrated in reasoning tasks. DALLE 2 is an AI model whi

saved by

• Amos You

related reading

• [2002.05202] GLU Variants Improve Transformerar5iv.labs.arxiv.org
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• 2002.05202arxiv.org
• transformer_attention.pdfarxiv.org
• Tensor-Transformer Variants are Surprisingly Performant — LessWronglesswrong.com
• Softplus Gated Attention > Sigmoid Gated Attention1a3orn.com
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• ReLU Activation Function in Deep Learning - GeeksforGeeksgeeksforgeeks.org
• [1606.08415] Gaussian Error Linear Units (GELUs)arxiv.org
• interpreting GPT: the logit lens — LessWronglesswrong.com