[2002.05202] GLU Variants Improve Transformer

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs. arXiv Operational Status Get status notifications via email or slack

[2002.05202] GLU Variants Improve Transformer Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Machine Learning arXiv:2002.05202 (cs) [Submitted on 12 Feb 2020] Title: GLU Variants Improve Transformer Authors: Noam Shazeer View a PDF of the paper titled GLU Variants Improve Transformer, by Noam Shazeer View PDF Abstract: Gated Linear Units ( arXiv:1612.08083 ) consist of the component-wise product of two linear projections, one of which is first passed through a sigmoid function. Va

related reading

• [2002.05202] GLU Variants Improve Transformerar5iv.labs.arxiv.org
• 2002.05202arxiv.org
• Activation function and GLU variants for Transformer models | by Tarique Anwar | Mediummedium.com
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Annotated Transformernlp.seas.harvard.edu
• Transformers from Scratche2eml.school
• Transformer Circuits Threadtransformer-circuits.pub
• The Annotated Transformernlp.seas.harvard.edu
• transformer_attention.pdfarxiv.org
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• [1606.08415] Gaussian Error Linear Units (GELUs)arxiv.org
• Transformer (deep learning) - Wikipediaen.wikipedia.org