Transformer Architecture: The Positional Encoding - Amirhossein Kazemnejad's Blog

Transformer architecture was introduced as a novel pure attention-only sequence-to-sequence architecture by Vaswani et al. Its ability for parallelizable training and its general performance improvement made it a popular option among NLP (and recently CV) researchers. Thanks to the several implementations in common deep learning frameworks, it became an easy option to experiment with for many students (including myself). Even though making it more accessible is a great thing, but on the downside it may cause the details of the model to be ignored. In this article, I don’t plan to explain its architecture in depth as there are currently several great tutorials on this topic (here, here, and here), but alternatively, I want to discuss one specific part of the transformer’s architecture - the positional encoding. When I read this part of the paper, it raised some questions in my head, which unfortunately the author had not provided sufficient information to answer them. So in this article

Transformer Architecture: The Positional Encoding - Amirhossein Kazemnejad's Blog --> Table of Content What is positional encoding and Why do we need it in the first place? Proposed method The intuition Other details Relative Positioning FAQ Summary References Transformer architecture was introduced as a novel pure attention-only sequence-to-sequence architecture by Vaswani et al. Its ability for parallelizable training and its general performance improvement made it a popular option among NLP (and recently CV) researchers. Thanks to the several implementations in common deep learning framewor

related reading

• Master Positional Encoding: Part I | Towards Data Sciencetowardsdatascience.com
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transformers from Scratche2eml.school
• Everything About Transformerskrupadave.com
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• transformer_attention.pdfarxiv.org
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Relative Positional Encoding - Jake Taejaketae.github.io
• The Annotated Transformernlp.seas.harvard.edu
• 9 Transformers – 6.390 - Intro to Machine Learningintroml.mit.edu