Relative Positional Encoding - Jake Tae

In this post, we will take a look at relative positional encoding, as introduced in Shaw et al (2018) and refined by Huang et al (2018). This is a topic I meant to explore earlier, but only recently was I able to really force myself to dive into this concept as I started reading about music generation with NLP language models. This is a separate topic for another post of its own, so let’s not get distracted.

In this post, we will take a look at relative positional encoding, as introduced in Shaw et al (2018) and refined by Huang et al (2018) . This is a topic I meant to explore earlier, but only recently was I able to really force myself to dive into this concept as I started reading about music generation with NLP language models. This is a separate topic for another post of its own, so let’s not get distracted. Let’s dive right into it! Concept If you’re already familiar with transformers, you probably know that transformers process inputs in parallel at once. This is one of the many reasons why

related reading

• Master Positional Encoding: Part I | Towards Data Sciencetowardsdatascience.com
• Transformer Architecture: The Positional Encoding - Amirhossein Kazemnejad's Blogkazemnejad.com
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Rotary Embeddings: A Relative Revolution | EleutherAI Blogblog.eleuther.ai
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• On N-dimensional Rotary Positional Embeddingsjerryxio.ng
• Transformers from Scratche2eml.school
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Everything About Transformerskrupadave.com
• Jane Street Blog - Using group theory to explore the space of positional encodings for attentionblog.janestreet.com
• transformer_attention.pdfarxiv.org
• Extending Context is Hard | kaiokendevkaiokendev.github.io