Relative Positional Encoding - Jake Tae
In this post, we will take a look at relative positional encoding, as introduced in Shaw et al (2018) and refined by Huang et al (2018). This is a topic I meant to explore earlier, but only recently was I able to really force myself to dive into this concept as I started reading about music generation with NLP language models. This is a separate topic for another post of its own, so let’s not get distracted.
In this post, we will take a look at relative positional encoding, as introduced in Shaw et al (2018) and refined by Huang et al (2018) . This is a topic I meant to explore earlier, but only recently was I able to really force myself to dive into this concept as I started reading about music generation with NLP language models. This is a separate topic for another post of its own, so let’s not get distracted. Let’s dive right into it! Concept If you’re already familiar with transformers, you probably know that transformers process inputs in parallel at once. This is one of the many reasons why
related reading
- Master Positional Encoding: Part I | Towards Data Sciencetowardsdatascience.com
- You could have designed state of the art positional encodinghuggingface.co
- Transformer Architecture: The Positional Encoding - Amirhossein Kazemnejad's Blogkazemnejad.com
- Rotary Embeddings: A Relative Revolution | EleutherAI Blogblog.eleuther.ai
- The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
- Jane Street Blog - Using group theory to explore the space of positional encodings for attentionblog.janestreet.com
- [2410.06205] Round and Round We Go! What makes Rotary Positional Encodings useful?arxiv.org
- A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
- On N-dimensional Rotary Positional Embeddingsjerryxio.ng
- The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
- Transformers from Scratche2eml.school
- 2410.06205arxiv.org