Rotary Embeddings: A Relative Revolution | EleutherAI Blog

Rotary Positional Embedding (RoPE) is a new type of position encoding that unifies absolute and relative approaches. We put it to the test.

Rotary position encoding as imagined by Janus TL;DR: # Rotary Positional Embedding (RoPE) is a new type of position encoding that unifies absolute and relative approaches. Developed by Jianlin Su in a series of blog posts earlier this year [12, 13] and in a new preprint [14], it has already garnered widespread interest in some Chinese NLP circles. This post walks through the method as we understand it, with the goal of bringing it to the attention of the wider academic community. In general we have found that across a large suite of setups including regular, linear, and local self-attention, i

related reading

• On N-dimensional Rotary Positional Embeddingsjerryxio.ng
• Relative Positional Encoding - Jake Taejaketae.github.io
• Master Positional Encoding: Part I | Towards Data Sciencetowardsdatascience.com
• Transformer Architecture: The Positional Encoding - Amirhossein Kazemnejad's Blogkazemnejad.com
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• GPT-2's positional embedding matrix is a helix — LessWronglesswrong.com
• Transformers from Scratche2eml.school
• Jane Street Blog - Using group theory to explore the space of positional encodings for attentionblog.janestreet.com
• Extending the Context of Pretrained LLMs by Dropping their Positional Embeddingspub.sakana.ai
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Tiny Mech Interp Projects: Emergent Positional Embeddings of Words - Neel Nandaneelnanda.io