Rotary Embeddings: A Relative Revolution | EleutherAI Blog

Rotary Positional Embedding (RoPE) is a new type of position encoding that unifies absolute and relative approaches. We put it to the test.

Rotary position encoding as imagined by Janus TL;DR: # Rotary Positional Embedding (RoPE) is a new type of position encoding that unifies absolute and relative approaches. Developed by Jianlin Su in a series of blog posts earlier this year [12, 13] and in a new preprint [14], it has already garnered widespread interest in some Chinese NLP circles. This post walks through the method as we understand it, with the goal of bringing it to the attention of the wider academic community. In general we have found that across a large suite of setups including regular, linear, and local self-attention, i

related reading

• On N-dimensional Rotary Positional Embeddingsjerryxio.ng
• [2410.06205] Round and Round We Go! What makes Rotary Positional Encodings useful?arxiv.org
• You could have designed state of the art positional encodinghuggingface.co
• 2410.06205arxiv.org
• Relative Positional Encoding - Jake Taejaketae.github.io
• Master Positional Encoding: Part I | Towards Data Sciencetowardsdatascience.com
• Transformer Architecture: The Positional Encoding - Amirhossein Kazemnejad's Blogkazemnejad.com
• Jane Street Blog - Using group theory to explore the space of positional encodings for attentionblog.janestreet.com
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• GPT-2's positional embedding matrix is a helix — LessWronglesswrong.com
• Transformers from Scratche2eml.school