N-dimensional Rotary Positional Embeddings
One of the simplest ways of encoding relative positional information in attention is to add a scalar to each of the attention logits, with a value somehow depending on the distance between the corresponding query and key (e.g. learned values in T5, fixed values decreasing with distance in ALiBi). However, this makes it difficult for a query to attend to any specific (key, relative position) pair. In particular, the query must have a component pointing in the direction of the desired key, but this increases the attention scores with all tokens with that key, regardless of their position. Rotary positional embeddings (RoPE) are an elegant solution to this problem. Essentially, the query and key vectors for every token are rotated by an angle proportional to the token's 1-d coordinate position. To be specific, each attention head has its 𝐷 channel dimensions divided into 𝐷 / 2 dimension pairs. For a given query or key input vector 𝑥 ∈ 𝑅 𝐷 located at position 𝑡 , the 𝑖 th di
On N-dimensional Rotary Positional Embeddings [back to home] On N-dimensional Rotary Positional Embeddings July 26th, 2025 · Jerry Xiong resolution min_freq max_freq n_freqs direction_spacing \(\pi / 2\) \(\pi / \varphi\) \(2\pi / \varphi\) axial Typical axial RoPE rotates half of each head's components based on distance along the x-axis, and the other half based on distance along the y-axis, which results in striped artifacts and poorly concentrated attention maps. Queries are unable to attend to tokens at specific positions without also attending to tokens with similar keys in the same row o
saved by
related reading
- Rotary Embeddings: A Relative Revolution | EleutherAI Blogblog.eleuther.ai
- [2410.06205] Round and Round We Go! What makes Rotary Positional Encodings useful?arxiv.org
- 2410.06205arxiv.org
- You could have designed state of the art positional encodinghuggingface.co
- Jane Street Blog - Using group theory to explore the space of positional encodings for attentionblog.janestreet.com
- The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
- Master Positional Encoding: Part I | Towards Data Sciencetowardsdatascience.com
- Relative Positional Encoding - Jake Taejaketae.github.io
- The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
- Transformer Architecture: The Positional Encoding - Amirhossein Kazemnejad's Blogkazemnejad.com
- GPT-2's positional embedding matrix is a helix — LessWronglesswrong.com
- Extending Context is Hard | kaiokendevkaiokendev.github.io