N-dimensional Rotary Positional Embeddings

One of the simplest ways of encoding relative positional information in attention is to add a scalar to each of the attention logits, with a value somehow depending on the distance between the corresponding query and key (e.g. learned values in T5, fixed values decreasing with distance in ALiBi). However, this makes it difficult for a query to attend to any specific (key, relative position) pair. In particular, the query must have a component pointing in the direction of the desired key, but this increases the attention scores with all tokens with that key, regardless of their position. Rotary positional embeddings (RoPE) are an elegant solution to this problem. Essentially, the query and key vectors for every token are rotated by an angle proportional to the token's 1-d coordinate position. To be specific, each attention head has its 𝐷 channel dimensions divided into 𝐷 / 2 dimension pairs. For a given query or key input vector 𝑥 ∈ 𝑅 𝐷 located at position 𝑡 , the 𝑖 th di

On N-dimensional Rotary Positional Embeddings [back to home] On N-dimensional Rotary Positional Embeddings July 26th, 2025 · Jerry Xiong resolution min_freq max_freq n_freqs direction_spacing \(\pi / 2\) \(\pi / \varphi\) \(2\pi / \varphi\) axial Typical axial RoPE rotates half of each head's components based on distance along the x-axis, and the other half based on distance along the y-axis, which results in striped artifacts and poorly concentrated attention maps. Queries are unable to attend to tokens at specific positions without also attending to tokens with similar keys in the same row o

saved by

• Winnie Xu

related reading

• Rotary Embeddings: A Relative Revolution | EleutherAI Blogblog.eleuther.ai
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Jane Street Blog - Using group theory to explore the space of positional encodings for attentionblog.janestreet.com
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Master Positional Encoding: Part I | Towards Data Sciencetowardsdatascience.com
• Relative Positional Encoding - Jake Taejaketae.github.io
• Transformer Architecture: The Positional Encoding - Amirhossein Kazemnejad's Blogkazemnejad.com
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• GPT-2's positional embedding matrix is a helix — LessWronglesswrong.com
• Transformers from Scratche2eml.school
• Extending the Context of Pretrained LLMs by Dropping their Positional Embeddingspub.sakana.ai
• The Annotated Transformernlp.seas.harvard.edu