[2410.06205] Round and Round We Go! What makes Rotary Positional Encodings useful?

Abstract:Positional Encodings (PEs) are a critical component of Transformer-based Large Language Models (LLMs), providing the attention mechanism with important sequence-position information. One of the most popular types of encoding used today in LLMs are Rotary Positional Encodings (RoPE), that rotate the queries and keys based on their relative distance. A common belief is that RoPE is useful because it helps to decay token dependency as relative distance increases. In this work, we argue that this is unlikely to be the core reason. We study the internals of a trained Gemma 7B model to understand how RoPE is being used at a mechanical level. We find that Gemma learns to use RoPE to construct robust "positional" attention patterns by exploiting the highest frequencies. We also find that, in general, Gemma greatly prefers to use the lowest frequencies of RoPE, which we suspect are used to carry semantic information. We mathematically prove interesting behaviours of RoPE and conduct experiments to verify our findings, proposing a modification of RoPE that fixes some highlighted issues and improves performance. We believe that this work represents an interesting step in better understanding PEs in LLMs, which we believe holds crucial value for scaling LLMs to large sizes and context lengths.

Published as a conference paper at ICLR 2025 ROUND AND ROUND W E G O ! L W HAT MAKES ROTARY P OSITIONAL E NCODINGS USEFUL ? Federico Barbero˚ Alex Vitvitskyi Christos Perivolaropoulos University of Oxford Google DeepMind Google DeepMind Razvan Pascanu Petar…

saved by

• Justin Wang

related reading

• 2410.06205arxiv.org
• On N-dimensional Rotary Positional Embeddingsjerryxio.ng
• Jane Street Blog - Using group theory to explore the space of positional encodings for attentionblog.janestreet.com
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Rotary Embeddings: A Relative Revolution | EleutherAI Blogblog.eleuther.ai
• Extending the Context of Pretrained LLMs by Dropping their Positional Embeddingspub.sakana.ai
• You could have designed state of the art positional encodinghuggingface.co
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• [2507.19457] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• The Big LLM Architecture Comparisonmagazine.sebastianraschka.com
• 1706.03762arxiv.org