[2410.06205] Round and Round We Go! What makes Rotary Positional Encodings useful?

Abstract:Positional Encodings (PEs) are a critical component of Transformer-based Large Language Models (LLMs), providing the attention mechanism with important sequence-position information. One of the most popular types of encoding used today in LLMs are Rotary Positional Encodings (RoPE), that rotate the queries and keys based on their relative distance. A common belief is that RoPE is useful because it helps to decay token dependency as relative distance increases. In this work, we argue that this is unlikely to be the core reason. We study the internals of a trained Gemma 7B model to understand how RoPE is being used at a mechanical level. We find that Gemma learns to use RoPE to construct robust "positional" attention patterns by exploiting the highest frequencies. We also find that, in general, Gemma greatly prefers to use the lowest frequencies of RoPE, which we suspect are used to carry semantic information. We mathematically prove interesting behaviours of RoPE and conduct experiments to verify our findings, proposing a modification of RoPE that fixes some highlighted issues and improves performance. We believe that this work represents an interesting step in better understanding PEs in LLMs, which we believe holds crucial value for scaling LLMs to large sizes and context lengths.

Published as a conference paper at ICLR 2025 ROUND AND ROUND W E G O ! L W HAT MAKES ROTARY P OSITIONAL E NCODINGS USEFUL ? Federico Barbero˚ Alex Vitvitskyi Christos Perivolaropoulos University of Oxford Google DeepMind Google DeepMind Razvan Pascanu Petar…

saved by

• Justin Wang

related reading

• 2410.06205arxiv.org
• On N-dimensional Rotary Positional Embeddingsjerryxio.ng
• Rotary Embeddings: A Relative Revolution | EleutherAI Blogblog.eleuther.ai
• You could have designed state of the art positional encodinghuggingface.co
• Jane Street Blog - Using group theory to explore the space of positional encodings for attentionblog.janestreet.com
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Master Positional Encoding: Part I | Towards Data Sciencetowardsdatascience.com
• How LLMs Actually Work | 0xkato0xkato.xyz
• Extending the Context of Pretrained LLMs by Dropping their Positional Embeddingspub.sakana.ai
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io