Using group theory to explore the space of positional encodings for attention

Attention is a computational primitive at the core of modern language models, allowing internal representations to reference and influence each other. It’s h...

Jane Street Blog - Using group theory to explore the space of positional encodings for attention Using group theory to explore the space of positional encodings for attention Apr 22, 2026 | 13 min read Share on Facebook Share on Twitter Share on LinkedIn By: Alok Puranik Attention is a computational primitive at the core of modern language models, allowing internal representations to reference and influence each other. It’s how these models handle sequential data in the first place. Yet, naively implemented, attention doesn’t have any notion of position. In the core attention computation, you

saved by

• Rishi Kothari
• Vihan Tiwari
• John Zhang
• Bhagyesh Kumar
• Toji Funko Pop

related reading

• Positional Encodings and Group Theory | 3Blue1Brown and Alok Puranikyoutube.com
• You could have designed state of the art positional encodinghuggingface.co
• Verbalizable Representations Form a Global Workspace in Language Modelstransformer-circuits.pub
• On N-dimensional Rotary Positional Embeddingsjerryxio.ng
• [2410.06205] Round and Round We Go! What makes Rotary Positional Encodings useful?arxiv.org
• Master Positional Encoding: Part I | Towards Data Sciencetowardsdatascience.com
• Rotary Embeddings: A Relative Revolution | EleutherAI Blogblog.eleuther.ai
• Understanding the J-Lensemma-x1.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• 2410.06205arxiv.org
• The absolute basics of representation theory of finite groups — LessWronglesswrong.com