Using group theory to explore the space of positional encodings for attention

Attention is a computational primitive at the core of modern language models, allowing internal representations to reference and influence each other. It’s h...

Jane Street Blog - Using group theory to explore the space of positional encodings for attention Using group theory to explore the space of positional encodings for attention Apr 22, 2026 | 13 min read Share on Facebook Share on Twitter Share on LinkedIn By: Alok Puranik Attention is a computational primitive at the core of modern language models, allowing internal representations to reference and influence each other. It’s how these models handle sequential data in the first place. Yet, naively implemented, attention doesn’t have any notion of position. In the core attention computation, you

saved by

• Rishi Kothari
• Vihan Tiwari
• John Zhang
• Bhagyesh Kumar
• Toji Funko Pop

related reading

• Master Positional Encoding: Part I | Towards Data Sciencetowardsdatascience.com
• You could have designed state of the art positional encodinghuggingface.co
• [2410.06205] Round and Round We Go! What makes Rotary Positional Encodings useful?arxiv.org
• On N-dimensional Rotary Positional Embeddingsjerryxio.ng
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• 2410.06205arxiv.org
• Your Transformer is Secretly an EOT Solver | Elements of a Vector Spaceelonlit.com
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Relative Positional Encoding - Jake Taejaketae.github.io
• Transformer Architecture: The Positional Encoding - Amirhossein Kazemnejad's Blogkazemnejad.com
• Rotary Embeddings: A Relative Revolution | EleutherAI Blogblog.eleuther.ai