You could have designed state of the art positional encoding

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Gall's Law A complex system that works is invariably found to have evolved from a simple system that worked John Gall This post walks you through the step-by-step discovery of state-of-the-art positional encoding in transformer models. We will achieve this by iteratively improving our approach to encoding position, arriving at Rotary Postional Encoding (RoPE) used in the latest LLama 3.2 release and most modern transformers. This post intends to limit the mathematical knowledge required to follow along, but some basic linear algebra, trigonometry and understanding of self attention is…

related reading

• Jane Street Blog - Using group theory to explore the space of positional encodings for attentionblog.janestreet.com
• Rotary Embeddings: A Relative Revolution | EleutherAI Blogblog.eleuther.ai
• [2410.06205] Round and Round We Go! What makes Rotary Positional Encodings useful?arxiv.org
• On N-dimensional Rotary Positional Embeddingsjerryxio.ng
• Master Positional Encoding: Part I | Towards Data Sciencetowardsdatascience.com
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• 2410.06205arxiv.org
• Relative Positional Encoding - Jake Taejaketae.github.io
• Positional Encodings and Group Theory | 3Blue1Brown and Alok Puranikyoutube.com
• The Annotated Transformernlp.seas.harvard.edu
• transformer_attention.pdfarxiv.org