You could have designed state of the art positional encoding

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Gall's Law A complex system that works is invariably found to have evolved from a simple system that worked John Gall This post walks you through the step-by-step discovery of state-of-the-art positional encoding in transformer models. We will achieve this by iteratively improving our approach to encoding position, arriving at Rotary Postional Encoding (RoPE) used in the latest LLama 3.2 release and most modern transformers. This post intends to limit the mathematical knowledge required to follow along, but some basic linear algebra, trigonometry and understanding of self attention is…

related reading

• Master Positional Encoding: Part I | Towards Data Sciencetowardsdatascience.com
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• [2410.06205] Round and Round We Go! What makes Rotary Positional Encodings useful?arxiv.org
• Transformer Architecture: The Positional Encoding - Amirhossein Kazemnejad's Blogkazemnejad.com
• Jane Street Blog - Using group theory to explore the space of positional encodings for attentionblog.janestreet.com
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Rotary Embeddings: A Relative Revolution | EleutherAI Blogblog.eleuther.ai
• 2410.06205arxiv.org
• On N-dimensional Rotary Positional Embeddingsjerryxio.ng
• Relative Positional Encoding - Jake Taejaketae.github.io
• Everything About Transformerskrupadave.com
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub