Positional Encodings and Group Theory | 3Blue1Brown and Alok Puranik - YouTube

Enjoy the videos and music you love, upload original content, and share it all with friends, family, and the world on YouTube.

Grant Sanderson of 3Blue1Brown and Alok Puranik, a researcher at Jane Street, work through Alok’s latest blog post on positional encodings in transformers. Starting from a few natural assumptions about attention, they use group and representation theory to explore what forms a positional encoding can take. Along the way, they connect the theory to familiar methods like RoPE (Rotary Positional Embedding) and ALiBi (Attention with Linear Biases), and investigate whether it reveals anything new. Read Alok's blog post here:…

saved by

• Ishaan Panigrahi

related reading

• Next-Latent Prediction Transformers Learn World Models feat. Jayden Teoh from MITyoutube.com
• Jane Street Blog - Using group theory to explore the space of positional encodings for attentionblog.janestreet.com
• RLMs are Compositional Generalizers with Alex Zhang from MITyoutube.com
• Transformers Laid Outgoyalpramod.github.io
• Stanford CS25: Transformers United V6 I From Representation Learning to World Modelingyoutube.com
• How trillion-parameter AI models work and how to serve themyoutube.com
• Your Transformer is Secretly an EOT Solver | Elements of a Vector Spaceelonlit.com
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Master Positional Encoding: Part I | Towards Data Sciencetowardsdatascience.com
• You could have designed state of the art positional encodinghuggingface.co
• The absolute basics of representation theory of finite groups — LessWronglesswrong.com
• On N-dimensional Rotary Positional Embeddingsjerryxio.ng