Everything About Transformers

A visual, intuitive deep dive into transformer architecture: history, design choices, and the why behind attention, masking, and positional encoding.

The research paper "Attention is All You Need" is regarded as one of the most important & groundbreaking publications in the realm of ML. The paper introduces the transformer architecture and the attention mechanism, yet many still struggle to wrap their head around it. When I posted my progress update on my encoder block written in CUDA (Python + Numba), a lot of responses echoed a similar theme: "I want to understand how transformers work from the ground up." This got me thinking. What really helped ME understand the transformer? It was story-telling & illustrations. Every model in the histo

saved by

• Kristen Cheung
• Ishaan Panigrahi

related reading

• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transformers Explained Visually (Part 3): Multi-head Attention, deep dive | Towards Data Sciencetowardsdatascience.com
• Transformers Laid Outgoyalpramod.github.io
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• How LLMs Actually Work | 0xkato0xkato.xyz
• transformer_attention.pdfarxiv.org
• Transformers from Scratche2eml.school
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• Some Intuition on Attention and the Transformereugeneyan.com
• Attention is all you need: Discovering the Transformer paper | Towards Data Sciencetowardsdatascience.com