Inside the Transformer: The Life of a Token - Aleksa Gordić

A deep dive into a modern dense transformer: YaRN, hybrid attention, soft capping, QK normalization, FLOPs/token, cluster sizing, and more.

In this post, I'll do a deep dive into the internals of a modern dense transformer [1] . I'll focus exclusively on the forward pass on a single GPU, as if we were about to perform a training step, while ignoring the backward pass and distributed systems details (in practice, large Transformers are sharded across multiple devices during both training and inference). As a running example, I'll use the exact architecture of Rnj 1.5 - a model I worked on with my team at Ashish Vaswani's AI Lab (Essential AI Labs). 💡 The team behind Rnj-1.5: Rnj 1.5 could not have happened without an amazing group

saved by

• Mazin K

related reading

• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Everything About Transformerskrupadave.com
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• The Annotated Transformernlp.seas.harvard.edu
• Transformers from Scratche2eml.school
• All the Transformer Math You Need to Know | How To Scale Your Modeljax-ml.github.io
• transformer_attention.pdfarxiv.org
• Transformers from scratch | peterbloem.nlpeterbloem.nl
• The Annotated Transformernlp.seas.harvard.edu
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io