You Could’ve Invented Transformers, by Gwern · Gwern.net

There are many Transformer⁠ tutorials (⁠eg⁠), but they generally focus on presenting a relatively contemporary implementation. This leaves the reader a bit mystified: this complex assemblage of MLPs and self-attention layers and normalization appears to have ‘dropped out of the sky’. How could Noam Shazeer and the other authors have invented the Transformer—divine benevolence (as all else)? Why does this complicated Transformer head block work so well for sequence prediction? Which parts are required, and which are there for optimization purposes? The Transformer was discovered by a huge amount of trial-and-error⁠ (much like resnets⁠), so following the historical sequence is neither possible nor enlightening. However, at this point, I believe we now understand Transformers & variations well enough that we can invent an imaginary history of Transformers for pedagogical purposes, in the vein of the “You Could Have Invented X” snowclone⁠ (eg. monads⁠, parser combinators, zippers, Fenwick

--- title: 'You Could’ve Invented Transformers' author: Gwern description: "Proposal for a ‘You Could Have Invented Transformers’ tutorial; someone should write a series showing a logical recreation of Transformers from primitive n -gram language models to full-strength Transformers." created: 2025-05-25 modified: 2025-05-25 status: finished importance: N/A confidence: log css-extension: dropcaps-de-zs toc-not backlink: True ... There are many [Transformer](https://arxiv.org/abs/1706.03762#google "‘Attention Is All You Need’, Vaswani et al 2017") tutorials ([eg](/gpt-2#transformer-tutorial)),

related reading

• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• The Annotated Transformernlp.seas.harvard.edu
• Transformers from Scratche2eml.school
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• Everything About Transformerskrupadave.com
• Transformers from scratch | peterbloem.nlpeterbloem.nl
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• transformer_attention.pdfarxiv.org
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• Transformer Circuits Threadtransformer-circuits.pub
• Transformer (deep learning) - Wikipediaen.wikipedia.org