The Illustrated GPT-2 (Visualizing Transformer Language Models) – Jay Alammar – Visualizing machine learning one concept at a time.

Discussions: Hacker News (64 points, 3 comments), Reddit r/MachineLearning (219 points, 18 comments) Translations: Simplified Chinese, French, Korean, Russian, Turkish This year, we saw a dazzling application of machine learning. The OpenAI GPT-2 exhibited impressive ability of writing coherent and passionate essays that exceed what we anticipated current language models are able to produce. The GPT-2 wasn’t a particularly novel architecture – it’s architecture is very similar to the decoder-only transformer. The GPT2 was, however, a very large, transformer-based language model trained on a massive dataset. In this post, we’ll look at the architecture that enabled the model to produce its results. We will go into the depths of its self-attention layer. And then we’ll look at applications for the decoder-only transformer beyond language modeling. My goal here is to also supplement my earlier post, The Illustrated Transformer, with more visuals explaining the inner-workings of transformers, and how they’ve evolved since the original paper. My hope is that this visual language will hopefully make it easier to explain later Transformer-based models as their inner-workings continue to evolve.

The Illustrated GPT-2 (Visualizing Transformer Language Models) Discussions: Hacker News (64 points, 3 comments) , Reddit r/MachineLearning (219 points, 18 comments) Translations: Simplified Chinese , French , Korean , Russian , Turkish This year, we saw a dazzling application of machine learning. The OpenAI GPT-2 exhibited impressive ability of writing coherent and passionate essays that exceed what we anticipated current language models are able to produce. The GPT-2 wasn’t a particularly novel architecture – it’s architecture is very similar to the decoder-only transformer. The GPT2 was, ho

related reading

• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• interpreting GPT: the logit lens — LessWronglesswrong.com
• transformer_attention.pdfarxiv.org
• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Everything About Transformerskrupadave.com
• Interfaces for Explaining Transformer Language Models – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• GPT in 60 Lines of NumPy | Jay Modyjaykmody.com
• The Illustrated Word2vec – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io