Transformers from Scratch

I procrastinated a deep dive into transformers for a few years. Finally the discomfort of not knowing what makes them tick grew too great for me. Here is that dive. Transformers were introduced in this 2017 paper as a tool for sequence transduction—converting one sequence of symbols to another. The most popular examples of this are translation, as in English to German. It has also been modified to perform sequence completion—given a starting prompt, carry on in the same vein and style. They have quickly become an indispensible tool for research and product development in natural language processing. Before we start, just a heads-up. We're going to be talking a lot about matrix multiplications and touching on backpropagation (the algorithm for training the model), but you don't need to know any of it beforehand. We'll add the concepts we need one at a time, with explanation. This isn't a short journey, but I hope you'll be glad you came. In the beginning were the words. So very many wor

I procrastinated a deep dive into transformers for a few years. Finally the discomfort of not knowing what makes them tick grew too great for me. Here is that dive. Transformers were introduced in this 2017 paper as a tool for sequence transduction-converting one sequence of symbols to another. The most popular examples of this are translation, as in English to German. It has also been modified to perform sequence completion-given a starting prompt, carry on in the same vein and style. They have quickly become an indispensible tool for research and product development in natural language proce

saved by

• Amir
• Chloe Chia
• Claire Wang
• Ratan Kaliani
• Jason Ernst
• Asma Lamgh
• Fobi Boateng
• Chloe Yan
• Saeejith Nair
• 2982
• Laurence Liang
• Dharmik Parmar

related reading

• Transformers from Scratchbrandonrohrer.com
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• transformer_attention.pdfarxiv.org
• A Conceptual Guide to Transformers: Part Ibenlevinstein.substack.com
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• The Annotated Transformernlp.seas.harvard.edu
• 1706.03762arxiv.org
• Everything About Transformerskrupadave.com
• Transformers from scratch | peterbloem.nlpeterbloem.nl
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• Attention is all you need: Discovering the Transformer paper | Towards Data Sciencetowardsdatascience.com