Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper? | Hacker News

First, we take a sequence of words and represent it as a grid of numbers: each column of the grid is a separate word, and each row of the grid is a measurement of some property of that word. Words with similar meanings are likely to have similar numerical values on a row-by-row basis. (During the training process, we create a dictionary of all possible words, with a column of numbers for each of those words. More on this later!) This grid is called the "context". Typical systems will have a context that spans several thousand columns and several thousand rows. Right now, context length (column count) is rapidly expanding (1k to 2k to 8k to 32k to 100k+!!) while the dimensionality of each word in the dictionary (row count) is pretty static at around 4k to 8k... Anyhow, the Transformer architecture takes that grid and passes it through a multi-layer transformation algorithm. The functionality of each layer is identical: receive the grid of numbers as input, then perform a mathematical tr

Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper? | Hacker News Hacker News new | past | comments | ask | show | jobs | submit login Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper? 644 points by Sai_ on May 17, 2023 | hide | past | favorite | 228 comments I have zero AI/ML knowledge but Steve Yegge on Medium thinks that the team behind Transformers deserves a Nobel. Makes me want to better understand this tech. Edit: thank you for some amazing top level responses and links to valuable content on this subject. benjismith on May

related reading

• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transformers from Scratche2eml.school
• Everything About Transformerskrupadave.com
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transformers from scratch | peterbloem.nlpeterbloem.nl
• Some Intuition on Attention and the Transformereugeneyan.com
• A Conceptual Guide to Transformers: Part Ibenlevinstein.substack.com
• 9 Transformers – 6.390 - Intro to Machine Learningintroml.mit.edu
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• transformer_attention.pdfarxiv.org
• Transformer (deep learning) - Wikipediaen.wikipedia.org