Chapter 1: Transformer Interpretability - ARENA

We're going to focus GPT-2 style transformers. Key feature: They generate text! You feed in language, and the model generates a probability distribution over tokens. And you can repeatedly sample from this to generate text! (To explain this in more detail - you feed in a sequence of length 𝑁 N, then sample from the probability distribution over the 𝑁 + 1 N+1-th word, use this to construct a new sequence of length 𝑁 + 1 N+1, then feed this new sequence into the model to get a probability distribution over the 𝑁 + 2 N+2-th word, and so on.) You give it a bunch of text, and train it to predict the next token. Importantly, if you give a model 100 tokens in a sequence, it predicts the next token for each prefix, i.e. it produces 100 logit vectors (= probability distributions) over the set of all words in our vocabulary, with the i-th logit vector representing the probability distribution over the token following the i-th token in the sequence. This is a key part of what allows trans

1️⃣ Understanding Inputs & Outputs of a Transformer Learning Objectives Understand what a transformer is used for Understand causal attention, and what a transformer's output represents - algebraic operations on tensors Learn what tokenization is, and how models do it Understand what logits are, and how to use them to derive a probability distribution over the vocabulary What is the point of a transformer? Transformers exist to model text! We're going to focus GPT-2 style transformers. Key feature: They generate text! You feed in language, and the model generates a probability distribution ove

related reading

• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• Transformers from Scratche2eml.school
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• The Annotated Transformernlp.seas.harvard.edu
• A Conceptual Guide to Transformers: Part Ibenlevinstein.substack.com
• interpreting GPT: the logit lens — AI Alignment Forumalignmentforum.org
• Transformer Circuits Threadtransformer-circuits.pub
• GPT in 60 Lines of NumPy | Jay Modyjaykmody.com
• How to generate text: using different decoding methods for language generation with Transformershuggingface.co
• The Annotated Transformernlp.seas.harvard.edu
• interpreting GPT: the logit lens — LessWronglesswrong.com