Chapter 1: Transformer Interpretability - ARENA

Here we introduce a toy 2L attention-only transformer trained specifically for today. Some changes to make them easier to interpret: - It has only attention blocks. - The positional embeddings are only added to the residual stream before calculating each key and query vector in the attention layers as opposed to the token embeddings - i.e. we compute queries as Q = (resid + pos_embed) @ W_Q + b_Q and same for keys, but values as V = resid @ W_V + b_V. This means that the residual stream can't directly encode positional information. - This turns out to make it way easier for induction heads to form, it happens 2-3x times earlier - see the comparison of two training runs here. (The bump in each curve is the formation of induction heads.) - The argument that does this below is positional_embedding_type="shortformer". - It has no MLP layers, no LayerNorms, and no biases. - There are separate embed and unembed matrices (i.e. the weights are not tied). We now define our model with a HookedTr

2️⃣ Finding induction heads Learning Objectives Understand what induction heads are, and the algorithm they are implementing Inspect activation patterns to identify basic attention head patterns, and write your own functions to detect attention heads for you Identify induction heads by looking at the attention patterns produced from a repeating random sequence Introducing Our Toy Attention-Only Model Here we introduce a toy 2L attention-only transformer trained specifically for today. Some changes to make them easier to interpret: - It has only attention blocks. - The positional embeddings are

saved by

• Lydia Nottingham

related reading

• Transformer (deep learning) - Wikipediaen.wikipedia.org
• Induction heads - illustrated — LessWronglesswrong.com
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• In-context Learning and Induction Headstransformer-circuits.pub
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
• Transformer Circuits Threadtransformer-circuits.pub
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Attribution Patching: Activation Patching At Industrial Scale - Neel Nandaneelnanda.io
• Transformers from Scratche2eml.school
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• The Annotated Transformernlp.seas.harvard.edu