Chapter 1: Transformer Interpretability - ARENA

This is a demo notebook for TransformerLens, a library I (Neel Nanda) wrote for doing mechanistic interpretability of GPT-2 Style language models. The goal of mechanistic interpretability is to take a trained model and reverse engineer the algorithms the model learned during training from its weights. It is a fact about the world today that we have computer programs that can essentially speak English at a human level (GPT-3, PaLM, etc), yet we have no idea how they work nor how to write one ourselves. This offends me greatly, and I would like to solve this! Mechanistic interpretability is a very young and small field, and there are a lot of open problems - if you would like to help, please try working on one! Check out my list of concrete open problems to figure out where to start. I wrote this library because after I left the Anthropic interpretability team and started doing independent research, I got extremely frustrated by the state of open source tooling. There's a lot of excellen

1️⃣ TransformerLens: Introduction Learning Objectives Load and run a HookedTransformer model Understand the basic architecture of these models Use the model's tokenizer to convert text to tokens, and vice versa Know how to cache activations, and to access activations from the cache Use circuitsvis to visualise attention heads Introduction Note - most of this is written from the POV of Neel Nanda. This is a demo notebook for TransformerLens , a library I ( Neel Nanda ) wrote for doing mechanistic interpretability of GPT-2 Style language models. The goal of mechanistic interpretability is to tak

related reading

• TransformerLens Documentationtransformerlensorg.github.io
• TransformerLens Documentationtransformerlensorg.github.io
• Transformer Circuits Threadtransformer-circuits.pub
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
• Exploratory Analysis Demo - TransformerLens Documentationtransformerlensorg.github.io
• Gears-Level Mental Models of Transformer Interpretability — LessWronglesswrong.com