Chapter 1: Transformer Interpretability - ARENA

Our first task is to extract hidden state activations from the model. For the Geometry of Truth approach, we extract the last-token activation at each specified layer. For declarative statements like "The city of Paris is in France.", the model's representation of whether the statement is true or false is concentrated at the final token position. Note - the Geometry of Truth paper probes specifically at the end-of-sentence punctuation token (the period / full stop). The datasets are designed so that every statement ends with a period, meaning the last token is always the period. This is important because the model's truth representation builds up over the sentence and is concentrated at the final punctuation mark. A few technical details to keep in mind. We use output_hidden_states=True in the forward pass to get all layer activations. outputs.hidden_states has length num_layers + 1: index 0 is the embedding output, and index i for i >= 1 is the output of layer i-1. We also need to han

1️⃣ Setup & visualizing truth representations Learning Objectives Extract hidden state activations from specified layers and token positions Implement PCA to visualize high-dimensional activations Observe that truth is linearly separable in activation space - even without supervision Understand which layers best represent truth via a layer sweep Extracting activations Our first task is to extract hidden state activations from the model. For the Geometry of Truth approach, we extract the last-token activation at each specified layer. For declarative statements like "The city of Paris is in Fran

related reading

• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• Transformer Circuits Threadtransformer-circuits.pub
• Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
• Neuronpedianeuronpedia.org
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activationstransformer-circuits.pub
• How well do truth probes generalise? — LessWronglesswrong.com
• Interfaces for Explaining Transformer Language Models – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• Language models can explain neurons in language modelsopenaipublic.blob.core.windows.net