Chapter 1: Transformer Interpretability - ARENA

Please send any problems / bugs on the #errata channel in the Slack group, and ask any questions on the dedicated channels for this chapter of material. If you want to change to dark mode, you can do this by clicking the three horizontal lines in the top-right, then navigating to Settings → Theme. Links to all other chapters: (0) Fundamentals, (1) Transformer Interpretability, (2) RL. Linear probes let you ask yes-or-no questions about a model's activations, and SAEs give you an unsupervised decomposition into interpretable features. Both are useful, but they share a limitation: you have to decide what you're looking for before you start looking. What if you could just ask a model's activations an open-ended question in plain English? "What concept is this layer encoding?" or "Is this model planning to lie?" That's the idea behind Activation Oracles: LLMs that have been trained to take another model's internal activations as input and answer arbitrary questions about them. The oracle r

[1.3.4] Activation Oracles Colab: exercises | solutions Please send any problems / bugs on the #errata channel in the Slack group , and ask any questions on the dedicated channels for this chapter of material. If you want to change to dark mode, you can do this by clicking the three horizontal lines in the top-right, then navigating to Settings → Theme. Links to all other chapters: (0) Fundamentals , (1) Transformer Interpretability , (2) RL . Introduction Linear probes let you ask yes-or-no questions about a model's activations, and SAEs give you an unsupervised decomposition into interpretab

related reading

• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Transformer Circuits Threadtransformer-circuits.pub
• Chapter 4: Alignment Science - ARENAlearn.arena.education
• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activationstransformer-circuits.pub
• ARENA - AI Safety Curriculumlearn.arena.education