Chapter 1: Transformer Interpretability - ARENA

Training SAEs can be very challenging, and new insights are being rapidly discovered. From Joseph Bloom: SAEs are an unsupervised method which attempts to trade off reconstruction accuracy against interpretability, which we achieve by inducing activation sparsity. Since we don’t have good metrics for interpretability / reconstruction quality, it’s hard to know when we are actually optimizing what we care about. On top of this, we’re trying to pick a good point on the pareto frontier between interpretability and reconstruction quality which is a hard thing to assess well. The main objective is to have your SAE learn a population of sparse latents (which are likely to be interpretable) without having some dense latents (latents which activate all the time and are likely uninterpretable) or too many dead latents (latents which never fire). In order to help us train SAEs, we've developed a large number of metrics which can be logged while we're training - we'll be discussing more of these

3️⃣ Training & Evaluating SAEs Learning Objectives Learn how to train SAEs using SAELens Understand how to interpret different metrics during training, and understand when & why SAE training fails to produce interpretable latents Get hands-on experience training SAEs in a variety of context: MLP output of TinyStories-1L, residual stream of Gemma-2-2B, attention output of a 2L model, etc Understand how to evaluate SAEs, and why simple metrics can be deceptive (not implemented yet) Introduction Training SAEs can be very challenging, and new insights are being rapidly discovered. From Joseph Bloo

related reading

• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• An Intuitive Explanation of Sparse Autoencoders for LLM Interpretability | Adam Karvonenadamkarvonen.github.io
• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
• A gentle introduction to sparse autoencoders — LessWronglesswrong.com
• Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research (GDM Mech Interp Team Progress Update #2) — LessWronglesswrong.com
• Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research (GDM Mech Interp Team Progress Update #2) — AI Alignment Forumalignmentforum.org
• Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research (GDM Mech Interp Team Progress Update #2) — AI Alignment Forumalignmentforum.org
• An Intuitive Explanation of Sparse Autoencoders for Mechanistic Interpretability of LLMs — LessWronglesswrong.com
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
• pdfopenreview.net
• A List of 45+ Mech Interp Project Ideas from Apollo Research’s Interpretability Team — AI Alignment Forumalignmentforum.org