Activation space interpretability may be doomed — LessWrong

TL;DR: There may be a fundamental problem with interpretability work that attempts to understand neural networks by decomposing their individual acti…

x Activation space interpretability may be doomed — LessWrong Apollo Research (org) Interpretability (ML & AI) AI Frontpage 2025 Top Fifty: 12 % 154 Activation space interpretability may be doomed by bilalchughtai , Lucius Bushnaq 8th Jan 2025 AI Alignment Forum 10 min read 34 154 Ω 58 TL;DR: There may be a fundamental problem with interpretability work that attempts to understand neural networks by decomposing their individual activation spaces in isolation: It seems likely to find features of the activations - features that help explain the statistical structure of activation spaces, rather

related reading

• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
• Toy Models of Superpositiontransformer-circuits.pub
• The Building Blocks of Interpretabilitydistill.pub
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
• An Intuitive Explanation of Sparse Autoencoders for LLM Interpretability | Adam Karvonenadamkarvonen.github.io
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
• Transformer Circuits Threadtransformer-circuits.pub
• Sparse Autoencoders Find Highly Interpretable Features in Language Modelsarxiv.org
• SAE feature geometry is outside the superposition hypothesis — AI Alignment Forumalignmentforum.org
• A List of 45+ Mech Interp Project Ideas from Apollo Research’s Interpretability Team — AI Alignment Forumalignmentforum.org
• Lucius Bushnaq's Shortform — LessWronglesswrong.com
• SAE feature geometry is outside the superposition hypothesis — LessWronglesswrong.com