Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research (GDM Mech Interp Team Progress Update #2) — AI Alignment Forum

Lewis Smith*, Sen Rajamanoharan*, Arthur Conmy, Callum McDougall, Janos Kramar, Tom Lieberum, Rohin Shah, Neel Nanda * = equal contribution The following piece is a list of snippets about research from the GDM mechanistic interpretability team, which we didn’t consider a good fit for turning into a paper, but which we thought the community might benefit from seeing in this less formal form. These are largely things that we found in the process of a project investigating whether sparse autoencoders (SAEs) were useful for downstream tasks, notably out-of-distribution probing. Our core motivation was that we, along with much of the interpretability community, had invested a lot of our energy into Sparse Autoencoder (SAE) research. But SAEs lack a ground truth of the “true” features in language models to compare to, making it pretty unclear how well they work. There is qualitative evidence that SAEs are clearly doing something, far more structure than you would expect by random chance. But

x Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research (GDM Mech Interp Team Progress Update #2) — AI Alignment Forum Interpretability (ML & AI) Sparse Autoencoders (SAEs) AI Curated 2025 Top Fifty: 39 % 58 Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research (GDM Mech Interp Team Progress Update #2) by lewis smith , Senthooran Rajamanoharan , Arthur Conmy , CallumMcDougall , Tom Lieberum , János Kramár , Rohin Shah , Neel Nanda 26th Mar 2025 Linkpost for deepmindsafetyresearch.medium.com 35 min read 15 58 Lewis Smith*, Sen Rajamanoharan*, Arth

related reading

• Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research (GDM Mech Interp Team Progress Update #2) — AI Alignment Forumalignmentforum.org
• Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research (GDM Mech Interp Team Progress Update #2) — LessWronglesswrong.com
• An Intuitive Explanation of Sparse Autoencoders for LLM Interpretability | Adam Karvonenadamkarvonen.github.io
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
• A Pragmatic Vision for Interpretability — AI Alignment Forumalignmentforum.org
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
• pdfopenreview.net
• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• A gentle introduction to sparse autoencoders — LessWronglesswrong.com
• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• A List of 45+ Mech Interp Project Ideas from Apollo Research’s Interpretability Team — AI Alignment Forumalignmentforum.org
• Prism: mapping interpretable concepts and features in a latent space of language | thesephist.comthesephist.com