Do sparse autoencoders find "true features"? — LessWrong

Thanks to Joseph Bloom and James Oldfield for giving feedback on drafts which helped improve the post In this post I'll discuss an apparent limitation of sparse autoencoders (SAEs) in their current formulation as they are applied to discovering the latent features within AI models such as transformer-based LLMs. In brief, I'll cover the following: We intend for SAEs to discover the "true features" (a term I'm borrowing from Anthropic's SAE paper) used by the target model e.g. a transformer-based LLM. There isn't a universally accepted definition of what "true features" are, but for now I'll use the term somewhat loosely to refer to something like: There may be other ways of thinking about features but this should give us enough to work with for our current purposes. Consider a toy setup where one of the hidden layers in the target model has 3 "true features" represented by the following directions in its activation space: Additionally, suppose that feature 1 and feature 2 occur far mor

x Do sparse autoencoders find "true features"? — LessWrong Interpretability (ML & AI) Sparse Autoencoders (SAEs) AI Frontpage 76 Do sparse autoencoders find "true features"? by Demian Till 22nd Feb 2024 13 min read 33 76 Thanks to Joseph Bloom and James Oldfield for giving feedback on drafts which helped improve the post In this post I'll discuss an apparent limitation of sparse autoencoders (SAEs) in their current formulation as they are applied to discovering the latent features within AI models such as transformer-based LLMs. In brief, I'll cover the following: I'll argue that the L1 regula

related reading

• An Intuitive Explanation of Sparse Autoencoders for LLM Interpretability | Adam Karvonenadamkarvonen.github.io
• A gentle introduction to sparse autoencoders — LessWronglesswrong.com
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
• A gentle introduction to sparse autoencodersnickjiang.substack.com
• pdfopenreview.net
• Open Source Replication & Commentary on Anthropic's Dictionary Learning Paper — LessWronglesswrong.com
• Matryoshka Sparse Autoencoders — LessWronglesswrong.com
• Sparse Autoencoders Find Highly Interpretable Features in Language Modelsarxiv.org
• [Interim research report] Taking features out of superposition with sparse autoencoders — LessWronglesswrong.com
• Identifying Functionally Important Features with End-to-End Sparse Dictionary Learning — LessWronglesswrong.com
• Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnettransformer-circuits.pub
• Interpretability with Sparse Autoencoders (Colab exercises) — LessWronglesswrong.com