Open Source Automated Interpretability for Sparse Autoencoder Features | EleutherAI Blog

Sparse autoencoders recover a diversity of interpretable, monosemantic features, but present an intractable problem of scale to human labelers. We investigate different techniques for generating and scoring arbitrary text explanations of SAE features, and release a open source library to allow people to do research on auto-interpreted features. Open source models generate and evaluate text explanations of SAE features reasonably well, albeit somewhat worse than closed models like Claude 3.5 Sonnet. Explanations found by LLMs are similar to explanations found by humans. Automatically interpreting 1.5M features of GPT-2 with the current pipeline would cost $1300 in API calls to Llama 3.1 or $8500 with Claude 3.5 Sonnet. Prior methods cost ~$200k. Code can be found at https://github.com/EleutherAI/sae-auto-interp. We built a small dashboard to explore explanations and their scores: https://cadentj.github.io/demo/ Sparse autoencoders decompose activations into a sum of sparse feature direc

Table of Contents Background Key Findings Generating Explanations Scoring explanations Results Explainers How does the explainer model size affect explanation quality? Providing more information to the explainer Giving the explainer different samples of top activating examples Visualizing activation distributions Scorers How do methods correlate with simulation? How does scorer model size affect scores? How much more scalable is detection/fuzzing? Filtering with known heuristics Positional Features Unigram features Sparse Feature Circuits Future Directions Appendix Background # Sparse autoenco

related reading

• An Intuitive Explanation of Sparse Autoencoders for LLM Interpretability | Adam Karvonenadamkarvonen.github.io
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
• Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnettransformer-circuits.pub
• Sparse Autoencoders Find Highly Interpretable Features in Language Modelsarxiv.org
• A gentle introduction to sparse autoencoders — LessWronglesswrong.com
• Self-explaining SAE features — AI Alignment Forumalignmentforum.org
• pdfopenreview.net
• A gentle introduction to sparse autoencodersnickjiang.substack.com
• An Intuitive Explanation of Sparse Autoencoders for Mechanistic Interpretability of LLMs — LessWronglesswrong.com
• Prism: mapping interpretable concepts and features in a latent space of language | thesephist.comthesephist.com
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
• Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnettransformer-circuits.pub