Toy Models of Superposition

It would be very convenient if the individual neurons of artificial neural networks corresponded to cleanly interpretable features of the input. For example, in an “ideal” ImageNet classifier, each neuron would fire only in the presence of a specific visual feature, such as the color red, a left-facing curve, or a dog snout. Empirically, in models we have studied, some of the neurons do cleanly map to features. But it isn't always the case that features correspond so cleanly to neurons, especially in large language models where it actually seems rare for neurons to correspond to clean features. This brings up many questions. Why is it that neurons sometimes align with features and sometimes don't? Why do some models and tasks have many of these clean neurons, while they're vanishingly rare in others? In this paper, we use toy models — small ReLU networks trained on synthetic data with sparse input features — to investigate how and when models represent more features than they have dime

Toy Models of Superposition Transformer Circuits Thread Toy Models of Superposition Authors Nelson Elhage ∗ , Tristan Hume ∗ , Catherine Olsson ∗ , Nicholas Schiefer ∗ , Tom Henighan , Shauna Kravec, Zac Hatfield-Dodds , Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg ∗ , Christopher Olah ‡ Affiliations Anthropic, Harvard Published Sept 14, 2022 * Core Research Contributor; ‡ Correspondence to colah@anthropic.com ; Author contributions statement below . It would be very convenient if the individual neurons of artificial neural

saved by

• Winnie Xu
• Katherine Huang
• Alex K. Chen
• Tazik Sh
• Claire Wang
• Alex Shaw
• Lowell Dennings
• Ulisse mini
• Hangyul Lyna Kim
• Braeden Hall
• 3943
• Lydia Nottingham

related reading

• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
• What Would Non-Linear Features Actually Look Like? — Liv Gortonlivgorton.com
• Softmax Linear Unitstransformer-circuits.pub
• how neural networks think at scalemarmik.xyz
• Interpretability Dreamstransformer-circuits.pub
• The Building Blocks of Interpretabilitydistill.pub
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
• Sparse Autoencoders Find Highly Interpretable Features in Language Modelsarxiv.org
• SAE feature geometry is outside the superposition hypothesis — LessWronglesswrong.com
• Feature Visualizationdistill.pub
• SAE feature geometry is outside the superposition hypothesis — AI Alignment Forumalignmentforum.org
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org