SAE feature geometry is outside the superposition hypothesis — AI Alignment Forum

Summary: Superposition-based interpretations of neural network activation spaces are incomplete. The specific locations of feature vectors contain crucial structural information beyond superposition, as seen in circular arrangements of day-of-the-week features and in the rich structures of feature UMAPs. We don’t currently have good concepts for talking about this structure in feature geometry, but it is likely very important for model computation. An eventual understanding of feature geometry might look like a hodgepodge of case-specific explanations, or supplementing superposition with additional concepts, or plausibly an entirely new theory that supersedes superposition. To develop this understanding, it may be valuable to study toy models in depth and do theoretical or conceptual work in addition to studying frontier models. Epistemic status: Decently confident that the ideas here are directionally correct. I’ve been thinking these thoughts for a while, and recently got round to wr

x SAE feature geometry is outside the superposition hypothesis — AI Alignment Forum Interpretability (ML & AI) Apollo Research (org) AI Frontpage 101 SAE feature geometry is outside the superposition hypothesis by jake_mendel 24th Jun 2024 14 min read 18 101 Written at Apollo Research Summary: Superposition-based interpretations of neural network activation spaces are incomplete. The specific locations of feature vectors contain crucial structural information beyond superposition, as seen in circular arrangements of day-of-the-week features and in the rich structures of feature UMAPs. We don’t

related reading

• SAE feature geometry is outside the superposition hypothesis — LessWronglesswrong.com
• Toy Models of Superpositiontransformer-circuits.pub
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
• Activation space interpretability may be doomed — LessWronglesswrong.com
• Interpretability Dreamstransformer-circuits.pub
• A gentle introduction to sparse autoencoders — LessWronglesswrong.com
• pdfopenreview.net
• An Intuitive Explanation of Sparse Autoencoders for LLM Interpretability | Adam Karvonenadamkarvonen.github.io
• What Would Non-Linear Features Actually Look Like? — Liv Gortonlivgorton.com
• Lucius Bushnaq's Shortform — LessWronglesswrong.com
• Circuits Updates — May 2023transformer-circuits.pub
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org