An Introduction to Exemplar Partitioning for Mechanistic Interpretability — LessWrong

Most of what we currently call "feature discovery" in language models is wrapped up in dictionary-learning methods like sparse autoencoders (SAEs) – which work, and which have been scaled to millions of features on frontier-scale models, but which bundle two distinct commitments into a single training objective: a reconstruction loss and a sparsity loss over a fixed size dictionary. Those commitments make sense if your goal is reconstructive decomposition – if you want to take an activation and rebuild it from a sparse code. They make less obvious sense if your aim is to find interpretable structure (directions? features?) in activation space, to retrieve representative examples, identify causal interventions, or measure how representations change across layers and inputs. And it turns out a lot of that doesn't really need the full SAE machinery. An Exemplar Partitioning dictionary built from Gemma-2-2B L12 activations at p2 (K = 5,129). Left: eight sample regions, each shown with its

x An Introduction to Exemplar Partitioning for Mechanistic Interpretability — LessWrong Interpretability (ML & AI) Sparse Autoencoders (SAEs) AI Frontpage 69 An Introduction to Exemplar Partitioning for Mechanistic Interpretability by Jessica Rumbelow 16th May 2026 Linkpost for www.leap-labs.com 13 min read 7 69 Most of what we currently call "feature discovery" in language models is wrapped up in dictionary-learning methods like sparse autoencoders (SAEs) – which work, and which have been scaled to millions of features on frontier-scale models, but which bundle two distinct commitments into a

saved by

• Yixiong Hao
• Dillon Nguyen

related reading

• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
• Sparse Autoencoders Find Highly Interpretable Features in Language Modelsarxiv.org
• An Intuitive Explanation of Sparse Autoencoders for LLM Interpretability | Adam Karvonenadamkarvonen.github.io
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
• Identifying Functionally Important Features with End-to-End Sparse Dictionary Learning — LessWronglesswrong.com
• Prism: mapping interpretable concepts and features in a latent space of language | thesephist.comthesephist.com
• Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnettransformer-circuits.pub
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
• pdfopenreview.net
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
• A List of 45+ Mech Interp Project Ideas from Apollo Research’s Interpretability Team — AI Alignment Forumalignmentforum.org
• The Building Blocks of Interpretabilitydistill.pub