Prism: mapping interpretable concepts and features in a latent space of language | thesephist.com

Foundation models gesture at a way of interacting with information that’s at once more natural and powerful than “classic” knowledge tools. But to build the kind of rich, directly interactive information interfaces I imagine, current foundation models and embeddings are far too opaque to humans. Models and their raw outputs resist understanding. Even when we go to great lengths to try to surface what the model is “thinking” through black-box methods like prompting or dimensionality reduction approaches like PCA and UMAP, we can’t deliver interaction experiences that feel as direct and predictable, as “extension-of-self”, as multi-touch. Solving this understandability gap is fundamental to unlocking richer interfaces to modern AI systems and the information they model. Good information interfaces open up not just new utility, but new understanding to users about their world, and without closing the understandability gap between frontier AI systems and humans, we can’t build good informa

Foundation models gesture at a way of interacting with information that’s at once more natural and powerful than “classic” knowledge tools. But to build the kind of rich, directly interactive information interfaces I imagine , current foundation models and embeddings are far too opaque to humans. Models and their raw outputs resist understanding. Even when we go to great lengths to try to surface what the model is “thinking” through black-box methods like prompting or dimensionality reduction approaches like PCA and UMAP, we can’t deliver interaction experiences that feel as direct and predict

saved by

• Winnie Xu
• Ratan Kaliani
• Hamidah Oderinwale
• KL Yap
• Wilbur Zhang

related reading

• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
• Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnettransformer-circuits.pub
• Transformer Circuits Threadtransformer-circuits.pub
• Sparse Autoencoders Find Highly Interpretable Features in Language Modelsarxiv.org
• Mapping the Mind of a Large Language Model \ Anthropicanthropic.com
• An Intuitive Explanation of Sparse Autoencoders for LLM Interpretability | Adam Karvonenadamkarvonen.github.io
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
• The Building Blocks of Interpretabilitydistill.pub
• pdfopenreview.net
• Painting With Concepts Using Diffusion Model Latentsgoodfire.ai
• A gentle introduction to sparse autoencoders — LessWronglesswrong.com
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub