Prism: mapping interpretable concepts and features in a latent space of language | thesephist.com
Foundation models gesture at a way of interacting with information that’s at once more natural and powerful than “classic” knowledge tools. But to build the kind of rich, directly interactive information interfaces I imagine, current foundation models and embeddings are far too opaque to humans. Models and their raw outputs resist understanding. Even when we go to great lengths to try to surface what the model is “thinking” through black-box methods like prompting or dimensionality reduction approaches like PCA and UMAP, we can’t deliver interaction experiences that feel as direct and predictable, as “extension-of-self”, as multi-touch. Solving this understandability gap is fundamental to unlocking richer interfaces to modern AI systems and the information they model. Good information interfaces open up not just new utility, but new understanding to users about their world, and without closing the understandability gap between frontier AI systems and humans, we can’t build good informa
Foundation models gesture at a way of interacting with information that’s at once more natural and powerful than “classic” knowledge tools. But to build the kind of rich, directly interactive information interfaces I imagine , current foundation models and embeddings are far too opaque to humans. Models and their raw outputs resist understanding. Even when we go to great lengths to try to surface what the model is “thinking” through black-box methods like prompting or dimensionality reduction approaches like PCA and UMAP, we can’t deliver interaction experiences that feel as direct and predict
saved by
related reading
- Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
- Verbalizable Representations Form a Global Workspace in Language Modelstransformer-circuits.pub
- Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
- What makes a good human interface? | thesephist.comthesephist.com
- Sparse Autoencoders Find Highly Interpretable Features in Language Modelsarxiv.org
- The Building Blocks of Interpretabilitydistill.pub
- An Intuitive Explanation of Sparse Autoencoders for LLM Interpretability | Adam Karvonenadamkarvonen.github.io
- An Introduction to Exemplar Partitioning for Mechanistic Interpretability — LessWronglesswrong.com
- Imagining better interfaces to language models | thesephist.comthesephist.com
- A gentle introduction to sparse autoencoders — LessWronglesswrong.com
- [2309.08600] Sparse Autoencoders Find Highly Interpretable Features in Language Modelsarxiv.org
- Language Model Circuits Are Sparse in the Neuron Basis | Transluce AItransluce.org