Transformers Represent Belief State Geometry in their Residual Stream — LessWrong
lesswrong.com · 14,324 words · saved by 1 readers
Produced while being an affiliate at PIBBSS[1]. The work was done initially with funding from a Lightspeed Grant, and then continued while at PIBBSS.…
x Transformers Represent Belief State Geometry in their Residual Stream — LessWrong Best of LessWrong 2024 MATS Program PIBBSS Interpretability (ML & AI) Transformers AI Curated 442 Transformers Represent Belief State Geometry in their Residual Stream by Adam Shai 16th Apr 2024 AI Alignment Forum 14 min read 103 442 Ω 145 Produced while being an affiliate at PIBBSS [1] . The work was done initially with funding from a Lightspeed Grant, and then continued while at PIBBSS. Work done in collaboration with @Paul Riechers , @Lucas Teixeira , @Alexander Gietelink Oldenziel , and Sarah Marzen . Paul
related reading
- A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
- Simplex Progress Report - July 2025 — LessWronglesswrong.com
- On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
- Simulators — LessWronglesswrong.com
- Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
- Gears-Level Mental Models of Transformer Interpretability — LessWronglesswrong.com
- In-context Learning and Induction Headstransformer-circuits.pub
- Thinking like Transformersrush.github.io
- interpreting GPT: the logit lens — LessWronglesswrong.com
- A Mechanistic Explanation of Prompt Injection (and why you should study roles) — LessWronglesswrong.com
- The Topological Trouble With Transformersarxiv.org
- A Conceptual Guide to Transformers: Part Ibenlevinstein.substack.com