Understanding the J-Lens
There has been a lot of excitement about Anthropic’s new interpretability technique - the Jacobian lens - lately, and its implications both as a tool for understanding model internals and as a possible bridge to neuroscience and the notion of a ‘global workspace.’ It’s a really interesting tool, and something I want to better understand. Notably, you don’t need very much math background to ‘get it’ - just a little bit of linear algebra (bases and linear transformations) and calculus (partial derivatives, the Jacobian). Let us play and let us explore. Put most simply, the J-lens is a way for us to understand the intermediate layers of a decoder transformer model. These models are composed of dozens of layers (GPT-2’s base model has 12 layers; its largest 1.5B parameter model has 48 layers), each of which is composed of an attention block and a feed-forward block. Each layer reads from the residual stream and writes to it - it’s an updating matrix of vectors, one per token position, whic
There has been a lot of excitement about Anthropic’s new interpretability technique - the Jacobian lens - lately, and its implications both as a tool for understanding model internals and as a possible bridge to neuroscience and the notion of a ‘global workspace.’ It’s a really interesting tool, and something I want to better understand. Notably, you don’t need very much math background to ‘get it’ - just a little bit of linear algebra (bases and linear transformations) and calculus (partial derivatives, the Jacobian). Let us play and let us explore. What is the J-lens? Put most simply,…
saved by
related reading
- A Review of Anthropic's Global Workspace Paper — LessWronglesswrong.com
- Verbalizable Representations Form a Global Workspace in Language Modelstransformer-circuits.pub
- the j-lens: finding an llm's unspoken concepts / chiragctxnn.github.io
- A global workspace in language models \ Anthropicanthropic.com
- interpreting GPT: the logit lens — LessWronglesswrong.com
- The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
- A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
- No Space Like J-Space - by Zvi Mowshowitzthezvi.substack.com
- In-context Learning and Induction Headstransformer-circuits.pub
- The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
- Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
- Transformers from Scratche2eml.school