flâneur

Understanding the J-Lens

emma-x1.github.io · 2,045 words · saved by 1 readers

There has been a lot of excitement about Anthropic’s new interpretability technique - the Jacobian lens - lately, and its implications both as a tool for understanding model internals and as a possible bridge to neuroscience and the notion of a ‘global workspace.’ It’s a really interesting tool, and something I want to better understand. Notably, you don’t need very much math background to ‘get it’ - just a little bit of linear algebra (bases and linear transformations) and calculus (partial derivatives, the Jacobian). Let us play and let us explore. Put most simply, the J-lens is a way for us to understand the intermediate layers of a decoder transformer model. These models are composed of dozens of layers (GPT-2’s base model has 12 layers; its largest 1.5B parameter model has 48 layers), each of which is composed of an attention block and a feed-forward block. Each layer reads from the residual stream and writes to it - it’s an updating matrix of vectors, one per token position, whic

There has been a lot of excitement about Anthropic’s new interpretability technique - the Jacobian lens - lately, and its implications both as a tool for understanding model internals and as a possible bridge to neuroscience and the notion of a ‘global workspace.’ It’s a really interesting tool, and something I want to better understand. Notably, you don’t need very much math background to ‘get it’ - just a little bit of linear algebra (bases and linear transformations) and calculus (partial derivatives, the Jacobian). Let us play and let us explore. What is the J-lens? Put most simply,…

saved by

related reading