My Attempt to Explain the Jacobian Lens: A Single Jacobian Already Works — Sida Liu
Anthropic’s Jacobian Lens (J-Lens) reads intermediate activations as vocabulary scores. Its readouts can reveal concepts involved in intermediate computation, including information the model does not immediately output. Panel A below, from 2.1 The Jacobian Lens, illustrates the starting point: measure how a small change at an intermediate activation affects final-layer activations at the current and future token positions. For source position 𝑡 at layer ℓ and target position 𝑡 ′ at final layer 𝐿 , that relationship is a Jacobian: J-Lens aggregates these Jacobians across token positions and dataset examples to obtain one matrix 𝐽 ℓ per layer. To inspect an activation: The highest-scoring tokens suggest concepts represented in the intermediate activation. But why it works? We need to interpret this new interpretability tool. A Mathematical Theory of the Jacobian Lens explains what survives Jacobian averaging: responses that are strong and consistently oriented across token l