Tuned Lens

This package provides a simple interface for training and evaluating tuned lenses. A tuned lens allows us to peek at the iterative computations a transformer uses to compute the next token. A lens into a transformer with n layers allows you to replace the last m layers of the model with an affine transformation (we call these affine translators). Each affine translator is trained to minimize the KL divergence between its prediction and the final output distribution of the original model. This means that after training, the tuned lens allows you to skip over these last few layers and see the best prediction that can be made from the model’s intermediate representations, i.e., the residual stream, at layer n - m. The reason we need to train an affine translator is that the representations may be rotated, shifted, or stretched from layer to layer. This training differentiates this method from simpler approaches that unembed the residual stream of the network directly using the unembedding

Tuned Lens 🔎 ¶ Tools for understanding how transformer predictions are built layer-by-layer. This package provides a simple interface for training and evaluating tuned lenses . A tuned lens allows us to peek at the iterative computations a transformer uses to compute the next token. What is a Lens? ¶ A lens into a transformer with n layers allows you to replace the last m layers of the model with an affine transformation (we call these affine translators). Each affine translator is trained to minimize the KL divergence between its prediction and the final output distribution of the original m

related reading

• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Annotated Transformernlp.seas.harvard.edu
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Exploratory Analysis Demo - TransformerLens Documentationtransformerlensorg.github.io
• Transformers from Scratche2eml.school
• Transformer Circuits Threadtransformer-circuits.pub
• The Annotated Transformernlp.seas.harvard.edu
• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• TransformerLens Documentationtransformerlensorg.github.io
• TransformerLens Documentationtransformerlensorg.github.io
• Transformer (deep learning) - Wikipediaen.wikipedia.org