TransformerLens Documentation

This is a library for doing mechanistic interpretability of GPT-2 Style language models. The goal of mechanistic interpretability is to take a trained model and reverse engineer the algorithms the model learned during training from its weights. It is a fact about the world today that we have computer programs that can essentially speak English at a human level (GPT-3, PaLM, etc), yet we have no idea how they work nor how to write one ourselves. This offends me greatly, and I would like to solve this!

TransformerLens ¶ (Formerly known as EasyTransformer) A Library for Mechanistic Interpretability of Generative Language Models ¶ This is a library for doing mechanistic interpretability of GPT-2 Style language models. The goal of mechanistic interpretability is to take a trained model and reverse engineer the algorithms the model learned during training from its weights. It is a fact about the world today that we have computer programs that can essentially speak English at a human level (GPT-3, PaLM, etc), yet we have no idea how they work nor how to write one ourselves. This offends me greatl

related reading

• TransformerLens Documentationtransformerlensorg.github.io
• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Transformer Circuits Threadtransformer-circuits.pub
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• In-context Learning and Induction Headstransformer-circuits.pub
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
• Exploratory Analysis Demo - TransformerLens Documentationtransformerlensorg.github.io
• interpreting GPT: the logit lens — LessWronglesswrong.com
• Thinking like Transformersrush.github.io
• A Comprehensive Mechanistic Interpretability Explainer & Glossary — Neel Nandaneelnanda.io
• Softmax Linear Unitstransformer-circuits.pub