the logit lens on gpt2 activations - Colab

this notebook demonstrates an observation i've made about gpt2 that helped me understand what it's doing internally. for details, read the blog post NOTE: there's a ton of code in this notebook and then pretty pictures at the end. you can look at the pictures w/o running the notebook. if you run the notebook, you can vary the input text (etc) and make pictures of your own :) in short: inputs/outputs the journey what looks like tokens? what tokens do the intermediate activation look like? interpretation (not too confident) gpt2 "thinks" mostly in the space of its predictions. its many layers are repeatedly revising an initial guess in light of revised-guesses it made about earlier tokens. unlike a transformer encoder (eg BERT, or the first part of universal transformer), gpt2 does not hold onto the initial tokens and slowly process them. it immediately converts them into tentative predictions, then works in prediction space. nostalgebraist, august 30 2020 setup: install, clone repo, etc

Google Colab Sign in

related reading

• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• interpreting GPT: the logit lens — LessWronglesswrong.com
• interpreting GPT: the logit lens — AI Alignment Forumalignmentforum.org
• Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
• Verbalizable Representations Form a Global Workspace in Language Modelstransformer-circuits.pub
• What Is ChatGPT Doing … and Why Does It Work?-Stephen Wolfram Writingswritings.stephenwolfram.com
• LLM Visualizationbbycroft.net
• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• Neuronpedianeuronpedia.org
• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activationstransformer-circuits.pub
• Transformer Circuits Threadtransformer-circuits.pub
• GPT2-SMALL · 7-RES-JB · 10199 ｜ Neuronpedianeuronpedia.org