A Mathematical Framework for Transformer Circuits
Transformer language models are an emerging technology that is gaining increasingly broad real-world use, for example in systems like GPT-3 , LaMDA , Codex , Meena , Gopher , and similar models. However, as these models scale, their open-endedness and high capacity creates an increasing scope for unexpected and sometimes harmful behaviors. Even years after a large model is trained, both creators and users routinely discover model capabilities – including problematic behaviors – they were previously unaware of.
A Mathematical Framework for Transformer Circuits Transformer Circuits Thread A Mathematical Framework for Transformer Circuits Authors Nelson Elhage ∗† , Neel Nanda ∗ , Catherine Olsson ∗ , Tom Henighan † , Nicholas Joseph † , Ben Mann † , Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds , Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, Chris Olah ‡ Affiliation Anthropic Published Dec 22, 2021 * Core Research Contributor; † Core Infrastruc
Explore this link on the map →saved by
- Winnie Xu
- Alex K. Chen
- Klaudia U.
- c l
- Pier-Luc Nadeau-Labadie
- Michelle Pan
- Claire Wang
- Asma Lamgh
- Alex Shaw
- Shubham Chandel
- Ivy Zhang
- James Webb
related reading
- The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
- Transformer Circuits Threadtransformer-circuits.pub
- Transformers from Scratche2eml.school
- Induction heads - illustrated — LessWronglesswrong.com
- The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
- Transformer (deep learning) - Wikipediaen.wikipedia.org
- Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
- transformer_attention.pdfarxiv.org
- Transformer (deep learning) - Wikipediaen.wikipedia.org
- A Conceptual Guide to Transformers: Part Ibenlevinstein.substack.com
- Everything About Transformerskrupadave.com
- The Annotated Transformernlp.seas.harvard.edu