Thinking like Transformers

Transformer models are foundational to AI systems. There are now countless explanations of “how transformers work?” in the sense of the architecture diagram at the heart of transformers. However this diagram does not provide any intuition into the computational model of this framework. As researchers become interested in how Transformers work, gaining intuition into their mechanisms becomes increasingly useful. Thinking like Transformers proposes a computational framework for Transformer-like calculations. The framework uses discrete computation to simulate Transformer computations. The resulting language RASP is a programming language where every program compiles down to a specific Transformer. In this blog post, I reimplemented a variant of RASP in Python (RASPy). The language is roughly compatible with the original version, but with some syntactic changes that I thought were fun. With this language, the author of the work Gail Weiss, provided a challenging set of puzzles to walk thr

Thinking like Transformer Thinking Like Transformers Paper by Gail Weiss, Yoav Goldberg, Eran Yahav Blog by Sasha Rush and Gail Weiss Library and Interactive Notebook: srush/raspy Transformer models are foundational to AI systems. There are now countless explanations of “how transformers work?” in the sense of the architecture diagram at the heart of transformers. svg However this diagram does not provide any intuition into the computational model of this framework. As researchers become interested in how Transformers work, gaining intuition into their mechanisms becomes increasingly useful. T

related reading

• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transformers from Scratche2eml.school
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• The Annotated Transformernlp.seas.harvard.edu
• Transformer Circuits Threadtransformer-circuits.pub
• transformer_attention.pdfarxiv.org
• Everything About Transformerskrupadave.com
• Thinking Like Transformersarxiv.org
• Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper? | Hacker Newsnews.ycombinator.com
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transformers from Scratchbrandonrohrer.com