Looped Transformers as Programmable Computers
Abstract:We present a framework for using transformer networks as universal computers by programming them with specific weights and placing them in a loop. Our input sequence acts as a punchcard, consisting of instructions and memory for data read/writes. We demonstrate that a constant number of encoder layers can emulate basic computing blocks, including embedding edit operations, non-linear functions, function calls, program counters, and conditional branches. Using these building blocks, we emulate a small instruction-set computer. This allows us to map iterative algorithms to programs that can be executed by a looped, 13-layer transformer. We show how this transformer, instructed by its input, can emulate a basic calculator, a basic linear algebra library, and in-context learning algorithms that employ backpropagation. Our work highlights the versatility of the attention mechanism, and demonstrates that even shallow transformers can execute full-fledged, general-purpose programs.
Looped Transformers as Programmable Computers Angeliki Giannouw *, Shashank Rajputw∗ , Jy-yong Sohnw , Kangwook Leew , Jason D. Leep , Dimitris Papailiopoulosw p Princeton University arXiv:2301.13196v1 [cs.LG] 30 Jan 2023…
saved by
related reading
- A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
- [1807.03819] Universal Transformersarxiv.org
- Thinking like Transformersrush.github.io
- The Annotated Transformernlp.seas.harvard.edu
- All the Transformer Math You Need to Know | How To Scale Your Modeljax-ml.github.io
- Transformers from Scratche2eml.school
- The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
- Can LLMs Be Computers?percepta.ai
- Thinking Like Transformersarxiv.org
- Are Transformers Turing-complete? A Good Disguise Is All You Need.lifeiscomputation.com
- transformer_attention.pdfarxiv.org
- Transformer (deep learning) - Wikipediaen.wikipedia.org