Transformers Are Secretly Collectives of Spin Systems | mcbal

✨ Update (April 2023): Consider reading Spin-Model Transformers: A Physics-Inspired Class of Transformer Modules where we continue building on the intuition of probing a spin system to engineer its collective response but get rid of the assumption of symmetric coupling matrices by shifting focus from equilibrium free energies to dynamical mean-field approximations of non-equilibrium vector-spin models. Come join the fun. In this post, we try to distill a unifying perspective out of ideas developed in a series of longer posts on understanding transformers as physical systems: We argue that a blueprint of the neural-network architecture of the archetypical transformer module can be derived from the structure of physical spin systems familiar from classical statistical mechanics. More specifically, we claim that the forward pass of transformer modules maps onto computing magnetizations in vector-spin models in response to incoming data. We imagine transformers as collectives of differenti

Transformers Are Secretly Collectives of Spin Systems Published on November 23, 2021 · Last updated on November 29, 2021 · Matthias Bal · 14 min read blog ✨ Update (April 2023): Consider reading Spin-Model Transformers: A Physics-Inspired Class of Transformer Modules where we continue building on the intuition of probing a spin system to engineer its collective response but get rid of the assumption of symmetric coupling matrices by shifting focus from equilibrium free energies to dynamical mean-field approximations of non-equilibrium vector-spin models. Introduction In this post, we try to di

saved by

• Clementine Yang

related reading

• Spin-Model Transformers | mcbalmcbal.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transformers from Scratche2eml.school
• Transformers from scratch | peterbloem.nlpeterbloem.nl
• Transformer Circuits Threadtransformer-circuits.pub
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• The Annotated Transformernlp.seas.harvard.edu
• Everything About Transformerskrupadave.com
• On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
• Transformer (deep learning) - Wikipediaen.wikipedia.org