Spin-Model Transformers | mcbal

In a series of previous blog posts, we have tried to connect the forward pass of a transformer neural-network module to computing mean magnetizations in disordered Ising-like vector-spin models with parameterized couplings and external magnetic fields. According to this perspective, the forward pass of a transformer module can be understood as computing statistical observables given a specific realization of quenched couplings and external magnetic fields while the backward pass nudges the parameterized couplings and external magnetic fields. Physically, the transformer module represents an interacting many-body system modulating its behavior by learning to respond to being probed and driven in all kinds of ways. However, both the mean-field message-passing approach of Deep Implicit Attention: A Mean-Field Theory Perspective on Attention Mechanisms (2021) and the saddle-point free-energy approach of Transformers from Spin Models: Approximate Free Energy Minimization (2021) inherently r

Spin-Model Transformers Published on June 19, 2022 · Last updated on December 7, 2023 · Matthias Bal · 50 min read blog Introduction ✨ TL;DR: We interpret and implement transformer modules as driven, disordered vector-spin models whose response behavior can be shaped by learning parameterized interactions, gradually steering a cascade of near-equilibrium steady-state magnetizations towards solving a given objective. Using dynamical mean-field theory, we show that a first-order approximation of the update equations for the magnetizations reproduces residual and attention terms. Going to second-

saved by

• Davis Brown

related reading

• Transformers Are Secretly Collectives of Spin Systems | mcbalmcbal.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Transformers from Scratche2eml.school
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• The Annotated Transformernlp.seas.harvard.edu
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• Transformer Circuits Threadtransformer-circuits.pub
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transformers from Scratchbrandonrohrer.com
• Transformers from scratch | peterbloem.nlpeterbloem.nl
• The Annotated Transformernlp.seas.harvard.edu