State Space Duality (Mamba-2) Part II - The Theory | Tri Dao

In Part I of this series, we defined the state space dual (SSD) model. In isolation, this model is relatively simple to define, and we claimed that it can be computed either as an SSM recurrence or with an attention-like pattern. If you just want to use the model, feel free to skip this post! In this post, we’ll dive into the theory behind the model. We’ll derive the SSD “duality” in two completely separate ways, one starting from the SSM perspective and one from the attention perspective. Each method is actually much more broad than the SSD model itself, and the union of these two strong generalizations is what we call the SSD framework. This framework provides a rich body of connections between state space models, attention, and structured matrices. While the SSD model can be viewed as a specific instantiation of each prong of the framework, the SSD framework is much more general opens up many directions for future work. For each of the two parts of this framework, we’ll Note that th

State Space Duality (Mamba-2) Part II - The Theory | Tri Dao State Space Duality (Mamba-2) Part II - The Theory Part I - The Model Part II - The Theory Part III - The Algorithm Part IV - The Systems In Part I of this series, we defined the state space dual (SSD) model . In isolation, this model is relatively simple to define, and we claimed that it can be computed either as an SSM recurrence or with an attention-like pattern. If you just want to use the model, feel free to skip this post! In this post, we’ll dive into the theory behind the model. We’ll derive the SSD “duality” in two completel

related reading

• State Space Duality (Mamba-2) Part II - The Theory | Goomba Labgoombalab.github.io
• State Space Duality (Mamba-2) Part I - The Model | Tri Daotridao.me
• State Space Duality (Mamba-2) Part I - The Model | Goomba Labgoombalab.github.io
• A Visual Guide to Mamba and State Space Modelsnewsletter.maartengrootendorst.com
• Mamba No. 5 (A Little Bit Of…) | Sparse Notesjameschen.io
• On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
• H3: Language Modeling with State Space Models and (Almost) No Attention · Hazy Researchhazyresearch.stanford.edu
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Linear Attention Fundamentals | Hailey Schoelkopfhaileyschoelkopf.github.io
• A Visual Guide to Mamba and State Space Models - Maarten Grootendorstmaartengrootendorst.com
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Structured State Spaces: Combining Continuous-Time, Recurrent, and Convolutional Models · Hazy Researchhazyresearch.stanford.edu