State Space Duality (Mamba-2) Part II - The Theory | Goomba Lab

Homepage of the Goomba AI Lab @ CMU MLD.

State Space Duality (Mamba-2) Part II - The Theory | Goomba Lab State Space Duality (Mamba-2) Part II - The Theory Part I - The Model Part II - The Theory Part III - The Algorithm Part IV - The Systems In Part I of this series, we defined the state space dual (SSD) model . In isolation, this model is relatively simple to define, and we claimed that it can be computed either as an SSM recurrence or with an attention-like pattern. If you just want to use the model, feel free to skip this post! In this post, we’ll dive into the theory behind the model. We’ll derive the SSD “duality” in two comple

related reading

• State Space Duality (Mamba-2) Part II - The Theory | Tri Daotridao.me
• State Space Duality (Mamba-2) Part I - The Model | Goomba Labgoombalab.github.io
• State Space Duality (Mamba-2) Part I - The Model | Tri Daotridao.me
• 2405.21060-Transformers are SSMs (Mamba2)arxiv.org
• Mamba No. 5 (A Little Bit Of…) | Sparse Notesjameschen.io
• A Visual Guide to Mamba and State Space Modelsnewsletter.maartengrootendorst.com
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• 2312.00752arxiv.org
• On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
• Log-Linear Attentionarxiv.org
• H3: Language Modeling with State Space Models and (Almost) No Attention · Hazy Researchhazyresearch.stanford.edu
• Linear Attention Fundamentals | Hailey Schoelkopfhaileyschoelkopf.github.io