Mamba 2 - Transformers are SSMs: Generalized Models and Efficient Algorithms Through SSS Duality - YouTube
youtube.com · 11,327 words · saved by 1 readers
Follow along using the transcript. Follow along using the transcript.
Paper here: https://arxiv.org/abs/2405.21060 Code!: https://github.com/state-spaces/mamba/blob/main/mamba_ssm/modules/mamba2.py Notes: https://drive.google.com/file/d/1--XGPFeXQyx4CPxgYjzR4qrLd-baLWQC/view?usp=sharing 00:00 Intro 01:45 SSMs 08:00 Quadratic form of an SSM 15:02 Expanded form of an SSM 24:00 Attention - it's all you need?? 29:55 Kernel attention 32:50 Linear attention 34:32 Relating attention to SSMs 38:35 Defining the M matrix 43:48 Splitting the M matrix 46:30 Off diagonal decomposition 54:00 Recurrent form of the off diagonal 1:03:30 Combining the M matrix blocks and code…
related reading
- 2405.21060-Transformers are SSMs (Mamba2)arxiv.org
- On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
- Mamba No. 5 (A Little Bit Of…) | Sparse Notesjameschen.io
- Mamba Explainedthegradient.pub
- State Space Duality (Mamba-2) Part II - The Theory | Goomba Labgoombalab.github.io
- Mamba: The Easy Wayjackcook.com
- A Visual Guide to Mamba and State Space Modelsnewsletter.maartengrootendorst.com
- A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
- State Space Duality (Mamba-2) Part I - The Model | Goomba Labgoombalab.github.io
- State Space Duality (Mamba-2) Part I - The Model | Tri Daotridao.me
- The Mamba Effect: State Space Models Taking on Transformershungleai.substack.com
- State Space Duality (Mamba-2) Part II - The Theory | Tri Daotridao.me