✳flâneur — a map of the web's best reading

Mamba No. 5 (A Little Bit Of…) | Sparse Notes

jameschen.io · 6,650 words · saved by 2 readers

In this post, I attempt to provide a walkthrough of the essence of the Mamba state space model architecture, occasionally sacrificing some rigor for intuition and overall pedagogical friendliness. I don’t assume readers have any familiarity with state space models, but I do assume some familiarity with machine learning and mathematical notation. If at any point you spot any errors, typos, or confusing wording, please let me know! Mamba is a state space model (SSM) architecture that improves upon the S4 architecture. Sometimes known as S6, it makes two important modifications to S4: Mamba parallelizes well during training, scales well with context length, performs inference efficiently, and most importantly, displays strong empirical results. Sequence models can be placed on a spectrum based on their approach to information representation, from highly compressed (e.g. RNNs) to highly explicit (e.g. transformers). Consider a vanilla RNN: The fixed size state ℎ 𝑡 − 1 h t−1 ​ represen

In this post, I attempt to provide a walkthrough of the essence of the Mamba state space model architecture, occasionally sacrificing some rigor for intuition and overall pedagogical friendliness. I don’t assume readers have any familiarity with state space models, but I do assume some familiarity with machine learning and mathematical notation. If at any point you spot any errors, typos, or confusing wording, please let me know! TL;DR Setting the stage Exhibit A: the RNN Pros Cons Exhibit B: the transformer Pros Cons Why Mamba? Why now? Linear time-invariant state space models Continuous form

Explore this link on the map →

saved by

related reading