Mamba: The Hard Way
This blog is about Mamba a recent neural architecture that can be roughly thought of as a modern recurrent neural network (RNN). The model works really well and is a legitimate competitor with the ubiquitous Transformer architecture. It has gotten a lot of attention. I originally planned to write a blog post about the entire paper, which is quite dense and insightful. However I become fascinated just by the S6 algorithm as described here. This algorithm describes how one can compute an extremely large RNN efficiently on modern hardware, and extends ideas explored in S4 and S5 from recent years. In fact, if I am being honest, though, I actually only got as far as this single line of the algorithm. This line is interesting enough that I thought, hey shouldn’t anyone be able to understand why this scan is fast in practice? Turns out this is a bit tricky. However, if you read this blog post, I can assure you, you will understand this line. (Perhaps more than you would ever want). To do thi
Explore this link on the map →