Simplifying S4 · Hazy Research

Explaining S4 from the first principles of signal processing.

Simplifying S4 · Hazy Research Jun 21, 2022 · 42 min read Simplifying S4 Chris Ré , Dan Fu , Karan Goel , Khaled Saab . One goal of deep learning research is to find the simplest architectures that lead to the amazing results that we've seen for the last few years. In that spirit, we discuss the recent S4 architecture, which we think is simple—the structured state space model at its heart has been the most basic building block for generations of electrical engineers. However, S4 has seemed mysterious—and there are some subtleties to getting it to work in deep learning settings efficiently. We

related reading

• Structured State Spaces: Combining Continuous-Time, Recurrent, and Convolutional Models · Hazy Researchhazyresearch.stanford.edu
• Tiny TPUtinytpu.com
• [2111.00396] Efficiently Modeling Long Sequences with Structured State Spacesarxiv.org
• The Little Book of Deep Learningfleuret.org
• Mamba No. 5 (A Little Bit Of…) | Sparse Notesjameschen.io
• Hands-On Machine Learning with Scikit-Learn and PyTorchageron.github.io
• Mamba: The Easy Wayjackcook.com
• A Visual Guide to Mamba and State Space Modelsnewsletter.maartengrootendorst.com
• DeltaNet Explained (Part I) | Songlin Yangsustcsonglin.github.io
• Your Transformer is Secretly an EOT Solver | Elements of a Vector Spaceelonlit.com
• Parallelizing Linear Transformers with the Delta Rule over Sequence Lengtharxiv.org
• [2111.00396] Efficiently Modeling Long Sequences with Structured State Spacesarxiv.org