Simplifying S4 · Hazy Research

Explaining S4 from the first principles of signal processing.

Simplifying S4 · Hazy Research Jun 21, 2022 · 42 min read Simplifying S4 Chris Ré , Dan Fu , Karan Goel , Khaled Saab . One goal of deep learning research is to find the simplest architectures that lead to the amazing results that we've seen for the last few years. In that spirit, we discuss the recent S4 architecture, which we think is simple—the structured state space model at its heart has been the most basic building block for generations of electrical engineers. However, S4 has seemed mysterious—and there are some subtleties to getting it to work in deep learning settings efficiently. We

related reading

• Structured State Spaces: Combining Continuous-Time, Recurrent, and Convolutional Models · Hazy Researchhazyresearch.stanford.edu
• The Annotated Transformernlp.seas.harvard.edu
• Mamba No. 5 (A Little Bit Of…) | Sparse Notesjameschen.io
• [2111.00396] Efficiently Modeling Long Sequences with Structured State Spacesarxiv.org
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• H-Nets - the Past | Goomba Labgoombalab.github.io
• Tiny TPUtinytpu.com
• Linear Attention Fundamentals | Hailey Schoelkopfhaileyschoelkopf.github.io
• [2111.00396] Efficiently Modeling Long Sequences with Structured State Spacesarxiv.org
• On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
• H3: Language Modeling with State Space Models and (Almost) No Attention · Hazy Researchhazyresearch.stanford.edu
• An Intuitive Explanation of Sparse Autoencoders for LLM Interpretability | Adam Karvonenadamkarvonen.github.io