Structured State Spaces: Combining Continuous-Time, Recurrent, and Convolutional Models · Hazy Research

In our previous post, we introduced the challenges of continuous time series and overviewed the three main deep learning paradigms for addressing them: recurrence, convolutions, and continuous-time models. We discussed their strengths and weaknesses, and summarized recent progress on connecting these families of models.

Structured State Spaces: Combining Continuous-Time, Recurrent, and Convolutional Models · Hazy Research Jan 14, 2022 · 22 min read Structured State Spaces: Combining Continuous-Time, Recurrent, and Convolutional Models Albert Gu , Karan Goel , Khaled Saab , and Chris Ré In our previous post, we introduced the challenges of continuous time series and overviewed the three main deep learning paradigms for addressing them: recurrence, convolutions, and continuous-time models. We discussed their strengths and weaknesses, and summarized recent progress on connecting these families of models. In this

saved by

• Rishi Kothari
• Amos You
• Yudhister Joel Kumar

related reading

• On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
• The Topological Trouble With Transformersarxiv.org
• A Visual Guide to Mamba and State Space Modelsnewsletter.maartengrootendorst.com
• The Unreasonable Effectiveness of Recurrent Neural Networkskarpathy.github.io
• A Visual Guide to Mamba and State Space Models - Maarten Grootendorstmaartengrootendorst.com
• Simplifying S4 · Hazy Researchhazyresearch.stanford.edu
• Diffusion is not necessarily Spectral Autoregression | Fabian Falckfabianfalck.com
• [2111.00396] Efficiently Modeling Long Sequences with Structured State Spacesarxiv.org
• Mamba No. 5 (A Little Bit Of…) | Sparse Notesjameschen.io
• H3: Language Modeling with State Space Models and (Almost) No Attention · Hazy Researchhazyresearch.stanford.edu
• H-Nets - the Past | Goomba Labgoombalab.github.io
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai