H3: Language Modeling with State Space Models and (Almost) No Attention · Hazy Research

Replacing attention with SSMs in language modeling.

H3: Language Modeling with State Space Models and (Almost) No Attention · Hazy Research Jan 23, 2023 · 7 min read H3: Language Modeling with State Space Models and (Almost) No Attention Dan Fu , Tri Dao , Khaled Saab , Armin Thomas , Atri Rudra , and Chris Ré . State space models (SSMs) are strong general-purpose sequence models, but have underperformed attention in language modeling. How can we close this gap? In this blog post, we’ll take a look at some critical capability gaps – using some simple synthetic languages of in-context as a guide. We’ll use our understanding to build H3 (Hungry H

saved by

• Winnie Xu
• Michelle Pan
• Varun Shenoy
• Karan Dalal
• Chloe Wong

related reading

• [2212.14052] Hungry Hungry Hippos: Towards Language Modeling with State Space Modelsarxiv.org
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
• Mamba Explainedthegradient.pub
• H-Nets - the Past | Goomba Labgoombalab.github.io
• Hyena Hierarchy: Towards Larger Convolutional Language Models · Hazy Researchhazyresearch.stanford.edu
• Do Language Models Need Sleep? Offline Recurrence for Improved Online Inferencearxiv.org
• 2312.00752arxiv.org
• A History of Large Language Modelsgregorygundersen.com
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• Dynamic Chunking for End-to-End Hierarchical Sequence Modelingarxiv.org