Hyena Hierarchy: Towards Larger Convolutional Language Models · Hazy Research
hazyresearch.stanford.edu · 1,682 words · saved by 2 readers
Hyena is a large language model that uses long convolutions and gating to reach attention quality with lower time complexity.
Hyena Hierarchy: Towards Larger Convolutional Language Models · Hazy Research Mar 7, 2023 · 9 min read Hyena Hierarchy: Towards Larger Convolutional Language Models Michael Poli* , Stefano Massaroli* , Eric Nguyen* , Dan Fu , Tri Dao , Stephen A. Baccus , Yoshua Bengio , Stefano Ermon , and Chris Ré . Hyena cubs are born with their eyes open and can walk within minutes! But also, Hyena is a new operator for large language models that uses long convolutions and gating, reaching attention quality with lower time complexity. (Hyena image generated by lexica.art) We're excited to share our latest
saved by
related reading
- A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
- From Deep to Long Learning? · Hazy Researchhazyresearch.stanford.edu
- H3: Language Modeling with State Space Models and (Almost) No Attention · Hazy Researchhazyresearch.stanford.edu
- H-Nets - the Past | Goomba Labgoombalab.github.io
- Post-Transformers - Hyena Hierarchy - by Alex Mackenziewhynowtech.substack.com
- Dynamic Chunking for End-to-End Hierarchical Sequence Modelingarxiv.org
- [2212.14052] Hungry Hungry Hippos: Towards Language Modeling with State Space Modelsarxiv.org
- Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
- Linear Attention Fundamentals | Hailey Schoelkopfhaileyschoelkopf.github.io
- Extending Context is Hard | kaiokendevkaiokendev.github.io
- Mamba: The Easy Wayjackcook.com
- 2502.11089arxiv.org