Hyena Hierarchy: Towards Larger Convolutional Language Models · Hazy Research

Hyena is a large language model that uses long convolutions and gating to reach attention quality with lower time complexity.

Hyena Hierarchy: Towards Larger Convolutional Language Models · Hazy Research Mar 7, 2023 · 9 min read Hyena Hierarchy: Towards Larger Convolutional Language Models Michael Poli* , Stefano Massaroli* , Eric Nguyen* , Dan Fu , Tri Dao , Stephen A. Baccus , Yoshua Bengio , Stefano Ermon , and Chris Ré . Hyena cubs are born with their eyes open and can walk within minutes! But also, Hyena is a new operator for large language models that uses long convolutions and gating, reaching attention quality with lower time complexity. (Hyena image generated by lexica.art) We're excited to share our latest

saved by

• Stephen Lu
• Ivy Zhang

related reading

• From Deep to Long Learning? · Hazy Researchhazyresearch.stanford.edu
• Post-Transformers - Hyena Hierarchy - by Alex Mackenziewhynowtech.substack.com
• H3: Language Modeling with State Space Models and (Almost) No Attention · Hazy Researchhazyresearch.stanford.edu
• Mamba: The Easy Wayjackcook.com
• The Big LLM Architecture Comparisonmagazine.sebastianraschka.com
• H-Nets - the Past | Goomba Labgoombalab.github.io
• A short note on some aspects of long context attention | nor's blognor-blog.pages.dev
• [2512.23675] End-to-End Test-Time Training for Long Contextarxiv.org
• 1706.03762arxiv.org
• Extending Context is Hard | kaiokendevkaiokendev.github.io
• Subquadratic — How SSA Makes Long Context Practicalsubq.ai
• [2507.04239] Scaling Context Requires Rethinking Attentionarxiv.org