H-Nets - the Past | Goomba Lab

Homepage of the Goomba AI Lab @ CMU MLD.

H-Nets - the Past | Goomba Lab H-Nets - the Past This post is part of a two-part series. H-Nets: the Past H-Nets: the Future [ Paper ] [ Code ] The H-Net model has been a dream of mine for years. I’ve been fortunate enough to be able to work with my student Sukjun Hwang who made the technical breakthroughs behind this model, solving (or at least making serious progress towards) what I consider to be a very difficult but foundational problem for deep learning. In this post, I provide an informal personal recounting of the motivation and history of this project, adding various context and discus

saved by

• Justin Wang
• Neel Redkar
• Sudarsh K
• Vincent Cheng
• Yudhister Joel Kumar
• Rohan Tibrewal

related reading

• On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
• H-Nets - the Future | Goomba Labgoombalab.github.io
• Dynamic Chunking for End-to-End Hierarchical Sequence Modelingarxiv.org
• Learning to (Learn at Test Time): RNNs with Expressive Hidden Statesarxiv.org
• The Annotated Transformernlp.seas.harvard.edu
• H3: Language Modeling with State Space Models and (Almost) No Attention · Hazy Researchhazyresearch.stanford.edu
• A History of Large Language Modelsgregorygundersen.com
• [2512.23675] End-to-End Test-Time Training for Long Contextarxiv.org
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• Hyena Hierarchy: Towards Larger Convolutional Language Models · Hazy Researchhazyresearch.stanford.edu
• GitHub - adam-maj/deep-learning: A deep-dive on the entire history of deep-learninggithub.com
• Mamba No. 5 (A Little Bit Of…) | Sparse Notesjameschen.io