H-Nets - the Future | Goomba Lab

Homepage of the Goomba AI Lab @ CMU MLD.

H-Nets - the Future | Goomba Lab H-Nets - the Future This post is part of a two-part series. H-Nets: the Past H-Nets: the Future [ Paper ] [ Code ] In this post, I’m going to try to convince you why H-Nets are fundamental and important. There was only so much content that could make it to the paper, and I think there are a lot of downstream consequences and interesting technical connections that we didn’t cover. Much of this will be based on deeper (but mostly unvalidated) intuitions I have and speculative implications about H-Nets. For fun, I’ll formulate several concrete hypotheses and predi

related reading

• H-Nets - the Past | Goomba Labgoombalab.github.io
• Dynamic Chunking for End-to-End Hierarchical Sequence Modelingarxiv.org
• H3: Language Modeling with State Space Models and (Almost) No Attention · Hazy Researchhazyresearch.stanford.edu
• On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
• A History of Large Language Modelsgregorygundersen.com
• Hyena Hierarchy: Towards Larger Convolutional Language Models · Hazy Researchhazyresearch.stanford.edu
• Sparse Autoencoders Find Highly Interpretable Features in Language Modelsarxiv.org
• machine learning imindslice.substack.com
• The Annotated Transformernlp.seas.harvard.edu
• [2212.14052] Hungry Hungry Hippos: Towards Language Modeling with State Space Modelsarxiv.org
• Looking back at speculative decodingresearch.google
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com