Post-Transformers - Hyena Hierarchy - by Alex Mackenzie
Welcome to new subs + thanks a million for the Zig love! This thing has grown a whole lot faster than my inner-pessimist could have imagined. So, grazie! If you enjoy these primers, I’d appreciate it if you favourite this post’s tweet or LinkedIn post to help spread the good (?) word of Why Now! As always, I’m at alex@tapestry.vc should you ever wish to chat. In particular, I’ve spent a lot of time thinking/writing about what new applications could exist thanks to LLMs — if you have too, I’d love to say hello. Hyena clans are arranged in what’s known as a “linear dominance hierarchy”. This means that there’s a one-to-one chain of command: Hyena 3 reports to Hyena 2, and Hyena 2 reports to, you guessed it, Hyena 1. This structure is also present in baboons, wolves, and perhaps most notably, chickens.. ever wondered where the phrase “pecking order” came from? (That one’s for free). Don’t worry, Why Now isn’t expanding it’s purview into primers on the animal kingdom. Although, do expect a
Post-Transformers - Hyena Hierarchy Breaking the Quadratic Barrier Alex Mackenzie Sep 04, 2023 11 Share Welcome to new subs + thanks a million for the Zig love! This thing has grown a whole lot faster than my inner-pessimist could have imagined. So, grazie! If you enjoy these primers, I’d appreciate it if you favourite this post’s tweet or LinkedIn post to help spread the good (?) word of Why Now! As always, I’m at alex@tapestry.vc should you ever wish to chat. In particular, I’ve spent a lot of time thinking/writing about what new applications could exist thanks to LLMs — if you have too, I’d
saved by
related reading
- Hyena Hierarchy: Towards Larger Convolutional Language Models · Hazy Researchhazyresearch.stanford.edu
- The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
- A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
- Transformers from Scratche2eml.school
- Everything About Transformerskrupadave.com
- H-Nets - the Past | Goomba Labgoombalab.github.io
- The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
- A Conceptual Guide to Transformers: Part Ibenlevinstein.substack.com
- Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
- On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
- Transformers Explained Visually (Part 1): Overview of Functionality | Towards Data Sciencetowardsdatascience.com
- Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper? | Hacker Newsnews.ycombinator.com