Post-Transformers - Hyena Hierarchy - by Alex Mackenzie

Welcome to new subs + thanks a million for the Zig love! This thing has grown a whole lot faster than my inner-pessimist could have imagined. So, grazie! If you enjoy these primers, I’d appreciate it if you favourite this post’s tweet or LinkedIn post to help spread the good (?) word of Why Now! As always, I’m at alex@tapestry.vc should you ever wish to chat. In particular, I’ve spent a lot of time thinking/writing about what new applications could exist thanks to LLMs — if you have too, I’d love to say hello. Hyena clans are arranged in what’s known as a “linear dominance hierarchy”. This means that there’s a one-to-one chain of command: Hyena 3 reports to Hyena 2, and Hyena 2 reports to, you guessed it, Hyena 1. This structure is also present in baboons, wolves, and perhaps most notably, chickens.. ever wondered where the phrase “pecking order” came from? (That one’s for free). Don’t worry, Why Now isn’t expanding it’s purview into primers on the animal kingdom. Although, do expect a

Post-Transformers - Hyena Hierarchy Breaking the Quadratic Barrier Alex Mackenzie Sep 04, 2023 11 Share Welcome to new subs + thanks a million for the Zig love! This thing has grown a whole lot faster than my inner-pessimist could have imagined. So, grazie! If you enjoy these primers, I’d appreciate it if you favourite this post’s tweet or LinkedIn post to help spread the good (?) word of Why Now! As always, I’m at alex@tapestry.vc should you ever wish to chat. In particular, I’ve spent a lot of time thinking/writing about what new applications could exist thanks to LLMs — if you have too, I’d

saved by

• Proby Shandilya

related reading

• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Everything About Transformerskrupadave.com
• Transformers from Scratche2eml.school
• Transformers from scratch | peterbloem.nlpeterbloem.nl
• transformer_attention.pdfarxiv.org
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Hyena Hierarchy: Towards Larger Convolutional Language Models · Hazy Researchhazyresearch.stanford.edu
• Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper? | Hacker Newsnews.ycombinator.com
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• What is an attention mechanism? | IBMibm.com