How Attention Got So Efficient [GQA/MLA/DSA] - YouTube

Enjoy the videos and music you love, upload original content, and share it all with friends, family, and the world on YouTube.

Attention mechanisms have been the key behind the recent AI boom. What happened after the multi-head attention in the seminal 2017 Transformer paper? In this video, we break down several core ideas that make attention efficient and scalable. 00:00 Introduction 00:35 Tokenization 01:21 Attention (vector form) 04:26 Attention (matrix form) 07:07 Key-Value caching 09:42 Multi-Query Attention (MQA) 11:03 Grouped Query Attention (GQA) 13:32 Multi-head Latent Attention (MLA) 15:37 MLA at inference time 18:15 Applying RoPE to MLA (decoupled RoPE) 22:18 DeepSeek Sparse Attention (DSA) 23:57…

saved by

• Jirat C
• Mason Wang

related reading

• Linear Attention Fundamentals | Hailey Schoelkopfhaileyschoelkopf.github.io
• On MLAplanetbanatt.net
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Multi Query Attention (MQA) and Grouped-Query Attention (GQA)tinkerd.net
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• The Big LLM Architecture Comparisonmagazine.sebastianraschka.com
• A Gentle Introduction to Multi-Head Latent Attention (MLA) - MachineLearningMastery.commachinelearningmastery.com
• 2502.11089arxiv.org
• [2603.23516] MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokensarxiv.org
• Transformers Explained Visually (Part 3): Multi-head Attention, deep dive | Towards Data Sciencetowardsdatascience.com
• How trillion-parameter AI models work and how to serve themyoutube.com
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io