On MLA

Multi-head Latent Attention (MLA) is a variant of multi-head attention which was introduced in the DeepSeek-V2 paper1. There are several variants of multi-head attention whose purpose is primarily to reduce the KV-cache size, which is a memory bottleneck that emerges from scaling large models. These methods, which include Group-Query Attention and Multi-Query Attention, are primarily considered performance tradeoffs, i.e. the performance is worse, but you get to scale them much further by reducing the memory overhead. In comparison, MLA accomplishes this by using a low-rank factorized projection matrix, operating a bit like multi-query attention where instead of repeating a single head several times, you decompress a latent vector to yield a unique, appropriate corresponding K and V head for each Q head. DeepSeek claims this not only helps the memory overhead, but also improves the model rather than suffering for its inclusion. The basic idea is as follows: There is also an additional

On MLA On MLA Table of Contents Understanding Multi-Head Latent Attention Why Low-Rank Factorization? Aside: Perturbations in Low-Rank Decomposition Related Work KV Cache MHA Variants Rotary Position Embeddings (RoPE) Experiments Experiment 1 Experiment 2 Experiment 3 Results Modeling Results Inference Time Experiments Discussion Appendix: Other Learnings Extra Experiments Interesting Pytorch Things Understanding Multi-Head Latent Attention Your browser does not support videos Multi-head Latent Attention (MLA) is a variant of multi-head attention which was introduced in the DeepSeek-V2 paper 1

related reading

• A Gentle Introduction to Multi-Head Latent Attention (MLA) - MachineLearningMastery.commachinelearningmastery.com
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Big LLM Architecture Comparisonmagazine.sebastianraschka.com
• LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
• Linear Attention Fundamentals | Hailey Schoelkopfhaileyschoelkopf.github.io
• Multi Query Attention (MQA) and Grouped-Query Attention (GQA)tinkerd.net
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• 2502.11089arxiv.org
• [2305.13245] GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointsarxiv.org