On MLA

planetbanatt.net · 5,729 words · saved by 1 readers

Multi-head Latent Attention (MLA) is a variant of multi-head attention which was introduced in the DeepSeek-V2 paper1. There are several variants of multi-head attention whose purpose is primarily to reduce the KV-cache size, which is a memory bottleneck that emerges from scaling large models. These methods, which include Group-Query Attention and Multi-Query Attention, are primarily considered performance tradeoffs, i.e. the performance is worse, but you get to scale them much further by reducing the memory overhead. In comparison, MLA accomplishes this by using a low-rank factorized projection matrix, operating a bit like multi-query attention where instead of repeating a single head several times, you decompress a latent vector to yield a unique, appropriate corresponding K and V head for each Q head. DeepSeek claims this not only helps the memory overhead, but also improves the model rather than suffering for its inclusion. The basic idea is as follows: There is also an additional

On MLA On MLA Table of Contents Understanding Multi-Head Latent Attention Why Low-Rank Factorization? Aside: Perturbations in Low-Rank Decomposition Related Work KV Cache MHA Variants Rotary Position Embeddings (RoPE) Experiments Experiment 1 Experiment 2 Experiment 3 Results Modeling Results Inference Time Experiments Discussion Appendix: Other Learnings Extra Experiments Interesting Pytorch Things Understanding Multi-Head Latent Attention Your browser does not support videos Multi-head Latent Attention (MLA) is a variant of multi-head attention which was introduced in the DeepSeek-V2 paper 1

On MLA

related reading