Multi-Query & Grouped-Query Attention
Recall that the standard self-attention mechanism in Transformer models1 computes an attention score between every pair of tokens in an input sequence of text. Internally, each token in the input sequence, represented by an embedding (that is, a vector of floating point values), is first projected into a query and key vector. For example, consider an input sequence of 5 tokens, such as the phrase Hope is a waking dream, with each token represented by an embedding of dimension 8, forming a 5 × 8 5×8 matrix. This matrix of input embeddings is passed separately through the Query Layer and Key Layer (each a linear layer), producing a query and key vector for each token: Attention scores are then computed by multiplying the query vectors by the transpose of the key vectors; or, put another way, the attention scores are computed by taking the dot product of each query vector with each key vector. The resulting matrix of attention scores can be thought of as representing the strength of some
Multi Query Attention (MQA) and Grouped-Query Attention (GQA) Multi-Query & Grouped-Query Attention Sunday, December 24, 2023 Multi-Head Self-Attention Recall that the standard self-attention mechanism in Transformer models 1 computes an attention score between every pair of tokens in an input sequence of text. Internally, each token in the input sequence, represented by an embedding (that is, a vector of floating point values), is first projected into a query and key vector. For example, consider an input sequence of 5 tokens, such as the phrase Hope is a waking dream , with each token repres
saved by
related reading
- How Attention Got So Efficient [GQA/MLA/DSA]youtube.com
- transformer_attention.pdfarxiv.org
- A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
- The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
- Understanding and Coding Self-Attention, Multi-Head Attention, Causal-Attention, and Cross-Attention in LLMsmagazine.sebastianraschka.com
- 2305.13245arxiv.org
- Transformers Explained Visually (Part 3): Multi-head Attention, deep dive | Towards Data Sciencetowardsdatascience.com
- [2305.13245] GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointsarxiv.org
- Linear Attention Fundamentals | Hailey Schoelkopfhaileyschoelkopf.github.io
- Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
- An Intuition for Attention | Jay Modyjaykmody.com
- Transformer (deep learning) - Wikipediaen.wikipedia.org