Multi-Query & Grouped-Query Attention

Recall that the standard self-attention mechanism in Transformer models1 computes an attention score between every pair of tokens in an input sequence of text. Internally, each token in the input sequence, represented by an embedding (that is, a vector of floating point values), is first projected into a query and key vector. For example, consider an input sequence of 5 tokens, such as the phrase Hope is a waking dream, with each token represented by an embedding of dimension 8, forming a 5 × 8 5×8 matrix. This matrix of input embeddings is passed separately through the Query Layer and Key Layer (each a linear layer), producing a query and key vector for each token: Attention scores are then computed by multiplying the query vectors by the transpose of the key vectors; or, put another way, the attention scores are computed by taking the dot product of each query vector with each key vector. The resulting matrix of attention scores can be thought of as representing the strength of some

Multi Query Attention (MQA) and Grouped-Query Attention (GQA) Multi-Query & Grouped-Query Attention Sunday, December 24, 2023 Multi-Head Self-Attention Recall that the standard self-attention mechanism in Transformer models 1 computes an attention score between every pair of tokens in an input sequence of text. Internally, each token in the input sequence, represented by an embedding (that is, a vector of floating point values), is first projected into a query and key vector. For example, consider an input sequence of 5 tokens, such as the phrase Hope is a waking dream , with each token repres

saved by

• Kevin Wang

related reading

• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• transformer_attention.pdfarxiv.org
• Transformers Explained Visually (Part 3): Multi-head Attention, deep dive | Towards Data Sciencetowardsdatascience.com
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• 1706.03762arxiv.org
• Cache strategies · Hugging Facehuggingface.co
• Understanding and Coding Self-Attention, Multi-Head Attention, Causal-Attention, and Cross-Attention in LLMsmagazine.sebastianraschka.com
• What is an attention mechanism? | IBMibm.com
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io