flâneur

Multi-Query & Grouped-Query Attention

tinkerd.net · 4,691 words · saved by 1 readers

Recall that the standard self-attention mechanism in Transformer models1 computes an attention score between every pair of tokens in an input sequence of text. Internally, each token in the input sequence, represented by an embedding (that is, a vector of floating point values), is first projected into a query and key vector. For example, consider an input sequence of 5 tokens, such as the phrase Hope is a waking dream, with each token represented by an embedding of dimension 8, forming a 5 × 8 5×8 matrix. This matrix of input embeddings is passed separately through the Query Layer and Key Layer (each a linear layer), producing a query and key vector for each token: Attention scores are then computed by multiplying the query vectors by the transpose of the key vectors; or, put another way, the attention scores are computed by taking the dot product of each query vector with each key vector. The resulting matrix of attention scores can be thought of as representing the strength of some

Multi Query Attention (MQA) and Grouped-Query Attention (GQA) Multi-Query & Grouped-Query Attention Sunday, December 24, 2023 Multi-Head Self-Attention Recall that the standard self-attention mechanism in Transformer models 1 computes an attention score between every pair of tokens in an input sequence of text. Internally, each token in the input sequence, represented by an embedding (that is, a vector of floating point values), is first projected into a query and key vector. For example, consider an input sequence of 5 tokens, such as the phrase Hope is a waking dream , with each token repres

saved by

related reading