2305.13245
arxiv.org · 3,525 words · saved by 1 readers
N/A
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie∗, James Lee-Thorp∗, Michiel de Jong∗ † Yury Zemlyanskiy, Federico Lebrón, Sumit Sanghai Google Research Abstract show…
related reading
- [2305.13245] GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointsarxiv.org
- transformer_attention.pdfarxiv.org
- [2005.14165] Language Models are Few-Shot Learnersarxiv.org
- 2305.14314arxiv.org
- 2502.11089arxiv.org
- Multi Query Attention (MQA) and Grouped-Query Attention (GQA)tinkerd.net
- [2205.14135] FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenessarxiv.org
- 1706.03762arxiv.org
- How Attention Got So Efficient [GQA/MLA/DSA]youtube.com
- 1911.02150arxiv.org
- 2403.09611.pdfarxiv.org
- 2005.14165arxiv.org