Optimizing Mixture of Block Attention
arxiv.org · 6,485 words · saved by 1 readers
N/A
Optimizing Mixture of Block Attention O PTIMIZING M IXTURE OF B LOCK ATTENTION Guangxuan Xiao1∗ Junxian Guo1∗ Kasra Mazaheri 1 Song Han1,2 1 MIT 2 NVIDIA https://github.com/mit-han-lab/flash-moba A BSTRACT Mixture of Block Attention…
related reading
- [2205.14135] FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenessarxiv.org
- 2502.11089arxiv.org
- [2502.13189] MoBA: Mixture of Block Attention for Long-Context LLMsarxiv.org
- 2502.13189arxiv.org
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- From Online Softmax to FlashAttentioncourses.cs.washington.edu
- Biao's Bloghebiao064.github.io
- [2603.23516] MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokensarxiv.org
- Overleaf Examplearxiv.org
- How Attention Got So Efficient [GQA/MLA/DSA]youtube.com
- Kimi Linear: An Expressive, Efficient Attention Architecturearxiv.org
- [2604.20920] Simplified Sparse Attention via Gist Tokensarxiv.org