Sparser Block-Sparse Attention via Token Permutation

N/A

Sparser Block-Sparse Attention via Token Permutation Xinghao Wang 1 2 Pengyu Wang 1 2 Dong Zhang 1 Chenkun Tan 1 2 Shaojun Zhou 1 2 Zhaoxiang Liu 3 Shiguo Lian 3 Fangxu Liu 4 Kai Song 4 Xipeng Qiu 1 2 5 Abstract 1. Introduction Modern Large Language Models (LLMs)…

related reading

• 2502.11089arxiv.org
• transformer_attention.pdfarxiv.org
• A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Modelsarxiv.org
• Sparse Autoencoders Find Highly Interpretable Features in Language Modelsarxiv.org
• [2205.14135] FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenessarxiv.org
• 2502.13189arxiv.org
• Sparse Attention Post-Training for Mechanistic Interpretabilityarxiv.org
• Optimizing Mixture of Block Attentionarxiv.org
• [2502.11089] Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentionarxiv.org
• Overleaf Examplearxiv.org
• [2604.20920] Simplified Sparse Attention via Gist Tokensarxiv.org
• [2502.13189] MoBA: Mixture of Block Attention for Long-Context LLMsarxiv.org