2502.13189

N/A

M O BA: M IXTURE OF B LOCK ATTENTION FOR L ONG -C ONTEXT LLM S T ECHNICAL R EPORT Enzhe Lu1 Zhejun Jiang1 Jingyuan Liu1 Yulun Du1 Tao Jiang1 Chao Hong1 Shaowei Liu1 Weiran He1 Enming Yuan1 Yuzhi Wang1 Zhiqi Huang1 Huan Yuan1 arXiv:2502.13189v1 [cs.LG] 18 Feb 2025…

related reading

• [2502.13189] MoBA: Mixture of Block Attention for Long-Context LLMsarxiv.org
• 2502.11089arxiv.org
• Optimizing Mixture of Block Attentionarxiv.org
• [2512.23675] End-to-End Test-Time Training for Long Contextarxiv.org
• [2205.14135] FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenessarxiv.org
• 2409.02908arxiv.org
• [2501.00663] Titans: Learning to Memorize at Test Timearxiv.org
• Extending Context is Hard | kaiokendevkaiokendev.github.io
• Large Language Diffusion Modelsarxiv.org
• [2603.23516] MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokensarxiv.org
• [2507.04239] Scaling Context Requires Rethinking Attentionarxiv.org
• Kimi Linear: An Expressive, Efficient Attention Architecturearxiv.org