2502.11089

N/A

# link_1lmz0khri25.pdf ## Metadata - PDFFormatVersion=1.5 - IsLinearized=false - IsAcroFormPresent=false - IsXFAPresent=false - IsCollectionPresent=false - IsSignaturesPresent=false - CreationDate=D:20250228013606Z - Creator=LaTeX with hyperref - ModDate=D:20250228013606Z - Custom.PTEX.Fullbanner=This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) kpathsea version 6.3.5 - Producer=pdfTeX-1.40.25 - Trapped=False ## Contents ### Page 1 Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse AttentionJingyang Yuan∗1,2, Huazuo Gao1, Damai Dai1, Junyu Luo2, Liang Zhao

saved by

• Chloe Chia
• Sebastian J Zhao
• Yang Xu

related reading

• [2603.23516] MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokensarxiv.org
• [2502.11089] Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentionarxiv.org
• Overleaf Examplearxiv.org
• 2409.02908arxiv.org
• transformer_attention.pdfarxiv.org
• Sparse Attention Post-Training for Mechanistic Interpretabilityarxiv.org
• [2604.20920] Simplified Sparse Attention via Gist Tokensarxiv.org
• 2502.13189arxiv.org
• Sparse Autoencoders Find Highly Interpretable Features in Language Modelsarxiv.org
• pdfopenreview.net
• 1706.03762arxiv.org
• A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Modelsarxiv.org