[2510.26692] Kimi Linear: An Expressive, Efficient Attention Architecture

Abstract:We introduce Kimi Linear, a hybrid linear attention architecture that, for the first time, outperforms full attention under fair comparisons across various scenarios -- including short-context, long-context, and reinforcement learning (RL) scaling regimes. At its core lies Kimi Delta Attention (KDA), an expressive linear attention module that extends Gated DeltaNet with a finer-grained gating mechanism, enabling more effective use of limited finite-state RNN memory. Our bespoke chunkwise algorithm achieves high hardware efficiency through a specialized variant of the Diagonal-Plus-Low-Rank (DPLR) transition matrices, which substantially reduces computation compared to the general DPLR formulation while remaining more consistent with the classical delta rule. We pretrain a Kimi Linear model with 3B activated parameters and 48B total parameters, based on a layerwise hybrid of KDA and Multi-Head Latent Attention (MLA). Our experiments show that with an identical training recipe, Kimi Linear outperforms full MLA with a sizeable margin across all evaluated tasks, while reducing KV cache usage by up to 75% and achieving up to 6 times decoding throughput for a 1M context. These results demonstrate that Kimi Linear can be a drop-in replacement for full attention architectures with superior performance and efficiency, including tasks with longer input and output lengths. To support further research, we open-source the KDA kernel and vLLM implementations, and release the pre-trained and instruction-tuned model checkpoints.

[2510.26692] Kimi Linear: An Expressive, Efficient Attention Architecture Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Computation and Language arXiv:2510.26692 (cs) [Submitted on 30 Oct 2025 ( v1 ), last revised 1 Nov 2025 (this version, v2)] Title: Kimi Linear: An Expressive, Efficient Attention Architecture Authors: Kimi Team : Yu Zhang , Zongyu Lin , Xingcheng Yao , Jiaxi Hu , Fanqing Meng , Chengyin Liu , Xin Men , Songlin Yang , Zhiyuan Li , Wentao Li , Enzhe Lu , Weizhou

related reading

• Kimi Linear: An Expressive, Efficient Attention Architecturearxiv.org
• Kimi Linear: An Expressive, Efficient Attention Architecturealphaxiv.org
• ali (@waterloo_intern) on Xx.com
• [2607.07953] Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routingarxiv.org
• Linear Attention Fundamentals | Hailey Schoelkopfhaileyschoelkopf.github.io
• The Big LLM Architecture Comparisonmagazine.sebastianraschka.com
• DeltaNet Explained (Part I) | Songlin Yangsustcsonglin.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Log-Linear Attentionarxiv.org
• Linear Transformers Are Faster After All – Manifest AImanifestai.com
• [2412.06464] Gated Delta Networks: Improving Mamba2 with Delta Rulearxiv.org
• 2502.11089arxiv.org