Future leakage in block-quantized attention | MatX

Quantizing attention improves efficiency on two fronts: the model has higher compute throughput, and loads fewer bytes per key/value. However, training with blo

January 09, 2026 Akshay Mishra, Reiner Pope, Sanjit Neelam, Daniel Heinlein, Vaclav Cvicek, Zaal Vasania, and James Hill-Khurana Quantizing attention improves efficiency on two fronts: the model has higher compute throughput, and loads fewer bytes per key/value. However, training with block quantized attention can break causal modeling. We present a fix that enables training with MXFP4 in both attention and the attention gradient. Causal modeling In causal language modeling, the final logits at position ii must depend only on tokens at positions ≤i\le i. Future leakage is when…

saved by

• Aaron Pham
• Akira Yoshiyama

related reading

• Quantization from the ground upngrok.com
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• All the Transformer Math You Need to Know | How To Scale Your Modeljax-ml.github.io
• 2502.11089arxiv.org
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• Linear Attention Fundamentals | Hailey Schoelkopfhaileyschoelkopf.github.io
• Overleaf Examplearxiv.org
• The 4-bitter Lesson | humans&humansand.ai
• ali (@waterloo_intern) on Xx.com
• 1.5x faster MoE training with custom MXFP8 kernels · Cursorcursor.com
• A Guide to Quantization in LLMs | Symbl.aisymbl.ai