Han Guo on X: "LLM training is built on fast MatMuls. But many surrounding ops still run as memory-bound kernels. CODA reparameterizes them to hide in the matmul’s shadow, fused into its epilogue before results leave the chip. Bonus: LLMs can write fast CODA kernels too (approaching SoLs). https://t.co/cOTeMUr4py" / X
To view keyboard shortcuts, press question mark View keyboard shortcuts Home Explore Notifications Chat Grok Premium Money Bookmarks Creator Studio Articles Profile More Post christina @luoluo Post See new posts Conversation Han Guo @HanGuo97 LLM training is built on fast MatMuls. But many surrounding ops still run as memory-bound kernels. CODA reparameterizes them to hide in the matmul’s shadow, fused into its epilogue before results leave the chip. Bonus: LLMs can write fast CODA kernels too (approaching SoLs). 3:25 PM · May 21, 2026 · 146.8K Views 11 106 555 437 Relevant View quotes Post your reply Reply Han Guo @HanGuo97 · May 21 LLM training is dominated by compute-heavy ops like MatMuls and attention. But it also has many memory-heavy ops: norms, activations, residuals, reductions. These mostly move tensors around. As FP8/NVFP4 make FLOPs cheaper, data movement gets harder to ignore. Fig: ~1B LLaMA-3 Show more 2 3 51 5.7K Han Guo @HanGuo97 · May 21 ML frameworks make trai
Han Guo @HanGuo97 LLM training is built on fast MatMuls. But many surrounding ops still run as memory-bound kernels. CODA reparameterizes them to hide in the matmul’s shadow, fused into its epilogue before results leave the chip. Bonus: LLMs can write fast CODA kernels too (approaching SoLs). 10:25 PM · May 21, 2026 202K Views 16 0 1 6 103 0 1 0 3 689 0 6 8 9 534 0 5 3 4 Read 16 replies
related reading
- How to Land a Frontier Lab Jobvladfeinberg.com
- Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
- LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
- How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklogsiboehm.com
- ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
- MatX: High-throughput chips for LLMsmatx.com
- Scalable MatMul-free Language Modelingarxiv.org
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programsarxiv.org
- Inside NVIDIA GPUs: Anatomy of high performance matmul kernels - Aleksa Gordićaleksagordic.com
- KernelBench: Can LLMs Write GPU Kernels?scalingintelligence.stanford.edu
- How is LLaMa.cpp possible?finbarr.ca