azyResearch/ThunderKittens: Tile primitives for speedy kernels

Tile primitives for speedy kernels

ThunderKittens: Tile primitives for speedy kernels ThunderKittens is a framework to make it easy to write fast deep learning kernels in CUDA. It is built around three key principles: Simplicity. ThunderKittens is stupidly simple to write. Extensibility. ThunderKittens is natively embedded into CUDA, so that if you need more than ThunderKittens can offer, it won’t get in your way of building it yourself. Speed. Kernels written in ThunderKittens should be at least as fast as those written from scratch -- especially because ThunderKittens can do things the “right” way under the hood. We…

saved by

• Devansh Pandey

related reading

• [2410.20399] ThunderKittens: Simple, Fast, and Adorable AI Kernelsarxiv.org
• Inside NVIDIA GPUs: Anatomy of high performance matmul kernels - Aleksa Gordićaleksagordic.com
• GPUs Go Brrr · Hazy Researchhazyresearch.stanford.edu
• We reverse-engineered Flash Attention 4modal.com
• KernelBench: Can LLMs Write GPU Kernels?scalingintelligence.stanford.edu
• PyTorch internals : ezyang's blogblog.ezyang.com
• Modern GPU Programming For MLSys — Modern GPU Programming For MLSysmlc.ai
• Together AI | The AI Native Cloudtogether.ai
• Mixture-of-Kittens: our open-source MoE megakernel for NVL72scursor.com
• GitHub - wafer-ai/gpu-perf-engineering-resources: A curated resource list for learning AI performance engineering, from GPU fundamentals to production inference.github.com
• GitHub - linkedin/Liger-Kernel: Efficient Triton Kernels for LLM Traininggithub.com
• The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com