flâneur

QR Decomp at the Speed of Light

ml-mike.com · 5,885 words · saved by 1 readers

How I placed 5th in the GPU MODE qr_v2 competition with common sense engineering, linear algebra tricks, an agent swarm, and a dream. I recently placed 5th in GPU MODE's qr_v2 competition, where a bunch of kernel writers and I raced to make batched QR decomposition lightning fast on NVIDIA's B200. If you missed the recent Twitter drama, QR has been getting attention because it can show up inside various rivaling deep learning optimizer algorithms. The competition made one thing very clear: the default PyTorch path is really slow for this workload. My final submission reached a 1499µs geometric mean on the validation set of 12 sizes, roughly 280× faster than torch.geqrf. Along the way I learned the B200 architecture from the ground up and found a few almost-heretical math tricks that make Householder QR much faster than I initially assumed possible. If you have never heard of Householder QR decomposition, then you're like me two weeks ago. I will build up the algorithm from scratch, dra

QR Decomp at the Speed of Light I recently placed 5th in GPU MODE's qr_v2 competition , where a bunch of kernel writers and I raced to make batched QR decomposition lightning fast on NVIDIA's B200. If you missed the recent Twitter drama, QR has been getting attention because it can show up inside various rivaling deep learning optimizer algorithms. The competition made one thing very clear: the default PyTorch path is really slow for this workload. My final submission reached a 1499µs geometric mean on the validation set of 12 sizes, roughly 280× faster than torch.geqrf . Along the

saved by

related reading