flâneur

We reverse-engineered Flash Attention 4 | Hacker News

news.ycombinator.com · 3,080 words · saved by 1 readers

As a fellow Tri Dao groupie and lucky duck who gets to build on Hopper/Blackwell clusters, I find it amazing how difficult it is becoming to write kernels that saturate GPU hardware. When I squint, there appears to be a trend emerging across work like FA4, monolithic (mega) kernels, etc. Namely, a subversion of the classic CUDA programming model in the form of fine grained task based parallelism, managed entirely in “user space”. Not exactly sure what’s ahead but I’m strapping in for a wild ride… I was also reminded of HazyResearch's MegaKernels. Didn't want to distract from the main thrust of the post, but definitely think that's a promising approach. Reductively, software engineering means taking an idea and mapping it into code. So one form of "reverse" engineering would be taking the code and extracting the ideas. That's what we did here. Because the source is public, there's quite a lot to work with from the start -- the warp specializations are named and there are helpful comment

We reverse-engineered Flash Attention 4 | Hacker News Hacker News new | past | comments | ask | show | jobs | submit login We reverse-engineered Flash Attention 4 ( modal.com ) 134 points by birdculture 8 months ago | hide | past | favorite | 48 comments refibrillator 8 months ago | next [–] Great exposition, loved the touch of humor. Please do the backward pass when it’s published. As a fellow Tri Dao groupie and lucky duck who gets to build on Hopper/Blackwell clusters, I find it amazing how difficult it is becoming to write kernels that saturate GPU hardware. When I squint, there appea

saved by

related reading