google seeming solved efficient attention - by Celeste 🌱

ceselder.substack.com · 1,114 words · saved by 1 readers

Bit more technical than usual but want to try out writing technical articles, thank you to @bycloud on twitter for giving me the idea to write about this. It’s really his discovery. He makes excellent videos on the state of LLM progress, I highly suggest checking him out. Traditional attention in transformers is O(n^2), which doesn’t scale well. So, there’s been a big race towards what we call subquadratic attention or even linear attention. It’s a very active area of research, with Kimi spearheading Kimi linear attention and Deepseek inventing Deepseek Sparse Attention (DSA). Both of these (chinese) labs publish their findings. This is not something OpenAI or Anthropic or Google does, they would rather keep this a secret. More on that later. So let’s say we want to test this right? Test for how good your attention really is? The closest we can get is a little benchmark where the models are supposed to retrieve “a needle out of a haystack”. Stuff like a big story of 1 million tokens an

EDIT: Or you know, they could be doing this in production, but that raises questions on why other frontier labs aren’t. [Epistemic status - speculative, but sort of grounded, might be wrong - don’t take too seriously] Alternative title: why Gemini 3 pro gets to be so big Bit more technical than usual but want to try out writing technical articles, thank you to @bycloud on twitter for giving me the idea to write about this. It’s really his discovery. He makes excellent videos on the state of LLM progress, I highly suggest checking him out. Traditional attention in transformers is O(n^2),…

google seeming solved efficient attention - by Celeste 🌱

related reading