flâneur

A short note on some aspects of long context attention | nor's blog

nor-blog.pages.dev · 13,990 words · saved by 2 readers

One of my recent side-quests is to understand which architectural choices for models scale well, to ensure that future work that interests me remains meaningful. This post is a byproduct of some exploration along those lines - initially I was going to write about some minor original work (theoretical with some experimental results) on how to scale QK-norm to long contexts well, but I quickly realized it might be better to also write a summary of what I think is good for long context from an attention perspective (which might be more speculative than the rest). It was especially clear when I started writing the post after getting experimental results - that was when I started searching literature for non-trivial baselines and found lots of places where potentially-not-ideal heuristics were used (including LLaMA 4 and Grok 2). So in this post, I will talk about some theory, some small-scale experimental results, and what I think is a reasonable starting point for long-context attention d

Table of Contents Introduction Meta Scoping What goes wrong at long context Training stability and QK-norm Looking at the distribution The Gaussian assumption and 2 log ⁡ n \sqrt{2 \log n} The Beta assumption and n 2 / ( d − 1 ) n^{2 / (d - 1)} Local vs global behavior and inductive biases Similar existing literature Scalable softmax Position-dependent scaling and scale-invariant attention Positional encodings and hybrid attention (local and global) Attention sinks and gating Revisiting QK-norm and norm information Experimental details Acknowledgements References Final notes Introduction

saved by

related reading