✳flâneur — a map of the web's best reading

"Attention", "Transformers", in Neural Network "Large Language Models"

bactra.org · 12,842 words · saved by 1 readers

I find this literature irritating and opaque. This is at least somewhat because I do not yet understand it well, and there's too much of it. But clearly I need to wrap my head around it, before I become technically obsolete. My scare quotes in the title of these notes thus derive in part from jealousy and fear. But only in part: the names here seem like proof positive that McDermott's critique of "wishful mnemonics" needs to be re-introduced into the basic curriculum of AI. So here's a related idea, due to (independently) E. A. Nadaraya (1964) and Geoffrey S. Watson (1964). Introduce a kernel function 𝐾 ( 𝑢 , 𝑣 ) which measures how similar 𝑢 is to 𝑣 ; this function should be non-negative, and should be maximized when 𝑢 = 𝑣 . Now use those as weights in the average: ∑ 𝑖 = 1 𝑛 𝑦 𝑖 𝐾 ( 𝑥 𝑖 , 𝑥 𝑜 ) ∑ 𝑗 = 1 𝑛 𝐾 ( 𝑥 𝑗 , 𝑥 𝑜 ) Dividing by the sum of the 𝐾 's ensures that this is indeed a weighted average. Thus Nadaraya-Watson smoothing, a.k.a. kernel smoothing

"Attention", "Transformers", in Neural Network "Large Language Models" Notebooks "Attention", "Transformers", in Neural Network "Large Language Models" Last update : 16 Jul 2026 20:59 First version : Late March 2023 \[ \newcommand{\Prob}[1]{\mathbb{P}\left( #1 \right)} \] I find this literature irritating and opaque. This is at least somewhat because I do not yet understand it well, and there's too much of it. But clearly I need to wrap my head around it, before I become technically obsolete. My scare quotes in the title of these notes thus derive in part from jealousy and fear. But only in pa

Explore this link on the map →

related reading