🧠 MLPs are Hebbian Memories: A Simple Recipe for Fact-Storing Transformers · Hazy Research
A kernel-memory view of fact-storing MLPs gives a closed-form construction, optimal capacity guarantees, and the first theoretical account of fact storage inside Transformer blocks.
⚡ TL;DR It sounds like sci-fi, but it's real: we can instantly build knowledge into a Transformer block, no training required! Previously, we showed that MLPs can be constructed to store facts. Since then, we found a much simpler and more powerful explanation: a Transformer MLP is naturally a Hebbian memory. This view lets us develop the first closed-form fact-storing MLP construction, requiring no gradient descent 🤯, that packs facts at the information-theoretically optimal rate and is usable within Transformer blocks for factual recall! Full paper team: Roberto Garcia*, Jerry Liu*,…
saved by
related reading
- Language Model Circuits Are Sparse in the Neuron Basis | Transluce AItransluce.org
- A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
- How might LLMs store facts | Deep Learning Chapter 7youtube.com
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- You could have invented transformers from memoryalexwang.ai
- The Continual Learning Problemjessylin.com
- Linear Attention Fundamentals | Hailey Schoelkopfhaileyschoelkopf.github.io
- Understanding Memorization via Loss Curvaturegoodfire.ai
- NL.pdfabehrouz.github.io
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- [2501.00663] Titans: Learning to Memorize at Test Timearxiv.org
- Do Machine Learning Models Memorize or Generalize?pair.withgoogle.com