You could have invented transformers from memory

Retrieving a character from memory. The recall worsens as the number of memories grows. Large language models today can recall obscure facts from every corner of the world. Most recently, I was running late for a friend’s board game night, so I asked Astra to remind me the rules to Dune: Imperium. It gladly narrated them to me as I drove over and helped me remember the rules well enough for me to

Retrieving a character from memory. The recall worsens as the number of memories grows. Large language models today can recall obscure facts from every corner of the world. Most recently, I was running late for a friend’s board game night, so I asked Astra to remind me the rules to Dune: Imperium. It gladly narrated them to me as I drove over and helped me remember the rules well enough for me to play again. We’re so used to asking models about anything that we take their vast knowledge for granted. How do models remember so much information? And how much can they store? In this post,…

related reading

• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• The Continual Learning Problemjessylin.com
• On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
• 🧠 MLPs are Hebbian Memories: A Simple Recipe for Fact-Storing Transformershazyresearch.stanford.edu
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
• [2501.00663] Titans: Learning to Memorize at Test Timearxiv.org
• Memory makes computation universal, remember?thinks.lol
• Understanding Memorization via Loss Curvaturegoodfire.ai
• ali (@waterloo_intern) on Xx.com
• arxiv.org/pdf/2505.24832arxiv.org
• All About Transformer Inferencejax-ml.github.io