[2210.07229] Mass-Editing Memory in a Transformer
Abstract:Recent work has shown exciting promise in updating large language models with new memories, so as to replace obsolete information or add specialized knowledge. However, this line of work is predominantly limited to updating single associations. We develop MEMIT, a method for directly updating a language model with many memories, demonstrating experimentally that it can scale up to thousands of associations for GPT-J (6B) and GPT-NeoX (20B), exceeding prior work by orders of magnitude. Our code and data are at this https URL.
View PDF HTML (experimental) Abstract:Recent work has shown exciting promise in updating large language models with new memories, so as to replace obsolete information or add specialized knowledge. However, this line of work is predominantly limited to updating single associations. We develop MEMIT, a method for directly updating a language model with many memories, demonstrating experimentally that it can scale up to thousands of associations for GPT-J (6B) and GPT-NeoX (20B), exceeding prior work by orders of magnitude. Our code and data are at this https URL. Comments: 18 pages, 11…
saved by
related reading
- Exact Expressive Power of Transformers with Paddingarxiv.org
- A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
- [2005.14165] Language Models are Few-Shot Learnersarxiv.org
- arxiv.org/pdf/2505.24832arxiv.org
- [2501.00663] Titans: Learning to Memorize at Test Timearxiv.org
- [2304.13343] Unleashing Infinite-Length Input Capacity for Large-scale Language Models with Self-Controlled Memory Systemarxiv.org
- Scalable MatMul-free Language Modelingarxiv.org
- [2603.23516] MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokensarxiv.org
- [2605.15156] MeMo: Memory as a Modelarxiv.org
- 2305.19370arxiv.org
- Do Language Models Need Sleep? Offline Recurrence for Improved Online Inferencearxiv.org
- [2512.23675] End-to-End Test-Time Training for Long Contextarxiv.org