Can a Language Model Learn Facts Continually in Its Weights? | Base Labs

Working to advance and democratize open-source intelligence.

Research · JUL 2026 Continual learning asks the weights to acquire facts after training and retain them through further writes. Abstract: Continual learning promises a language model that keeps acquiring knowledge after training, with each new fact written into its weights. Whether weight writes can support accumulation remains undecided. We follow invented facts written into Qwen3 models from creation through sequences of twenty to one hundred later writes, using held-out questions of five types, with the original model given the fact in its prompt as the reference. Across these…

saved by

• Nathan Chen

related reading

• The Continual Learning Problemjessylin.com
• You can’t imitation-learn how to continual-learn — LessWronglesswrong.com
• [2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasksarxiv.org
• Why We Need Continual Learning | Andreessen Horowitza16z.com
• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• What are the real problems of continual learning?infinitefaculty.substack.com
• [2511.08579] Training Language Models to Explain Their Own Computationsarxiv.org
• [2605.12484] Learning, Fast and Slow: Towards LLMs That Adapt Continuallyarxiv.org
• Base Models Know How to Reason, Thinking Models Learn Whenarxiv.org
• 2212.03827arxiv.org
• Training Large Language Models to Reason in a Continuous Latent Spacearxiv.org
• Self-Adapting Language Modelsarxiv.org