Can a Language Model Learn Facts Continually in Its Weights? | Base Labs

Working to advance and democratize open-source intelligence.

Research · JUL 2026 Continual learning asks the weights to acquire facts after training and retain them through further writes. Abstract: Continual learning promises a language model that keeps acquiring knowledge after training, with each new fact written into its weights. Whether weight writes can support accumulation remains undecided. We follow invented facts written into Qwen3 models from creation through sequences of twenty to one hundred later writes, using held-out questions of five types, with the original model given the fact in its prompt as the reference. Across these…

saved by

• Nathan Chen

related reading

• Why We Need Continual Learning | Andreessen Horowitza16z.com
• The Continual Learning Problemjessylin.com
• Understanding Memorization via Loss Curvaturegoodfire.ai
• [2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasksarxiv.org
• Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Dataalignment.anthropic.com
• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• [2605.15156] MeMo: Memory as a Modelarxiv.org
• Self-Adapting Language Modelsarxiv.org
• What's so hard about continuous learning?seangoedecke.com
• Supermemory — Memory and continual learning for agentssupermemory.ai
• Physics of LMs 3.1arxiv.org
• What are the real problems of continual learning?infinitefaculty.substack.com