[1606.09282] Learning without Forgetting

Abstract:When building a unified vision system or gradually adding new capabilities to a system, the usual assumption is that training data for all tasks is always available. However, as the number of tasks grows, storing and retraining on such data becomes infeasible. A new problem arises where we add new capabilities to a Convolutional Neural Network (CNN), but the training data for its existing capabilities are unavailable. We propose our Learning without Forgetting method, which uses only new task data to train the network while preserving the original capabilities. Our method performs favorably compared to commonly used feature extraction and fine-tuning adaption techniques and performs similarly to multitask learning that uses original task data we assume unavailable. A more surprising observation is that Learning without Forgetting may be able to replace fine-tuning with similar old and new task datasets for improved new task performance.

View PDF HTML (experimental) Abstract:When building a unified vision system or gradually adding new capabilities to a system, the usual assumption is that training data for all tasks is always available. However, as the number of tasks grows, storing and retraining on such data becomes infeasible. A new problem arises where we add new capabilities to a Convolutional Neural Network (CNN), but the training data for its existing capabilities are unavailable. We propose our Learning without Forgetting method, which uses only new task data to train the network while preserving the original…

saved by

• Sophie Wang

related reading

• [1606.09282] Learning without Forgettingarxiv.org
• Understanding Memorization via Loss Curvaturegoodfire.ai
• The Continual Learning Problemjessylin.com
• NL.pdfabehrouz.github.io
• Why We Need Continual Learning | Andreessen Horowitza16z.com
• Loss of plasticity in deep continual learning | Naturenature.com
• Self-Distillation Enables Continual Learningarxiv.org
• Meta-Learning: Learning to Learn Fastlilianweng.github.io
• Bitter Lessons from Distillation Robustifies Unlearningbrucewlee.com
• Early Data Exposure Improves Robustness to Subsequent Fine-Tuningarxiv.org
• Learning to Remember — Lawrence Fengar-forum.github.io
• augustus odena on X: "I have a bunch of thoughts about continual learning and nothing to do with them (I'm working on something else) so I figured I'd just turn them into a post: First: I think people use "continual learning" to point at a cluster of issues that are related but distinct. I'll list" / Xx.com