The Dimpled Manifold Model of Adversarial Examples in Machine Learning

The extreme fragility of deep neural networks when presented with tiny perturbations in their inputs was independently discovered by several research groups in 2013, but in spite of enormous effort these adversarial examples remained a baffling phenomenon with no clear explanation. In this paper we introduce a new conceptual framework (which we call the Dimpled Manifold Model) which provides a simple explanation for why adversarial examples exist, why their perturbations have such tiny norms, why these perturbations look like random noise, and why a network which was adversarially trained with incorrectly labeled images can still correctly classify test images. In the last part of the paper we describe the results of numerous experiments which strongly support this new model, and in particular our assertion that adversarial perturbations are roughly perpendicular to the low dimensional manifold which contains all the training examples.

The Dimpled Manifold Model of Adversarial Examples in Machine Learning Adi Shamir Odelia Melamed Faculty of Math&CS Faculty of Math&CS Weizmann Institute of Science Weizmann Institute of Science Israel…

related reading

• Neural Networks, Manifolds, and Topology -- colah's blogcolah.github.io
• Adversarial Examples Are Not Bugs, They Are Features – gradient sciencegradientscience.org
• A Recipe for Training Neural Networkskarpathy.github.io
• Some Lessons from Adversarial Machine Learning | FAR.AIfar.ai
• Feature Visualizationdistill.pub
• [1610.00768] Technical Report on the CleverHans v2.1.0 Adversarial Examples Libraryarxiv.org
• A Recipe for Training Neural Networkskarpathy.github.io
• The Little Book of Deep Learningfleuret.org
• Understanding deep learning requires rethinking generalizationarxiv.org
• The Decade of Deep Learning | Leo Gaobmk.sh
• Solving adversarial attacks in computer vision as a baby version of general AI alignment | Stanislav Fortstanislavfort.com
• Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugesarxiv.org