✳flâneur — a map of the web's best reading

[1609.04836] On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima

arxiv.org · 732 words · saved by 1 readers

Abstract:The stochastic gradient descent (SGD) method and its variants are algorithms of choice for many Deep Learning tasks. These methods operate in a small-batch regime wherein a fraction of the training data, say $32$-$512$ data points, is sampled to compute an approximation to the gradient. It has been observed in practice that when using a larger batch there is a degradation in the quality of the model, as measured by its ability to generalize. We investigate the cause for this generalization drop in the large-batch regime and present numerical evidence that supports the view that large-batch methods tend to converge to sharp minimizers of the training and testing functions - and as is well known, sharp minima lead to poorer generalization. In contrast, small-batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation. We discuss several strategies to attempt to help large-batch methods eliminate this generalization gap.

[1609.04836] On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Machine Learning arXiv:1609.04836 (cs) [Submitted on 15 Sep 2016 ( v1 ), last revised 9 Feb 2017 (this version, v2)] Title: On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima Authors: Nitish Shirish Keskar , Dheevatsa Mudigere , Jorge Nocedal , Mikhail Smelyanskiy , Ping Tak Peter Tang View a PDF of the paper titled

Explore this link on the map →

related reading