ImageNet Training in Minutes – arXiv Vanity

arxiv-vanity.com · 8,057 words · saved by 1 readers

Since its creation, the ImageNet 1-k benchmark set has played a significant role as a benchmark for ascertaining the accuracy of different deep neural net (DNN) models on the classification problem. Moreover, in recent years it has also served as the principal benchmark for assessing different approaches to DNN training. Finishing a 90-epoch ImageNet-1k training with ResNet-50 on a NVIDIA M40 GPU takes 14 days. This training requires 1018 single precision operations in total. On the other hand, the world’s current fastest supercomputer can finish 2×1017 single precision operations per second. If we can make full use of the computing capability of a supercomputer for DNN training, we should be able to finish the 90-epoch ResNet-50 training in five seconds. Over the last two years a number of researchers have focused on how to close this significant performance gap through scaling DNN training to larger numbers of processors. Most successful approaches to scaling the training of ImageNet have used the approach of synchronous stochastic gradient descent. However, to scale synchronous stochastic gradient descent one must also increase the batch size used in each iteration.

ImageNet Training in Minutes Yang You 1 , Zhao Zhang 2 , Cho-Jui Hsieh 3 , James Demmel 1 , Kurt Keutzer 1 UC Berkeley 1 , TACC 2 , UC Davis 3 {youyang, demmel, keutzer}@cs.berkeley.edu; zzhang@tacc.utexas.edu; chohsieh@ucdavis.edu Abstract Since its creation, the ImageNet-1k benchmark set has played a significant role as a benchmark for ascertaining the accuracy of different deep neural net (DNN) models on the classification problem. Moreover, in recent years it has also served as the principal benchmark for assessing different approaches to DNN training. Finishing a 90-epoch ImageNet-1k trai

ImageNet Training in Minutes – arXiv Vanity

related reading