Jane Street Blog - Does batch size matter?

This post is aimed at readers who are already familiar with stochastic gradient descent (SGD) and terms like “batch size”. For an introduction to these ideas...

Jane Street Blog - Does batch size matter? Does batch size matter? Oct 31, 2017 | 12 min read Share on Facebook Share on Twitter Share on LinkedIn By: Chris Hardin This post is aimed at readers who are already familiar with stochastic gradient descent (SGD) and terms like “batch size”. For an introduction to these ideas, I recommend Goodfellow et al.’s Deep Learning , in particular the introduction and, for more about SGD, Chapter 8. The relevance of SGD is that it has made it feasible to work with much more complex models than was formerly possible. There is a lot of talk about batch size in

related reading

• Batch size 1 or ∞kothasuhas.github.io
• [2101.12176] On the Origin of Implicit Regularization in Stochastic Gradient Descentarxiv.org
• Just Ask for Generalization | Eric Jangevjang.com
• Scaling Laws, Carefully | Lil'Loglilianweng.github.io
• Why Does SGD Love Flat Minima? Marginally Better blogrishit-dagli.github.io
• Making Deep Learning go Brrrr From First Principleshorace.io
• Notes on the Origin of Implicit Regularization in SGDinference.vc
• Understanding deep learning requires rethinking generalizationarxiv.org
• [1706.02677] Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourarxiv-vanity.com
• A Recipe for Training Neural Networkskarpathy.github.io
• Scaling is subtler than it seemsberen.io
• [1609.04836] On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimaarxiv.org