Deep Double Descent (cross-posted on OpenAI blog)

By Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever This is a lightly edited and expanded version of the following post on the OpenAI blog about the followi…

By Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever This is a lightly edited and expanded version of the following post on the OpenAI blog about the following paper . While I usually don’t advertise my own papers on this blog, I thought this might be of interest to theorists, and a good follow up to my prior post . I promise not to make a habit out of it. –Boaz TL;DR: Our paper shows that double descent occurs in conventional modern deep learning settings: visual classification in the presence of label noise (CIFAR 10, CIFAR 100) and machine

related reading

• [2303.14151] Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzlearxiv.org
• Understanding “Deep Double Descent” — LessWronglesswrong.com
• Understanding deep learning requires rethinking generalizationarxiv.org
• The Scaling Hypothesis · Gwern.netgwern.net
• [2605.01172] A Theory of Generalization in Deep Learningarxiv.org
• Infinite Limits of Neural Networks - Kempner Institutekempnerinstitute.harvard.edu
• Just Ask for Generalization | Eric Jangevjang.com
• Maybe I was too harsh on deep learning theory (three days ago) — LessWronglesswrong.com
• Double descent in human learning · Chris Saidchris-said.io
• Scaling Laws, Carefully | Lil'Loglilianweng.github.io
• [2604.21691] There Will Be a Scientific Theory of Deep Learningarxiv.org
• A Theory of Deep Learning | Elements of a Vector Spaceelonlit.com