Understanding the Neural Tangent Kernel – EigenTales

My attempt at distilling the ideas behind the neural tangent kernel that is making waves in recent theoretical deep learning research.

Understanding the Neural Tangent Kernel This gif depicts the training dynamics of a neural network. Find out how by reading the rest of this post. A flurry of recent papers in theoretical deep learning tackles the common theme of analyzing neural networks in the infinite-width limit. At first, this limit may seem impractical and even pointless to study. However, it turns out that neural networks in this regime simplify to linear models with a kernel called the neural tangent kernel . Gradient descent is therefore very simple to study. While this may seem promising at first, empirical results s

saved by

• Neel Redkar
• Sudarsh K
• Yudhister Joel Kumar

related reading

• Some Math behind Neural Tangent Kernel | Lil'Loglilianweng.github.io
• Understanding the Neural Tangent Kernel – EigenTalesrajatvd.github.io
• nn-notes.pdfboris-hanin.github.io
• Lecture 7 - Deep Learning Foundations: Neural Tangent Kernelsyoutube.com
• [2604.06366] Stochastic Gradient Descent in the Saddle-to-Saddle Regime of Deep Linear Networksarxiv.org
• Neural Networks, Manifolds, and Topology -- colah's blogcolah.github.io
• Neural tangent kernel - Wikipediaen.wikipedia.org
• Neural networks and deep learningneuralnetworksanddeeplearning.com
• Maybe I was too harsh on deep learning theory (three days ago) — LessWronglesswrong.com
• Yes you should understand backprop | by Andrej Karpathy | Mediumkarpathy.medium.com
• A Recipe for Training Neural Networkskarpathy.github.io
• [2605.01172] A Theory of Generalization in Deep Learningarxiv.org