Understanding the Neural Tangent Kernel – EigenTales

My attempt at distilling the ideas behind the neural tangent kernel that is making waves in recent theoretical deep learning research.

Understanding the Neural Tangent Kernel This gif depicts the training dynamics of a neural network. Find out how by reading the rest of this post. A flurry of recent papers in theoretical deep learning tackles the common theme of analyzing neural networks in the infinite-width limit. At first, this limit may seem impractical and even pointless to study. However, it turns out that neural networks in this regime simplify to linear models with a kernel called the neural tangent kernel . Gradient descent is therefore very simple to study. While this may seem promising at first, empirical results s

saved by

• Neel Redkar
• Sudarsh K
• Yudhister Joel Kumar

related reading

• Some Math behind Neural Tangent Kernel | Lil'Loglilianweng.github.io
• Understanding the Neural Tangent Kernel – EigenTalesrajatvd.github.io
• Neural tangent kernel - Wikipediaen.wikipedia.org
• [2605.01172] A Theory of Generalization in Deep Learningarxiv.org
• [1806.07572] Neural Tangent Kernel: Convergence and Generalization in Neural Networksarxiv.org
• Deriving Muonjeremybernste.in
• Modular Manifolds - Thinking Machines Labthinkingmachines.ai
• Neural network training makes beautiful fractals | Jascha’s blogsohl-dickstein.github.io
• Infinite Limits of Neural Networks - Kempner Institutekempnerinstitute.harvard.edu
• A Theory of Deep Learning | Elements of a Vector Spaceelonlit.com
• On neural scaling and the quanta hypothesisericjmichaud.com
• [2604.21691] There Will Be a Scientific Theory of Deep Learningarxiv.org