The most cited paper of the century is a brilliant hack - YouTube
Welch Labs explores the technical breakthrough of residual networks and how skip connections solved the shattered gradient problem in deep learning. This architectural innovation transformed neural networks, enabling unprecedented depth and shifting the fundamental understanding of how these models store and refine information during the learning process. Follow along using the transcript. Welch Labs explores the technical breakthrough of residual networks and how skip connections solved the shattered gradient problem in deep learning. This architectural innovation transformed neural networks, enabling unprecedented depth and shifting the fundamental understanding of how these models store and refine information during the learning process. Follow along using the transcript.
Thank you to Jane Street for sponsoring this video: https://www.janestreet.com/join-jane-street/internships/?utm_source=welchlabs26&utm_medium=youtube&utm_campaign=welch-labs Welch Labs Book: https://www.welchlabs.com/resources/ai-book-ezrzm-msrmc Patreon: https://www.patreon.com/c/welchlabs SECTIONS 0:00 - Intro 1:04 - When Deep Learning Stopped Working 3:17 - Tracing Data Forward 8:18 - The Degradation Problem 9:11 - Probing the Model 13:20 - Loss Landscapes 15:49 - Shattered Gradients 16:39 - Residual Networks 21:22 - Jane Street is a Fascinating Place to Work 23:45 - ResNets Force a…
saved by
related reading
- Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
- The Decade of Deep Learning | Leo Gaobmk.sh
- The Annotated Transformernlp.seas.harvard.edu
- What is Residual Connection? | Towards Data Sciencetowardsdatascience.com
- Residual neural network - Wikipediaen.wikipedia.org
- ResNets: Why do they perform better than Classic ConvNets? (Conceptual Analysis) | Towards Data Sciencetowardsdatascience.com
- H-Nets - the Past | Goomba Labgoombalab.github.io
- Cutting the Skip: Training Residual-Free Transformersarxiv.org
- bachlechner21a.pdfproceedings.mlr.press
- NL.pdfabehrouz.github.io
- Understanding deep learning requires rethinking generalizationarxiv.org
- GitHub - adam-maj/deep-learning: A deep-dive on the entire history of deep-learninggithub.com