Your Softmax Is Broken. Here's the Fix. - YouTube

Enjoy the videos and music you love, upload original content, and share it all with friends, family, and the world on YouTube.

Standard implementations of softmax, log-softmax, and cross-entropy in every major framework have numerical BUGS! Where does the gradient go, and why does a rounding error make a model's loss spike? Check out the video to find out! Chapters 0:00 The paradox: 99.999997% sure, yet zero gradient 1:46 Softmax and cross-entropy from scratch 5:40 Floating point: why 1 + something small = 1 10:06 Softmax Collapse: the gradient falls off a cliff 13:38 The fix: avoid adding one to a tiny number 17:23 Adam optimizer, grokking, and the Slingshot spikes 20:53 Why margins stop at log(1/ε) 24:07 Softmax…

saved by

• Ishaan Panigrahi

related reading

• Training Models Too Big for One GPU | ZeRO & FSDPyoutube.com
• Attention Is Off By One – Evan Millerevanmiller.org
• Making Deep Learning go Brrrr From First Principleshorace.io
• Understanding softmax and the negative log-likelihoodljvmiranda921.github.io
• A Recipe for Training Neural Networkskarpathy.github.io
• Yes you should understand backprop | by Andrej Karpathy | Mediumkarpathy.medium.com
• Scaling Laws, Carefully | Lil'Loglilianweng.github.io
• the bug that taught me more about PyTorch than years of using it | Elana Simonelanapearl.github.io
• A Recipe for Training Neural Networkskarpathy.github.io
• The 4-bitter Lesson | humans&humansand.ai
• Softmax Linear Unitstransformer-circuits.pub
• CS231n Deep Learning for Computer Visioncs231n.github.io