the bug that taught me more about PyTorch than years of using it | Elana Simon

a loss plateau that looked like my mistake turned out to be a PyTorch bug. tracking it down meant peeling back every layer of abstraction, from optimizer internals to GPU kernels. Elana Simon Stanford University Oct. 22, 2025 Expected to fix: my hyperparameters. Actually had to fix: PyTorch backend. My training loss plateaued and wouldn’t budge. Obviously I’d screwed something up. I tried every hyperparameter combination, rewrote my loss function, spent days assuming I’d made some stupid mistake. Because it’s always user error. This time, it wasn’t. It was a niche PyTorch bug that forced me through layers of abstraction I normally never think about: optimizer internals, memory layouts, dispatch systems, kernel implementations. Taught me more about the framework than years of using it. I had a surprisingly fun time with this bug hunt and wrote up the whole investigation step-by-step, explaining framework internals as they become necessary to crack the case. If you enjoy debugging myster

the bug that taught me more about PyTorch than years of using it | Elana Simon the bug that taught me more about PyTorch than years of using it a loss plateau that looked like my mistake turned out to be a PyTorch bug. tracking it down meant peeling back every layer of abstraction, from optimizer internals to GPU kernels. Expected to fix: my hyperparameters. Actually had to fix: PyTorch backend. My training loss plateaued and wouldn’t budge. Obviously I’d screwed something up. I tried every hyperparameter combination, rewrote my loss function, spent days assuming I’d made some stupid mistake.

saved by

• Nathan Chen
• Florent Tavernier
• Anirudh Valiveru

related reading

• The Continual Learning Problemjessylin.com
• Making Deep Learning go Brrrr From First Principleshorace.io
• 1.5x faster MoE training with custom MXFP8 kernels · Cursorcursor.com
• Using PyTorch + NumPy? You're making a mistake. · Tanel Pärnamaatanelp.github.io
• PyTorch internals : ezyang's blogblog.ezyang.com
• A Recipe for Training Neural Networkskarpathy.github.io
• Muon: An optimizer for hidden layers in neural networks | Keller Jordan blogkellerjordan.github.io
• Open Weights isn't Open Trainingworkshoplabs.ai
• A Recipe for Training Neural Networkskarpathy.github.io
• The Annotated Transformernlp.seas.harvard.edu
• Debugging Reinforcement Learning Systemsandyljones.com
• Yes you should understand backprop | by Andrej Karpathy | Mediumkarpathy.medium.com