PyTorch + NumPy Bug

A bug that plagues thousands of open-source ML projects.

Update: The problem is now fixed in Pytorch but can still happen in tensorflow-keras. Discussion on reddit . Bugs in ML code are notoriously hard to fix - they don’t cause compile errors but silently regress accuracy. Once you have endured the pain and fixed one of these, the lesson is forever etched into your brain, right? Wrong . Recently, an old foe made a comeback - a familiar bug bit me again! As before, the performance improved significantly after fixing it. The bug was subtle and easy to make. How many others has it done damage to? Curious, I downloaded over a hundred thousand repositor

saved by

• Vincent Huang

related reading

• Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
• the bug that taught me more about PyTorch than years of using it | Elana Simonelanapearl.github.io
• PyTorch internals : ezyang's blogblog.ezyang.com
• Tensors — PyTorch Tutorials 2.12.0+cu130 documentationpytorch.org
• PiTorch: ML on Baremetal Raspberry Pis | projectsmasonjwang.com
• Tensors — PyTorch Tutorials 2.13.0+cu130 documentationdocs.pytorch.org
• PyTorch (@PyTorch) / Xx.com
• Accelerating Generative AI with PyTorch II: GPT, Fast – PyTorchpytorch.org
• The 37 Implementation Details of Proximal Policy Optimization · The ICLR Blog Trackiclr-blog-track.github.io
• GPT in 60 Lines of NumPy | Jay Modyjaykmody.com
• Learning PyTorch with Examples — PyTorch Tutorials 2.12.0+cu130 documentationpytorch.org
• PyTorch, a year in…. – PyTorchpytorch.org