Taking PyTorch for Granted | wh

A while back I challenged myself to implement micrograd in Rust using only the standard library. Along the way, I thought that it would be a fun to attempt to implement a fully functioning Tensor library on top of micrograd. I thought that my familarity with PyTorch will make this easier but having to do so without the higher-level abstractions of Python turned out much harder than expected. In t

A while back I challenged myself to implement micrograd in Rust using only the standard library. Along the way, I thought that it would be a fun to attempt to implement a fully functioning Tensor library on top of micrograd. I thought that my familarity with PyTorch will make this easier but having to do so without the higher-level abstractions of Python turned out much harder than expected. In this post, I hope to share some of my learnings throughout this process which forced me to think deeply about how PyTorch actually works under the hood. You might be wondering why understanding the…

related reading

• PyTorch internals : ezyang's blogblog.ezyang.com
• Making Deep Learning go Brrrr From First Principleshorace.io
• PyTorch in One Hour: From Tensors to Training Neural Networks on Multiple GPUssebastianraschka.com
• Learning PyTorch with Examples — PyTorch Tutorials 2.12.0+cu130 documentationpytorch.org
• Tensor Considered Harmfulnlp.seas.harvard.edu
• Tensors — PyTorch Tutorials 2.12.0+cu130 documentationpytorch.org
• A Recipe for Training Neural Networkskarpathy.github.io
• Tensors — PyTorch Tutorials 2.13.0+cu130 documentationdocs.pytorch.org
• Tiny TPUtinytpu.com
• Yes you should understand backprop | by Andrej Karpathy | Mediumkarpathy.medium.com
• the bug that taught me more about PyTorch than years of using it | Elana Simonelanapearl.github.io
• Transformers from scratch | peterbloem.nlpeterbloem.nl