Reverse-mode automatic differentiation from scratch, in Python

The site of Sid.

Automatic differentiation is the foundation upon which deep learning frameworks lie. Deep learning models are typically trained using gradient based techniques, and autodiff makes it easy to get gradients, even from enormous, complex models. ‘Reverse-mode autodiff’ is the autodiff method used by most deep learning frameworks, due to its efficiency and accuracy. Let’s: Look at how reverse-mode autodiff works. Create a minimal autodiff framework in Python. The small autodiff framework will deal with scalars. However we will look at a method of vectorising it with NumPy. We will also look at how

related reading

• Differentiable Programming from Scratchthenumb.at
• Calculus on Computational Graphs: Backpropagation -- colah's blogcolah.github.io
• Gradients, Backprop, and Derivatives in Tensorflow [Draft]blog.sdan.net
• Yes you should understand backprop | by Andrej Karpathy | Mediumkarpathy.medium.com
• Making Deep Learning go Brrrr From First Principleshorace.io
• Matrix_Calc_for_DL.pdfsamfinlayson.com
• CS231n Deep Learning for Computer Visioncs231n.github.io
• The Matrix Calculus You Need For Deep Learningarxiv.org
• Learning PyTorch with Examples — PyTorch Tutorials 2.12.0+cu130 documentationpytorch.org
• CS231n Deep Learning for Computer Visioncs231n.github.io
• Understanding the Neural Tangent Kernel – EigenTaleseigentales.com
• Transformers from scratch | peterbloem.nlpeterbloem.nl