Reverse-mode automatic differentiation from scratch, in Python
sidsite.com · 4,045 words · saved by 1 readers
The site of Sid.
Automatic differentiation is the foundation upon which deep learning frameworks lie. Deep learning models are typically trained using gradient based techniques, and autodiff makes it easy to get gradients, even from enormous, complex models. ‘Reverse-mode autodiff’ is the autodiff method used by most deep learning frameworks, due to its efficiency and accuracy. Let’s: Look at how reverse-mode autodiff works. Create a minimal autodiff framework in Python. The small autodiff framework will deal with scalars. However we will look at a method of vectorising it with NumPy. We will also look at how
related reading
- Differentiable Programming from Scratchthenumb.at
- Calculus on Computational Graphs: Backpropagation -- colah's blogcolah.github.io
- 2404.17625arxiv.org
- Matrix_Calc_for_DL.pdfsamfinlayson.com
- The Matrix Calculus You Need For Deep Learningarxiv.org
- Just know stuff. (Or, how to achieve success in a machine learning PhD.) · Patrick Kidgerkidger.site
- di.ku.dk/english/research/phd/phd-theses/2025/dissertation_Robert.pdfdi.ku.dk
- 1806.07366arxiv.org
- Ya so far this is the best introduction to neural networks from first principles...news.ycombinator.com
- torch.Tensor.backwardpytorch.org
- [2403.14606] The Elements of Differentiable Programmingarxiv.org
- Yes you should understand backprop | by Andrej Karpathy | Mediumkarpathy.medium.com