[2403.14606] The Elements of Differentiable Programming
Abstract:Artificial intelligence has recently experienced remarkable advances, fueled by large models, vast datasets, accelerated hardware, and, last but not least, the transformative power of differentiable programming. This new programming paradigm enables end-to-end differentiation of complex computer programs (including those with control flows and data structures), making gradient-based optimization of program parameters possible. As an emerging paradigm, differentiable programming builds upon several areas of computer science and applied mathematics, including automatic differentiation, graphical models, optimization and statistics. This book presents a comprehensive review of the fundamental concepts useful for differentiable programming. We adopt two main perspectives, that of optimization and that of probability, with clear analogies between the two. Differentiable programming is not merely the differentiation of programs, but also the thoughtful design of programs intended for differentiation. By making programs differentiable, we inherently introduce probability distributions over their execution, providing a means to quantify the uncertainty associated with program outputs.
The Elements of Differentiable Programming Mathieu Blondel Google DeepMind mblondel@google.com arXiv:2403.14606v4 [cs.LG] 3 Aug 2026 Vincent Roulet…
saved by
related reading
- Differentiable Programming from Scratchthenumb.at
- 2404.17625arxiv.org
- Book: Alice’s Adventures in a differentiable wonderlandsscardapane.it
- Alice's adventures in a differentiable wonderlandnews.ycombinator.com
- Calculus on Computational Graphs: Backpropagation -- colah's blogcolah.github.io
- The Little Book of Deep Learningfleuret.org
- Neural Networks, Types, and Functional Programming -- colah's blogcolah.github.io
- Ya so far this is the best introduction to neural networks from first principles...news.ycombinator.com
- 342285bb2a8cadef22f667eeb6a63732-Paper.pdfproceedings.neurips.cc
- Understanding Deep Learningudlbook.github.io
- Just know stuff. (Or, how to achieve success in a machine learning PhD.) · Patrick Kidgerkidger.site
- DiffTaichi: Differentiable Programming for Physical Simulationarxiv.org