Keep the gradient flowing
I have a dirty secret. Well, I actually have many. But one of them is that I never understood the basic algorithms behind reinforcement learning. So I plan to remedy this with a series of blog posts, where I will cover some of the basic RL algorithms, from REINFORCE to … The Cost of Differentiating Through Optimization ⊕ Differentiating through optimization is a fundamental problem in hyperparameter optimization, dataset distillation, meta-learning and optimization as a layer, to name a few. In this blog post we'll look into one of the main approaches to differentiate through optimization: unrolled differentiation. With the help of polynomials, we'll be able to derive … Faster rates under strong convexity ⊕ This blog post discusses the convergence rate of the Stochastic Gradient Descent with Stochastic Polyak Step-size (SGD-SPS) algorithm for minimizing a finite sum objective. Building upon the proof of the previous post, we show that the convergence rate can be improved to O(1/t) unde
In Part 1, we saw that the asymptotic variance of the vanilla REINFORCE estimator scales cubically ($O(T^3)$) with trajectory length $T$. In this second post, we explore how baselines systematically tame this variance. We formulate them through the classical Monte Carlo technique of control variates, showing how centering … I have a dirty secret. Well, I actually have many. But one of them is that I never understood the basic algorithms behind reinforcement learning. So I plan to remedy this with a series of blog posts, where I will cover foundational RL algorithms, from REINFORCE to the…
saved by
related reading
- Policy Gradients Part 1: The REINFORCE Estimatorfa.bianp.net
- A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
- Part 3: Intro to Policy Optimization - Spinning Up documentationspinningup.openai.com
- Policy Gradient Algorithms | Lil'Loglilianweng.github.io
- Debugging Reinforcement Learning Systemsandyljones.com
- Effortless optimization through gradient flows – Machine Learning Research Blogfrancisbach.com
- ML Mentorship: Some Q/A about RLevjang.com
- Why Momentum Really Worksdistill.pub
- Flow Matching Policy Gradientsflowreinforce.github.io
- Understanding Policy Gradients | John Lambertjohnwlambert.github.io
- Deep Reinforcement Learning: Pong from Pixelskarpathy.github.io
- Machine Learning Research Blog – Francis Bachfrancisbach.com