Keep the gradient flowing

I have a dirty secret. Well, I actually have many. But one of them is that I never understood the basic algorithms behind reinforcement learning. So I plan to remedy this with a series of blog posts, where I will cover some of the basic RL algorithms, from REINFORCE to … The Cost of Differentiating Through Optimization ⊕ Differentiating through optimization is a fundamental problem in hyperparameter optimization, dataset distillation, meta-learning and optimization as a layer, to name a few. In this blog post we'll look into one of the main approaches to differentiate through optimization: unrolled differentiation. With the help of polynomials, we'll be able to derive … Faster rates under strong convexity ⊕ This blog post discusses the convergence rate of the Stochastic Gradient Descent with Stochastic Polyak Step-size (SGD-SPS) algorithm for minimizing a finite sum objective. Building upon the proof of the previous post, we show that the convergence rate can be improved to O(1/t) unde

In Part 1, we saw that the asymptotic variance of the vanilla REINFORCE estimator scales cubically ($O(T^3)$) with trajectory length $T$. In this second post, we explore how baselines systematically tame this variance. We formulate them through the classical Monte Carlo technique of control variates, showing how centering … I have a dirty secret. Well, I actually have many. But one of them is that I never understood the basic algorithms behind reinforcement learning. So I plan to remedy this with a series of blog posts, where I will cover foundational RL algorithms, from REINFORCE to the…

saved by

• Benedict Neo

related reading

• Policy Gradient Algorithms | Lil'Loglilianweng.github.io
• A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
• Policy Gradients Part 1: The REINFORCE Estimatorfa.bianp.net
• Policy gradient methoden.wikipedia.org
• Why Momentum Really Worksdistill.pub
• Part 3: Intro to Policy Optimization - Spinning Up documentationspinningup.openai.com
• Understanding Policy Gradients | John Lambertjohnwlambert.github.io
• Stochastic gradient descent - Wikipediaen.m.wikipedia.org
• NeurIPS-2021-understanding-end-to-end-model-based-reinforcement-learning-methods-as-implicit-parameterization-Supplemental.pdflis.csail.mit.edu
• Deep Reinforcement Learning: Pong from Pixelskarpathy.github.io
• [1707.06347] Proximal Policy Optimization Algorithmsarxiv.org
• [1609.04747] An overview of gradient descent optimization algorithmsarxiv.org