Policy Gradients Part 1: The REINFORCE Estimator
I have a dirty secret. Well, I actually have many. But one of them is that I never understood the basic algorithms behind reinforcement learning. So I plan to remedy this with a series of blog posts, where I will cover some of the basic RL algorithms, from REINFORCE to …
I have a dirty secret. Well, I actually have many. But one of them is that I never understood the basic algorithms behind reinforcement learning. So I plan to remedy this with a series of blog posts, where I will cover foundational RL algorithms, from REINFORCE to the frontier. Here's the first one, where I cover one of the oldest practical gradient estimators for reinforcement learning, the REINFORCE estimator, and show how it can get away without differentiating through the environment. Setup This is the notation I'll follow throughout the series. An agent interacts with an environment…
saved by
related reading
- A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
- A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
- State of RL for reasoning LLMs | A. Weersaweers.de
- Understanding Policy Gradients | John Lambertjohnwlambert.github.io
- Policy gradient methoden.wikipedia.org
- Policy Gradient Algorithms | Lil'Loglilianweng.github.io
- Part 3: Intro to Policy Optimization - Spinning Up documentationspinningup.openai.com
- Part 1: Key Concepts in RL - Spinning Up documentationspinningup.openai.com
- Deep Reinforcement Learning: Pong from Pixelskarpathy.github.io
- An Updated Introduction to Reinforcement Learning | Sri's Blogsrianumakonda.com
- RL_Notes__final_.pdfjubayer-ibn-hamid.github.io
- RLHF Bookrlhfbook.com