Part 3: Intro to Policy Optimization — Spinning Up documentation

In this section, we’ll discuss the mathematical foundations of policy optimization algorithms, and connect the material to sample code. We will cover three key results in the theory of policy gradients: In the end, we’ll tie those results together and describe the advantage-based expression for the policy gradient—the version we use in our Vanilla Policy Gradient implementation. Here, we consider the case of a stochastic, parameterized policy, . We aim to maximize the expected return . For the purposes of this derivation, we’ll take to give the finite-horizon undiscounted return, but the derivation for the infinite-horizon discounted return setting is almost identical. We would like to optimize the policy by gradient ascent, eg The gradient of policy performance, , is called the policy gradient, and algorithms that optimize the policy this way are called policy gradient algorithms. (Examples include Vanilla Policy Gradient and TRPO. PPO is often referred to as a policy gradient algori

Part 3: Intro to Policy Optimization - Spinning Up documentation --> Docs >> Part 3: Intro to Policy Optimization Edit on GitHub Part 3: Intro to Policy Optimization ¶ Table of Contents Part 3: Intro to Policy Optimization Deriving the Simplest Policy Gradient Implementing the Simplest Policy Gradient Expected Grad-Log-Prob Lemma Don’t Let the Past Distract You Implementing Reward-to-Go Policy Gradient Baselines in Policy Gradients Other Forms of the Policy Gradient Recap In this section, we’ll discuss the mathematical foundations of policy optimization algorithms, and connect the

saved by

• Claire Wang
• Michael Frew
• Vihaan Sondhi
• Vincent Cheng
• Emil Ryd
• Julian H

related reading

• Vanilla Policy Gradient - Spinning Up documentationspinningup.openai.com
• Understanding Policy Gradients | John Lambertjohnwlambert.github.io
• Policy Gradient Algorithms | Lil'Loglilianweng.github.io
• Deep Reinforcement Learning: Pong from Pixelskarpathy.github.io
• From REINFORCE to Dr. GRPOlancelqf.github.io
• Part 1: Key Concepts in RL - Spinning Up documentationspinningup.openai.com
• The 37 Implementation Details of Proximal Policy Optimization · The ICLR Blog Trackiclr-blog-track.github.io
• Policy gradient methoden.wikipedia.org
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• Policy Gradient with PyTorchhuggingface.co
• Policy Gradients Part 1: The REINFORCE Estimatorfa.bianp.net
• A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io