Policy Gradients [Draft]

Material adapted from Sergey Levine’s CS294-112 Fall 2018 and Josh Achiam’s Spinning Up With RL What is a Policy Gradient and how it is useful? The goal in reinforcement learning is to find a policy with high expected return. Policy-gradient methods adjust a policy’s action probabilities using rewards from sampled trajectories. Following the gradient can improve the policy, but noisy estimates a

Material adapted from Sergey Levine’s CS294-112 Fall 2018 and Josh Achiam’s Spinning Up With RL What is a Policy Gradient and how it is useful? The goal in reinforcement learning is to find a policy with high expected return. Policy-gradient methods adjust a policy’s action probabilities using rewards from sampled trajectories. Following the gradient can improve the policy, but noisy estimates and local optima mean that reaching an optimal policy is not guaranteed. Derivation Consider a stochastic policy πθ\pi_{\theta} and a trajectory τ=(s0,a0,…,sT−1,aT−1,sT)\tau = (s_0, a_0, \ldots,…

related reading

• Part 3: Intro to Policy Optimization - Spinning Up documentationspinningup.openai.com
• Understanding Policy Gradients | John Lambertjohnwlambert.github.io
• Policy Gradient Algorithms | Lil'Loglilianweng.github.io
• Policy Gradients Part 1: The REINFORCE Estimatorfa.bianp.net
• Policy gradient methoden.wikipedia.org
• Vanilla Policy Gradient - Spinning Up documentationspinningup.openai.com
• A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
• RL_Notes__final_.pdfjubayer-ibn-hamid.github.io
• Part 1: Key Concepts in RL - Spinning Up documentationspinningup.openai.com
• [2011.01859] A Study of Policy Gradient on a Class of Exactly Solvable Modelsarxiv.org
• From REINFORCE to Dr. GRPOlancelqf.github.io
• An Updated Introduction to Reinforcement Learning | Sri's Blogsrianumakonda.com