[2006.11266] An operator view of policy gradient methods

We cast policy gradient methods as the repeated application of two operators: a policy improvement operator , which maps any policy to a better one , and a projection operator , which finds the best approximation of …

An operator view of policy gradient methods Dibya Ghosh Google Brain &Marlos C. Machado Google Brain &Nicolas Le Roux Google Brain Abstract We cast policy gradient methods as the repeated application of two operators: a policy improvement operator ℐ ℐ {\mathcal{I}} , which maps any policy π 𝜋 \pi to a better one ℐ ​ π ℐ 𝜋 {\mathcal{I}}\pi , and a projection operator 𝒫 𝒫 {\mathcal{P}} , which finds the best approximation of ℐ ​ π ℐ 𝜋 {\mathcal{I}}\pi in the set of realizable policies. We use this framework to introduce operator-based versions of well-known policy gradient methods such as R

related reading

• [1707.06347] Proximal Policy Optimization Algorithmsarxiv.org
• Part 3: Intro to Policy Optimization - Spinning Up documentationspinningup.openai.com
• Policy Gradient Algorithms | Lil'Loglilianweng.github.io
• Policy gradient methoden.wikipedia.org
• A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
• Understanding Policy Gradients | John Lambertjohnwlambert.github.io
• From REINFORCE to Dr. GRPOlancelqf.github.io
• [2011.01859] A Study of Policy Gradient on a Class of Exactly Solvable Modelsarxiv.org
• Flow Matching Policy Gradientsflowreinforce.github.io
• RL_Notes__final_.pdfjubayer-ibn-hamid.github.io
• Policy Gradients Part 1: The REINFORCE Estimatorfa.bianp.net
• Proximal Policy Optimization Algorithmsalphaxiv.org