[2006.11266] An operator view of policy gradient methods

We cast policy gradient methods as the repeated application of two operators: a policy improvement operator , which maps any policy to a better one , and a projection operator , which finds the best approximation of …

An operator view of policy gradient methods Dibya Ghosh Google Brain &Marlos C. Machado Google Brain &Nicolas Le Roux Google Brain Abstract We cast policy gradient methods as the repeated application of two operators: a policy improvement operator ℐ ℐ {\mathcal{I}} , which maps any policy π 𝜋 \pi to a better one ℐ ​ π ℐ 𝜋 {\mathcal{I}}\pi , and a projection operator 𝒫 𝒫 {\mathcal{P}} , which finds the best approximation of ℐ ​ π ℐ 𝜋 {\mathcal{I}}\pi in the set of realizable policies. We use this framework to introduce operator-based versions of well-known policy gradient methods such as R

related reading

• Policy Gradient Algorithms | Lil'Loglilianweng.github.io
• Part 3: Intro to Policy Optimization - Spinning Up documentationspinningup.openai.com
• A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
• An Updated Introduction to Reinforcement Learning | Sri's Blogsrianumakonda.com
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• [1707.06347] Proximal Policy Optimization Algorithmsarxiv.org
• Part 1: Key Concepts in RL - Spinning Up documentationspinningup.openai.com
• The 37 Implementation Details of Proximal Policy Optimization · The ICLR Blog Trackiclr-blog-track.github.io
• RLHF Bookrlhfbook.com
• Policy Gradient with PyTorchhuggingface.co
• State of RL for reasoning LLMs | A. Weersaweers.de
• From REINFORCE to Dr. GRPOlancelqf.github.io