High variance in RL (and how to measure it)

Summary

Summary Policy Gradient algorithms in RL are known to have ‘high variance’, leading to a family of algorithms and 1000 of papers across more than 10 years (Actor-Critics, PPO, DPO, GRPO), clipping tricks, normalisation tricks, learning rate warm up tricks, which all attempt to reduce this variance. To examine this variance, we’ll implement Policy Gradient from scratch and collect the gradients by pytorch hackery. Implementing Policy Gradient From Scratch The derivation of the objective function for Policy Gradient, is the following: \[J(\theta) = E_{s,a} [ \log \pi_{\theta} (a \mid s)…

saved by

• Akira Yoshiyama

related reading

• Understanding Policy Gradients | John Lambertjohnwlambert.github.io
• RL_Notes__final_.pdfjubayer-ibn-hamid.github.io
• PPO for LLMs: A Guide for Normal Peoplecameronrwolfe.substack.com
• f514cec81cb148559cf475e7426eed5e-Paper.pdfproceedings.neurips.cc
• [1707.06347] Proximal Policy Optimization Algorithmsarxiv.org
• GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com
• [1709.06560] Deep Reinforcement Learning that Mattersarxiv.org
• Policy Gradient Algorithms | Lil'Loglilianweng.github.io
• The 37 Implementation Details of Proximal Policy Optimization · The ICLR Blog Trackiclr-blog-track.github.io
• Debugging Reinforcement Learning Systemsandyljones.com
• Is Frontier Asynchronous RL Solved? — Luke J. Huangluk-huang.github.io
• Part 3: Intro to Policy Optimization - Spinning Up documentationspinningup.openai.com