High variance in RL (and how to measure it)

Summary

Summary Policy Gradient algorithms in RL are known to have ‘high variance’, leading to a family of algorithms and 1000 of papers across more than 10 years (Actor-Critics, PPO, DPO, GRPO), clipping tricks, normalisation tricks, learning rate warm up tricks, which all attempt to reduce this variance. To examine this variance, we’ll implement Policy Gradient from scratch and collect the gradients by pytorch hackery. Implementing Policy Gradient From Scratch The derivation of the objective function for Policy Gradient, is the following: \[J(\theta) = E_{s,a} [ \log \pi_{\theta} (a \mid s)…

saved by

• Akira Yoshiyama

related reading

• Understanding Policy Gradients | John Lambertjohnwlambert.github.io
• PPO for LLMs: A Guide for Normal Peoplecameronrwolfe.substack.com
• RL_Notes__final_.pdfjubayer-ibn-hamid.github.io
• Policy Gradient Algorithms | Lil'Loglilianweng.github.io
• [1707.06347] Proximal Policy Optimization Algorithmsarxiv.org
• [1709.06560] Deep Reinforcement Learning that Mattersarxiv.org
• GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com
• The 37 Implementation Details of Proximal Policy Optimization · The ICLR Blog Trackiclr-blog-track.github.io
• State of RL for reasoning LLMs | A. Weersaweers.de
• [2602.19362] LLMs Can Learn to Reason Via Off-Policy RLarxiv.org
• Deep RL Bootcamp - Lecturessites.google.com
• Is Frontier Asynchronous RL Solved? — Luke J. Huangluk-huang.github.io