Harsh Bhatt on X: "The Math of RL: From Policy Gradients to PPO and GRPO" / X

The Math of RL: From Policy Gradients to PPO and GRPO

Reinforcement Learning is becoming very popular since last few years especially in LLMs with the techniques like RLHF (Reinforcement Learning Human Feedback) and RLVR (Reinforcement Learning Verifiable Rewards). RL got the very high attention when DeepMind built and trained AlphaGo, but there came a period when shift moved towards unsupervised technique adversarial loss with GANs, semi-supervised learning with transformers, but as LLMs uplifted the entire deep learning, researchers looked backed into RL. Because of RL we can able to align LLMs using RLHF and make them expert in skills like…

saved by

• Carlos Guerrero

related reading

• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• State of RL for reasoning LLMs | A. Weersaweers.de
• Policy Gradient Algorithms | Lil'Loglilianweng.github.io
• RLHF Bookrlhfbook.com
• PPO for LLMs: A Guide for Normal Peoplecameronrwolfe.substack.com
• Deep Reinforcement Learning: Pong from Pixelskarpathy.github.io
• GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com
• From REINFORCE to Dr. GRPOlancelqf.github.io
• Why GRPO is Important and How it Worksghost.oxen.ai
• Part 1: Key Concepts in RL - Spinning Up documentationspinningup.openai.com
• A Guide to Reinforcement Learning Post-Training for LLMs: PPO, DPO, GRPO, and Beyondhuggingface.co
• Understanding Policy Gradients | John Lambertjohnwlambert.github.io