Harsh Bhatt on X: "The Math of RL: From Policy Gradients to PPO and GRPO" / X

The Math of RL: From Policy Gradients to PPO and GRPO

Reinforcement Learning is becoming very popular since last few years especially in LLMs with the techniques like RLHF (Reinforcement Learning Human Feedback) and RLVR (Reinforcement Learning Verifiable Rewards). RL got the very high attention when DeepMind built and trained AlphaGo, but there came a period when shift moved towards unsupervised technique adversarial loss with GANs, semi-supervised learning with transformers, but as LLMs uplifted the entire deep learning, researchers looked backed into RL. Because of RL we can able to align LLMs using RLHF and make them expert in skills like…

saved by

• Carlos Guerrero

related reading

• Policy Gradient Algorithms | Lil'Loglilianweng.github.io
• State of RL for reasoning LLMs | A. Weersaweers.de
• A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• A Guide to Reinforcement Learning Post-Training for LLMs: PPO, DPO, GRPO, and Beyondhuggingface.co
• From REINFORCE to Dr. GRPOlancelqf.github.io
• RLHF Bookrlhfbook.com
• PPO for LLMs: A Guide for Normal Peoplecameronrwolfe.substack.com
• RLHF | John Lambertjohnwlambert.github.io
• Illustrating Reinforcement Learning from Human Feedback (RLHF)huggingface.co
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com