rl-for-llms.md

GitHub Gist: instantly share code, notes, and snippets.

Reinforcement Learning for Language Models Yoav Goldberg, April 2023. Why RL? With the release of the ChatGPT model and followup large language models (LLMs), there was a lot of discussion of the importance of "RLHF training", that is, "reinforcement learning from human feedback". I was puzzled for a while as to why RL (Reinforcement Learning) is better than learning from demonstrations (a.k.a supervised learning) for training language models. Shouldn't learning from demonstrations (or, in language model terminology "instruction fine tuning", learning to immitate human written answers) be suff

related reading

• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• RLHF & Post-Training Course by Nathan Lambertrlhfbook.com
• How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
• DeepSeek-R1arxiv.org
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• State of RL for reasoning LLMs | A. Weersaweers.de
• rlhfbook.com/book.pdfrlhfbook.com
• [2203.02155] Training language models to follow instructions with human feedbackarxiv.org
• Offline RL and Large Language Models - by Sergey Levinesergeylevine.substack.com
• Introduction | RLHF and Post-Training Book by Nathan Lambertrlhfbook.com
• Training language models to follow instructions with human feedback.pdfproceedings.neurips.cc
• Unsupervised Elicitation of Language Modelsarxiv.org