A simple technical explanation of RLH(AI)F | Kairos.fm

Understanding reinforcement learning from human, or AI, feedback.

Large language models (LLM) like ChatGPT or Claude are trained on a huge amount of text. This training has made LLMs really good at predicting the next word so that it says coherent things. For example, if you start with “once upon a,” the model predicts “time” as the next word, having seen this pattern many times during its training. Researchers then built on the LLM’s next-word prediction skills, training LLMs to perform tasks like answering questions or summarising text. However, training on unfiltered internet text means these models can also: Help people with harmful tasks, ranging…

saved by

• Gabby Chan
• Alex Yun
• Jackson Kozlowski

related reading

• Illustrating Reinforcement Learning from Human Feedback (RLHF)huggingface.co
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• LLM Training: RLHF and Its Alternativesmagazine.sebastianraschka.com
• RLHF | John Lambertjohnwlambert.github.io
• Introduction | RLHF and Post-Training Book by Nathan Lambertrlhfbook.com
• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• RLHF & Post-Training Course by Nathan Lambertrlhfbook.com
• How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
• Constitutional AI: Harmlessness from AI Feedbackarxiv.org
• Constitutional AI: RLHF On Steroidsastralcodexten.substack.com
• Reinforcement learning from human feedback - Wikipediaen.wikipedia.org
• [2309.00267] RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackarxiv.org