Constitutional AI: RLHF On Steroids

...

AIs like GPT-4 go through several different1 types of training. First, they train on giant text corpuses in order to work at all. Later, they go through a process called “reinforcement learning through human feedback” (RLHF) which trains them to be “nice”. RLHF is why they (usually) won’t make up fake answers to your questions, tell you how to make a bomb, or rank all human races from best to worst. RLHF is hard. The usual method is to make human crowdworkers rate thousands of AI responses as good or bad, then train the AI towards the good answers and away from the bad answers. But having…

saved by

• Zi T
• Franklyn Wang

related reading

• Constitutional AI: Harmlessness from AI Feedbackarxiv.org
• Constitutional AI: Harmlessness from AI feedbackanthropic.com
• Illustrating Reinforcement Learning from Human Feedback (RLHF)huggingface.co
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• [2212.08073] Constitutional AI: Harmlessness from AI Feedbackar5iv.labs.arxiv.org
• A simple technical explanation of RLH(AI)F | Kairos.fmkairos.fm
• Video and transcript of talk on writing AI constitutionsjoecarlsmith.com
• Constitutional AI vs. RLHF vs. Deliberative Alignment — LessWronglesswrong.com
• Teaching Claude Whyalignment.anthropic.com
• Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover — LessWronglesswrong.com
• Claude's Constitutional Structure - by Zvi Mowshowitzthezvi.substack.com
• The Artificiality of Alignmentjoinreboot.org