✳flâneur — a map of the web's best reading

Reward Modeling | RLHF Book by Nathan Lambert

rlhfbook.com · 6,918 words · saved by 1 readers

Reward models are core to the modern approach to RLHF. Reward models broadly have been used extensively in reinforcement learning research as a proxy for environment rewards [1]. The practice is closely related to inverse reinforcement learning, where the problem is to approximate an agent’s reward function given trajectories of behavior [2], and other areas of deep reinforcement learning. Reward models were proposed, in their modern form, as a tool for studying the value alignment problem [3]. The most common reward model predicts the probability that a piece of text was close to a “preferred” piece of text from the training comparisons. Later in this section we also compare these to Outcome Reward Models (ORMs) that predict the probability that a completion results in a correct answer or a Process Reward Model (PRM) that assigns a score to each step in reasoning. When not indicated, the reward models mentioned are those predicting preference between text. There are two popular expres

--> RLHF Book --> Reward Modeling | RLHF and Post-Training Book by Nathan Lambert Reinforcement Learning from Human Feedback A short introduction to RLHF and post-training focused on language models. Nathan Lambert Lecture 2: IFT, Reward Modeling, Rejection Sampling (Chap. 4, 5, & 9) Reward Modeling Reward models are core to the modern approach to RLHF by being where the complex human preferences are learned. They are what enable our models to learn from hard-to-specify signals. They compress complex features in the data into a representation that can be used in downstream training – a sort of

Explore this link on the map →

related reading