Direct Preference Optimization (DPO) | by João Lages | Medium

In May 2023, this algorithm was introduced in Direct Preference Optimization: Your Language Model is Secretly a Reward Model. DPO comes as a direct alternative to Reinforcement Learning from Human Feedback (RLHF) and has been gaining a lot of popularity since it does not require a reward model. It was recently used in Zephyr, the best 7B language model at the time of writing. This method is applied to generative language models (LMs), such as GPT, Llama, Zephyr, and T5. The objective of DPO is the same as RLHF: improve the alignment of language models to human preferences. DPO is applied to preference data, which basically consists of a dataset of triplets (prompt, chosen answer, rejected answer). In other words, for each prompt, there is a better response and a worse response. This type of data is also collected for RLHF, with the objective to train a reward model, that is later used to train the model with reinforcement learning. In DPO, there is no reinforcement learning, and the mo

In May 2023, this algorithm was introduced in Direct Preference Optimization: Your Language Model is Secretly a Reward Model. DPO comes as a direct alternative to Reinforcement Learning from Human Feedback (RLHF) and has been gaining a lot of popularity since it does not require a reward model. It was recently used in Zephyr, the best 7B language model at the time of writing. This method is applied to generative language models (LMs), such as GPT, Llama, Zephyr, and T5. The objective of DPO is the same as RLHF: improve the alignment of language models to human preferences. DPO is applied to pr