NeurIPS Poster $\beta$-DPO: Direct Preference Optimization with Dynamic $\beta$

The NeurIPS Logo above may be used on presentations. Right-click and choose download. It is a vector graphic and may be used at any scale.

Direct Preference Optimization (DPO) has emerged as a compelling approach for training Large Language Models (LLMs) to adhere to human preferences. However, the performance of DPO is sensitive to the fine-tuning of its trade-off parameter $\beta$, as well as to the quality of the preference data. We analyze the impact of $\beta$ and data quality on DPO, uncovering that optimal $\beta$ values vary with the informativeness of pairwise data. Addressing the limitations of static $\beta$ values, we introduce a novel framework that dynamically calibrates $\beta$ at the batch level, informed by…

saved by

• Clem Yang

related reading

• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• Fine-tune Llama 2 with DPOhuggingface.co
• Preference Tuning LLMs with Direct Preference Optimization Methodshuggingface.co
• Modifying Large Language Model Post-Training for Diverse Creative Writingarxiv.org
• Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studyarxiv.org
• [2507.19457] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningarxiv.org
• Probe-Based Data Attribution: Surfacing and Mitigating Undesirable Behaviors in LLM Post-Traininggoodfire.ai
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• [2507.06187] The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gainsarxiv.org
• [2605.11134] Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Trainingarxiv.org
• Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencesarxiv.org