[2507.06187] The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gains

YOU make open access possible! Tell us why you support #openaccess and give to arXiv this week to help keep science open for all. Help | Advanced Search arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs. arXiv Operational Status

[2507.06187] The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gains Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Artificial Intelligence arXiv:2507.06187 (cs) [Submitted on 8 Jul 2025] Title: The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gains Authors: Scott Geng , Hamish Ivison , Chun-Liang Li , Maarten Sap , Jerry Li , Ranjay Krishna , Pang Wei Koh View a PDF of the paper titled The Delta Learning Hypothesis: Prefe

related reading

• [2507.06187] The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gainsarxiv.org
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
• Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Dataalignment.anthropic.com
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• Automated Weak-to-Strong Researcheralignment.anthropic.com
• RLHF & Post-Training Course by Nathan Lambertrlhfbook.com
• The Scaling Hypothesis · Gwern.netgwern.net
• The Only Important Technology Is The Internet - Kevin Lukevinlu.ai
• [2605.11134] Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Trainingarxiv.org
• Andrej Karpathy — AGI is still a decade awaydwarkesh.com
• [2606.12360] Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signalarxiv.org