FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real Users

arxiv.org · 11,301 words · saved by 1 readers

Effective personalization of LLMs is critical for a broad range of user-interfacing applications such as virtual assistants and content curation. Inspired by the strong in-context capabilities of LLMs, we propose few-shot preference optimization (FSPO), an algorithm for LLM personalization that reframes reward modeling as a meta-learning problem. Under FSPO, an LLM learns to quickly infer a personalized reward function for a user via a few labeled preferences. FSPO also utilizes user description rationalization (RAT) to encourage better reward modeling and instruction following, recovering performance with the oracle user description. Since real-world preference data is challenging to collect at scale, we propose careful design choices to construct synthetic preference datasets for personalization, generating over 1M synthetic personalized preferences using publicly available LLMs. To successfully transfer from synthetic data to real users, we find it crucial for the data to exhibit bo

\reportnumber \correspondingauthor anikait@stanford.edu. Project Website: https://fewshot-preference-optimization.github.io/ FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real Users Anikait Singh Stanford University Sheryl Hsu Stanford University Kyle Hsu Stanford University Eric Mitchell Stanford University OpenAI Stefano Ermon Stanford University Tatsunori Hashimoto Stanford University Archit Sharma Stanford University Google DeepMind Chelsea Finn Stanford University Abstract Effective personalization of LLMs is critical for a broad range of user-interfacing applicatio

FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real Users

related reading