[2606.06614] Re-Centering Humans in LLM Personalization

Abstract:Despite growing interest, most evaluations of large language models' (LLMs') personalization abilities have relied on synthetic data. It remains unclear how well current personalization systems work for real users. In this paper, we study the gap in LLM personalization performance when using synthetic versus human data. We collect human conversations (550 conversations) and judgments across three stages of personalization: extracting user attributes from conversations (5,949 judgments), pairing relevant attributes with new prompts (11,919), and incorporating relevant attributes into a personalized response (1,101). Incorporating human data reveals system limitations at each stage. Models struggle to extract attributes from human conversations, disagree with human judgments on relevant attributes, and generate personalized responses that humans judge no better than generic responses (though that LLM judges widely rate as better). We introduce two lightweight training-based interventions that shift automated personalization evaluation closer to human data in our first two stages. However, in our third stage we find that learned reward models achieve only modest correlation with human ratings, suggesting that human-aligned personalization quality judgments are difficult to model directly. Our collected data provides a foundation for studying how models should extract, select, and incorporate user information in ways that humans find useful.

Abstract:Despite growing interest, most evaluations of large language models' (LLMs') personalization abilities have relied on synthetic data. It remains unclear how well current personalization systems work for real users. In this paper, we study the gap in LLM personalization performance when using synthetic versus human data. We collect human conversations (550 conversations) and judgments across three stages of personalization: extracting user attributes from conversations (5,949 judgments), pairing relevant attributes with new prompts (11,919), and incorporating relevant attributes into a

saved by

• surya
• 7165

related reading

• [2606.06614] Re-Centering Humans in LLM Personalizationarxiv.org
• Guardian Angels: LLM Personalization for Productivity and Security · Gwern.netgwern.net
• The Persona Selection Model: Why AI Assistants might Behave like Humansalignment.anthropic.com
• Patterns for Building LLM-based Systems & Productseugeneyan.com
• FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real Usersarxiv.org
• The bitter lesson of LLM evalsparsed.com
• Your AI Product Needs Evals – Hamel's Blog - Hamel Husainhamel.dev
• llm assistant personas seem increasingly incoherent (some subjective observations) — LessWronglesswrong.com
• LLM Evaluation doesn't need to be complicatedphilschmid.de
• [2506.13023] A Practical Guide for Evaluating LLMs and LLM-Reliant Systemsarxiv.org
• Discovering Language Model Behaviors with Model-Written Evaluations — LessWronglesswrong.com
• LLM evaluation: a beginner's guideevidentlyai.com