[2606.06614] Re-Centering Humans in LLM Personalization

Abstract:Despite growing interest, most evaluations of large language models' (LLMs') personalization abilities have relied on synthetic data. It remains unclear how well current personalization systems work for real users. In this paper, we study the gap in LLM personalization performance when using synthetic versus human data. We collect human conversations (550 conversations) and judgments across three stages of personalization: extracting user attributes from conversations (5,949 judgments), pairing relevant attributes with new prompts (11,919), and incorporating relevant attributes into a personalized response (1,101). Incorporating human data reveals system limitations at each stage. Models struggle to extract attributes from human conversations, disagree with human judgments on relevant attributes, and generate personalized responses that humans judge no better than generic responses (though that LLM judges widely rate as better). We introduce two lightweight training-based interventions that shift automated personalization evaluation closer to human data in our first two stages. However, in our third stage we find that learned reward models achieve only modest correlation with human ratings, suggesting that human-aligned personalization quality judgments are difficult to model directly. Our collected data provides a foundation for studying how models should extract, select, and incorporate user information in ways that humans find useful.

Re-Centering Humans in LLM Personalization Lechen Zhang† Jiarui Liu♯ Tal August† † ♯ University of Illinois Urbana-Champaign Carnegie Mellon University {lechenz3, taugust}@illinois.edu jiaruil5@andrew.cmu.edu Abstract…

saved by

• Vincent Huang

related reading

• [2605.21739] AttuneBench: A Conversation-Based Benchmark for LLM Emotional Intelligencearxiv.org
• [2608.19549] Generating Diverse Personas for User Simulators to Test Interview Dialogue Systemsarxiv.org
• [2506.06266] Cartridges: Lightweight and general-purpose long context representations via self-studyarxiv.org
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• [2604.16812] Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• HumanLM: Simulating Users with State Alignment Beats Response Imitationarxiv.org
• [2409.19759] Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMsarxiv.org
• Probing Persona-Dependent Preferences in Language Modelsarxiv.org
• [2508.19227] Generative Interfaces for Language Modelsarxiv.org
• Quantifying the Utility of User Simulators for Building Collaborative LLM Assistantsarxiv.org
• Evaluating LLMs for Impersonationarxiv.org