[2505.10831] Creating General User Models from Computer Use

Abstract:Human-computer interaction has long imagined technology that understands us-from our preferences and habits, to the timing and purpose of our everyday actions. Yet current user models remain fragmented, narrowly tailored to specific apps, and incapable of the flexible reasoning required to fulfill these visions. This paper presents an architecture for a general user model (GUM) that learns about you by observing any interaction you have with your computer. The GUM takes as input any unstructured observation of a user (e.g., device screenshots) and constructs confidence-weighted propositions that capture user knowledge and preferences. GUMs can infer that a user is preparing for a wedding they're attending from messages with a friend. Or recognize that a user is struggling with a collaborator's feedback on a draft by observing multiple stalled edits and a switch to reading related work. GUMs introduce an architecture that infers new propositions about a user from multimodal observations, retrieves related propositions for context, and continuously revises existing propositions. To illustrate the breadth of applications that GUMs enable, we demonstrate how they augment chat-based assistants with context, manage OS notifications to selectively surface important information, and enable interactive agents that adapt to preferences across apps. We also instantiate proactive assistants (GUMBOs) that discover and execute useful suggestions on a user's behalf using their GUM. In our evaluations, we find that GUMs make calibrated and accurate inferences about users, and that assistants built on GUMs proactively identify and perform actions that users wouldn't think to request explicitly. Altogether, GUMs introduce methods that leverage multimodal models to understand unstructured context, enabling long-standing visions of HCI and entirely new interactive systems that anticipate user needs.

[2505.10831] Creating General User Models from Computer Use --> Computer Science > Human-Computer Interaction arXiv:2505.10831 (cs) [Submitted on 16 May 2025 ( v1 ), last revised 21 Sep 2025 (this version, v3)] Title: Creating General User Models from Computer Use Authors: Omar Shaikh , Shardul Sapkota , Shan Rizvi , Eric Horvitz , Joon Sung Park , Diyi Yang , Michael S. Bernstein View a PDF of the paper titled Creating General User Models from Computer Use, by Omar Shaikh and 6 other authors View PDF HTML (experimental) Abstract: Human-computer interaction has long imagined technology that un

saved by

• Caden Juang
• Ishaan Panigrahi

related reading

• [2505.10831] Creating General User Models from Computer Usearxiv.org
• Co-Creating Life Goals from Everyday Computer Usearxiv.org
• Interaction Models: A Scalable Approach to Human-AI Collaboration - Thinking Machines Labthinkingmachines.ai
• The Persona Selection Model: Why AI Assistants might Behave like Humansalignment.anthropic.com
• The First Fully General Computer Action Model | blogsi.inc
• [2603.05923] Learning Next Action Predictors from Human-Computer Interactionarxiv.org
• Creating General User Models from Computer Usearxiv.org
• trees are harlequins, words are harlequins - the voidnostalgebraist.tumblr.com
• Using Artificial Intelligence to Augment Human Intelligencedistill.pub
• AI Horseless Carriages | koomen.devkoomen.dev
• Fara1.5 - A family of frontier computer use agent models - Microsoft Researchmicrosoft.com
• Language Models can Solve Computer Tasksarxiv.org