[2505.10831] Creating General User Models from Computer Use
Abstract:Human-computer interaction has long imagined technology that understands us-from our preferences and habits, to the timing and purpose of our everyday actions. Yet current user models remain fragmented, narrowly tailored to specific apps, and incapable of the flexible reasoning required to fulfill these visions. This paper presents an architecture for a general user model (GUM) that learns about you by observing any interaction you have with your computer. The GUM takes as input any unstructured observation of a user (e.g., device screenshots) and constructs confidence-weighted propositions that capture user knowledge and preferences. GUMs can infer that a user is preparing for a wedding they're attending from messages with a friend. Or recognize that a user is struggling with a collaborator's feedback on a draft by observing multiple stalled edits and a switch to reading related work. GUMs introduce an architecture that infers new propositions about a user from multimodal observations, retrieves related propositions for context, and continuously revises existing propositions. To illustrate the breadth of applications that GUMs enable, we demonstrate how they augment chat-based assistants with context, manage OS notifications to selectively surface important information, and enable interactive agents that adapt to preferences across apps. We also instantiate proactive assistants (GUMBOs) that discover and execute useful suggestions on a user's behalf using their GUM. In our evaluations, we find that GUMs make calibrated and accurate inferences about users, and that assistants built on GUMs proactively identify and perform actions that users wouldn't think to request explicitly. Altogether, GUMs introduce methods that leverage multimodal models to understand unstructured context, enabling long-standing visions of HCI and entirely new interactive systems that anticipate user needs.
# link_1j62hqbosxb.pdf ## Metadata - PDFFormatVersion=1.7 - Language=en - IsLinearized=false - IsAcroFormPresent=false - IsXFAPresent=false - IsCollectionPresent=false - IsSignaturesPresent=false - Author=Omar Shaikh; Shardul Sapkota; Shan Rizvi; Eric Horvitz; Joon Sung Park; Diyi Yang; Michael S. Bernstein - CreationDate=D:20250923013652+00'00' - Creator=arXiv GenPDF (tex2pdf:) - Custom.DOI=https://doi.org/10.48550/arXiv.2505.10831 - Custom.License=http://arxiv.org/licenses/nonexclusive-distrib/1.0/ - Custom.PTEX.Fullbanner=This is pdfTeX, Version 3.141592653-2.6-1.40.28 (TeX Live 2025) kpath
Explore this link on the map →saved by
related reading
- [2505.10831] Creating General User Models from Computer Usearxiv.org
- Guardian Angels: LLM Personalization for Productivity and Security · Gwern.netgwern.net
- Co-Creating Life Goals from Everyday Computer Usearxiv.org
- The Persona Selection Model: Why AI Assistants might Behave like Humansalignment.anthropic.com
- trees are harlequins, words are harlequins - the voidnostalgebraist.tumblr.com
- [2603.05923] Learning Next Action Predictors from Human-Computer Interactionarxiv.org
- Interaction Models: A Scalable Approach to Human-AI Collaboration - Thinking Machines Labthinkingmachines.ai
- Language Models, World Models, and Human Model-Buildinglingo.csail.mit.edu
- AI Horseless Carriages | koomen.devkoomen.dev
- [2304.03442] Generative Agents: Interactive Simulacra of Human Behaviorarxiv.org
- Using Artificial Intelligence to Augment Human Intelligencedistill.pub
- Patterns for Building LLM-based Systems & Productseugeneyan.com