NAP — Next-Action Prediction

A system that records user screen activity, labels it into action descriptions, and trains LLMs to predict next user actions.

Learning Next Action Predictors from Human-Computer Interaction Omar Shaikh1, Valentin Teutschbein2*, Kanishk Gandhi1*, Yikun Chi1, Nick Haber1, Thomas Robinson1, Nilam Ram1, Byron Reeves1, Sherry Yang1,3, Michael Bernstein1, Diyi Yang1 1Stanford University 2Hasso Plattner Institute 3New York University Read the Paper GitHub (coming soon!) What Will You Do Next? Language models today are hopelessly restricted to seeing us through a narrow keyhole. They see our prompts, and they construct memories to make sense of them. But they know nothing of what brought us to them in the first…

saved by

• Caden Juang

related reading

• [2603.05923] Learning Next Action Predictors from Human-Computer Interactionarxiv.org
• [2505.10831] Creating General User Models from Computer Usearxiv.org
• The First Fully General Computer Action Model | blogsi.inc
• Mind Mapper: Modeling and Predicting Behavioral Patterns from Everyday Conversations with Wearable AI Systems and LLMs – MIT Media Labmedia.mit.edu
• The Persona Selection Model: Why AI Assistants might Behave like Humansalignment.anthropic.com
• [2505.10831] Creating General User Models from Computer Usearxiv.org
• More problemscursor.com
• Tiny Interaction Models - Rajan Agarwalrajan.sh
• LLM Powered Autonomous Agents | Lil'Loglilianweng.github.io
• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activationstransformer-circuits.pub
• Behavior Latticing: Inferring User Motivations from Unstructured Interactionsarxiv.org
• The Shape of AI | UX Patterns for Artificial Intelligence Designshapeof.ai