NAP — Next-Action Prediction

A system that records user screen activity, labels it into action descriptions, and trains LLMs to predict next user actions.

Learning Next Action Predictors from Human-Computer Interaction Omar Shaikh1, Valentin Teutschbein2*, Kanishk Gandhi1*, Yikun Chi1, Nick Haber1, Thomas Robinson1, Nilam Ram1, Byron Reeves1, Sherry Yang1,3, Michael Bernstein1, Diyi Yang1 1Stanford University 2Hasso Plattner Institute 3New York University Read the Paper GitHub (coming soon!) What Will You Do Next? Language models today are hopelessly restricted to seeing us through a narrow keyhole. They see our prompts, and they construct memories to make sense of them. But they know nothing of what brought us to them in the first…

saved by

• Caden Juang

related reading

• [2603.05923] Learning Next Action Predictors from Human-Computer Interactionarxiv.org
• [2505.10831] Creating General User Models from Computer Usearxiv.org
• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activationstransformer-circuits.pub
• [2505.10831] Creating General User Models from Computer Usearxiv.org
• Tiny Interaction Models - Rajan Agarwalrajan.sh
• Co-Creating Life Goals from Everyday Computer Usearxiv.org
• The Near Future of AI is Action-Drivenjmcdonnell.substack.com
• [2602.10556] LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transferarxiv.org
• [2206.11795] Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videosarxiv.org
• user_interactions.pdfself-distillation.github.io
• Scaling Activation Oracles to Trillion-Parameter Modelstransluce.org
• [2512.11949] Neural Chameleons: Language Models Can Learn to Hide Their Thoughts from Unseen Activation Monitorsarxiv.org