Training VLM for CUA - Tzafon
How Tzafon trains Vision Language Models for Computer Use Agents using reinforcement learning, addressing the limitations of SFT and improving generalization across environments.
Research 2026-02-26 Training VLM for CUA How Tzafon trains Vision Language Models for Computer Use Agents using reinforcement learning, addressing the limitations of SFT and improving generalization across environments. Contributors: Nikita Khomich*, Leopold Pluto Hermansson*, David Dinucu Jianu, Ido Hakimi, Yerniyaz Nurgabylov, Noga Bregman, Simon Koser, Noah Löfquist, Mark Rogers *Core contributors Many have tried to solve the task of getting an LLM to use a computer. Until recently, these have all mostly relied on SFT. The reason for this is that it’s easy to define, e.g. just figure out th
saved by
related reading
- Illustrating Reinforcement Learning from Human Feedback (RLHF)huggingface.co
- RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
- RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackarxiv.org
- Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Modelsarxiv.org
- State of RL for reasoning LLMs | A. Weersaweers.de
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- [2511.08579] Training Language Models to Explain Their Own Computationsarxiv.org
- Language Models can Solve Computer Tasksarxiv.org
- Training Language Models to Explain Their Own Computationsarxiv.org
- Self-Adapting Language Modelsarxiv.org
- You Can Learn Tokenization End-to-End with Reinforcement Learningarxiv.org
- Reinforcement Learning for Knowledge Awareness – kalomaze's kalomazing blogkalomaze.bearblog.dev