The First Fully General Computer Action Model | blog
We trained a model on our 11-million-hour video dataset. Our model can explore complex websites, complete multi-action CAD modeling sequences, and drive a car in the real world, all at 30 FPS.
We designed FDM-1, a foundation model for computer use. FDM-1 is trained on videos from a portion of our 11-million-hour screen recording dataset, which we labeled using an inverse dynamics model that we trained. Our video encoder can compress almost 2 hours of 30 FPS video in only 1M tokens. FDM-1 is the first model with the long-context training needed to become a coworker for CAD, finance, engineering, and eventually ML research, and it consistently improves with scale. It trains and infers directly on video instead of screenshots and can learn unsupervised from the entirety of the internet
saved by
- Pranav
- Justin Wang
- Elizabeth Qiu
- Rajan Agarwal
- Tasha Pais
- Ratan Kaliani
- HudZah
- Emma Guo
- Eshaan Moorjani
- Rishi Kothari
- Neel Redkar
- Aaron Pham
related reading
- GEN-1.5: Embodied Foundation Models are One-Shot Learners - Generalist AIgeneralistai.com
- Generalist - GEN-0 / Embodied Foundation Models That Scale with Physical Interactiongeneralistai.com
- Causal Video Models Are Data-Efficient Robot Policy Learners | Rhoda AIrhoda.ai
- ACT-1: A Robot Foundation Model Trained on Zero Robot Data | Sunday Robotics | The helpful robotics companysunday.ai
- 1X World Model | From Video to Action: A New Way Robots Learn1x.tech
- RTFM: A Real-Time Frame Model | World Labsworldlabs.ai
- A Steerable Model with Emergent Capabilitiespi.website
- Scaling Video Pretraining with Imagination Modelsinductionlabs.com
- How not to do research - Rajan Agarwalrajan.sh
- Training VLM for CUA — Tzafontzafon.ai
- [2206.11795] Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videosarxiv.org
- GameNGengamengen.github.io