How not to do research - Rajan Agarwal

Lessons learned from building multiplayer world models. Built a video tokenizer with spatial attention and a dynamics model with action spaces.

How not to do research - Rajan Agarwal I don't usually share when things go wrong. Like most, my public work tends to be the stuff that worked, but I learned a lot from this project, and I want to share what I learned about the problem, and how not to do research. I tried to train a Genie-style world model that could learn to segment two players and their independent action spaces from Pong video alone, without labels or hardcoded structure. The model kept collapsing to degenerate solutions. It would always track the ball as "player 1" or even treat the score as an agent. I eventually realized

saved by

• Tasha Pais
• Yudhister Joel Kumar
• Rohan Kanti
• Ishan Mukherjee
• Vihan Tiwari
• Aaron Liu
• Nathan Chen

related reading

• The First Fully General Computer Action Model | blogsi.inc
• The Model That Dreams the Worldmoe-capital.com
• World Models | Rohit Bandarurohitbandaru.github.io
• How to train a frontier-level world modelnext-state.github.io
• World Models: Computing the Uncomputablenotboring.co
• [2509.24527] Training Agents Inside of Scalable World Modelsarxiv.org
• Genie 3: A new frontier for world modelsdeepmind.google
• MIRA - Blog postmira-wm.com
• Can Video World Models Track Unobserved World States?joonghyuk.com
• pdfopenreview.net
• H-Nets - the Past | Goomba Labgoombalab.github.io
• Efficient World Models with Context-Aware Tokenizationarxiv.org