How not to do research - Rajan Agarwal

Lessons learned from building multiplayer world models. Built a video tokenizer with spatial attention and a dynamics model with action spaces.

How not to do research - Rajan Agarwal I don't usually share when things go wrong. Like most, my public work tends to be the stuff that worked, but I learned a lot from this project, and I want to share what I learned about the problem, and how not to do research. I tried to train a Genie-style world model that could learn to segment two players and their independent action spaces from Pong video alone, without labels or hardcoded structure. The model kept collapsing to degenerate solutions. It would always track the ball as "player 1" or even treat the score as an agent. I eventually realized

related reading

• The First Fully General Computer Action Model | blogsi.inc
• World Models | Rohit Bandarurohitbandaru.github.io
• World Models: Computing the Uncomputablenotboring.co
• pdfopenreview.net
• Efficient World Models with Context-Aware Tokenizationarxiv.org
• [2603.05438] Planning in 8 Tokens: A Compact Discrete Tokenizer for Latent World Modelarxiv.org
• LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixelsle-wm.github.io
• The Model That Dreams the Worldmoe-capital.com
• World Modelsworldmodels.github.io
• Explore | alphaXivalphaxiv.org
• A Functional Taxonomy of World Models - Dr. Fei-Fei Lidrfeifei.substack.com
• Introducing Dreamer: Scalable Reinforcement Learning Using World Modelsresearch.google