Next-Latent Prediction Transformers Learn World Models feat. Jayden Teoh from MIT - YouTube
it's very interesting to see where the next-token transformer still fall short in surprisingly ways while crushing problems that look even more daunting.good...
it's very interesting to see where the next-token transformer still fall short in surprisingly ways while crushing problems that look even more daunting. good exemple is drawing manhattan map from directions or the little path-star graphs problem. I had the pleasure of chatting with jayden teoh from MIT who walks us through next-latent prediction (NextLat), a simple fix in the training objective he worked on when he was still an undergrad! the trick is super tiny: bolt a small MLP onto a GPT and make it predict the transformer's own next hidden state. to pull that off, the internal…
saved by
related reading
- AI Is Learning at the Wrong Level of Abstraction — Matthieu Wyartyoutube.com
- RLMs are Compositional Generalizers with Alex Zhang from MITyoutube.com
- Recursive Language Models — Alex Zhang, MIT PhDyoutube.com
- Stanford CS25: Transformers United V6 I From Representation Learning to World Modelingyoutube.com
- Stanford CS25: Transformers United V6 I Serving Transformers: Lessons from the Trenchesyoutube.com
- Stanford CS25: Transformers United V6 I Overview of Transformersyoutube.com
- AI researchers talk continual learning over a game of HORSEyoutube.com
- Transformers Laid Outgoyalpramod.github.io
- How LLMs Actually Work | 0xkato0xkato.xyz
- How trillion-parameter AI models work and how to serve themyoutube.com
- Everything About Transformerskrupadave.com
- Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com