Home | Substack

Training big models is really hard, and as the models get bigger and expand into new domains, it’s only getting harder. LLMs use lots of text data, while VLMs require data with text and images, and vision-language-action (VLA) models in robotics require lots of data of robots performing real tasks in the real world. This hits agents especially hard: whether you want to control a real-world robot or take actions to fulfill user requests on the web, data of real-world interactions with action labels can’t be obtained as cheaply as text and images from the web. It’s no wonder that researchers have been trying to find a way to substitute the Next Best Thing in place of real data with observations and actions, in an attempt to get a best of both worlds: the power and generalization that comes with training huge models on huge datasets, at a cost that is much lower than standard methods for training foundation models on in-domain data. The Next Best Thing While authentic real-world data has

saved by

• Krish Mehta