Robot-use agents

web.mit.edu · 581 words · saved by 5 readers

In the same way as an LLM can use tools like calculators, web search, and even complete computers (“computer-use agents”), an LLM can also use a robot as a tool. Think of it as Claude acting as the puppeteer of a robot body. This ability has been researched for years2 3 4, but a common view remained that while LLMs might be useful for high-level planning, they were not yet suitable for full control. Researchers argued that LLM latency is too high to keep up with real-world dynamics5, that the models had poor spatial intelligence6, and that LLMs lacked sufficiently strong causal and physical understanding7. These limitations are starting to melt away with newer models like Fable and Astra. The debate continues, but it's time to take seriously that these AIs could soon be quite competent at general-purpose robot use. Indeed, recent demos provide tantalizing evidence that something is starting to change.8 9 10 It's worth thinking about the implications of this change. If LLMs can competen

This year we are going to see many LLMs1 being tested as robot-use agents. In the same way as an LLM can use tools like calculators, web search, and even complete computers (“computer-use agents”), an LLM can also use a robot as a tool. Think of it as Claude acting as the puppeteer of a robot body. This ability has been researched for years2 3 4, but a common view remained that while LLMs might be useful for high-level planning, they were not yet suitable for full control. Researchers argued that LLM latency is too high to keep up with real-world dynamics5, that the models had poor spatial…

Robot-use agents

saved by

related reading