Microsoft Makes ChatGPT Code For Robots! Beyond the Words Lies the Catch

While Microsoft makes ChatGPT code for robots, experts say chatGPT is far away from achieving the kind of autonomy

Microsoft is about to attempt another stunt with chatGPT. It will employ the canny AI chatbot to deliver commands to robots and drones. A team of Microsoft researchers has put their research findings in the paper, 'ChatGPT for Robotics: Design Principles and Model Abilities', dwelling on how ChatGPT can unlock a new robotic paradigm. The research primarily explores how it is possible for a user to give instructions to a robot in spite of being coding illiterate. The proposed design principles will allow the user to sit in the loop and provide feedback to the large language model (LLM). As per the paper, researchers have aimed to apply LLMs knowledge without the need for fine-tuning. "Our goal with this research is to see if ChatGPT can think beyond the text, and reason about the physical world to help with robotics tasks," the Microsoft researchers wrote.

 In the technical paper, they demonstrate how chatGPT can solve robotics puzzles, and its deployment in the manipulation, aerial, and navigation domains. Indeed, the chatbot can quickly translate natural language instructions into code. But can it take straightforward and precise instructions like other AI assistants do? Not quite. It requires detailed instructions about the process, as to what happens with specific parts of the drones or what kind of instructions chatGPT had to generate code. No wonder, the team towards the end of the paper proposes PromptCraft, a collaborative open-source platform, that helps in developing positive and negative prompting strategies when working with LLMs in a robotics context. "Over time we aim to provide different environments where users can test their prompts, and welcome new contributions," mention the researchers.

According to the paper authored by, Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor, the project defines a high-level API that ChatGPT can understand and mapped it to lower-level robot functions. Then they wrote elaborate text prompts for chatGPT to explain the bot, the task goals, available functions, and also the limitations. ChatGPT could generate device-specific code to achieve the set simulation goal. It seems like chatGPT could accomplish spatio-temporal reasoning, taking into account the details of the physical world.  "We see that ChatGPT is able to appropriately use the provided API functions, reason about the ball's appearance and call relevant OpenCV functions, and command the robot's velocity based on a proportional controller," they explain in the paper.

For someone who has a reasonable knowledge of how such models work, this sort of reasoning should sound absurd because for eg., the autonomous vehicle industry hasn't achieved it yet so chatGPT couldn't have either.

However, thankfully, the Microsoft team acknowledges chatGPT's limitations and that it requires external supervision. "We emphasize that these tools should not be given full control of the robotics pipeline, especially for safety-critical applications. Given the propensity of LLMs to eventually generate incorrect responses, it is fairly important to ensure solution quality and safety of the code with human supervision before executing it on the robot," the paper states.