Microsoft Researchers Are Using ChatGPT to Robots, Drones

The Microsoft researchers' use of ChatGPT to control drones and robots can be found here

Microsoft is now using the chatbot to control robots, even though ChatGPT is best known as an artificial intelligence program that can write essays and answer questions.

On Monday, the company's researchers published a paper on how ChatGPT can simplify the process of programming software commands to control various robots, such as mechanical arms and drones.

"We still rely heavily on hand-written code to control robots," the Microsoft researchers wrote. Microsoft, on the other hand, writes some of its software using ChatGPT.

Because the AI model was trained on extensive libraries of human text, including software code, this is possible with ChatGPT. ChatGPT has already demonstrated its ability to write and debug programs in a variety of languages by responding to text-based requests. As a consequence of this, the researchers at Microsoft decided to investigate the possibility of utilizing the same capabilities to write code for robotics hardware.

The scientists stated, "Incidentally, ChatGPT can do a great deal all alone, however, it needs some assistance." The scientists originally cleared up for the artificial intelligence program the different orders it could use to control a particular robot to help it record as a hard copy of the code.

"In addition to describing the goal of the task, we write a text prompt for ChatGPT that makes it clear which functions from the high-level library are available. "The prompt can also contain information about task constraints or how ChatGPT should form its answers," the researchers added.

The team demonstrated the method in several ways, one of which involved writing computer code to use ChatGPT to control an aerial drone. Microsoft experts first fed (Opens in another window) the mimicked knowledge chatbot a fairly extended brief fanning out the PC orders it could write to control the robot. The researchers could then instruct ChatGPT to control the robot in a variety of ways. This included instructing ChatGPT to identify a beverage using the drone's camera, such as coconut water and a Coca-Cola can.

The group said that ChatGPT "asked clarification questions when the user's instructions were ambiguous" and "wrote complex code structures for the drone such as a zig-zag pattern to visually inspect shelves."

The chatbot was additionally trained by the specialists in a single example: Take a selfie on a surface that is reflective. The code for the robot to fly before a mirror and take the selfie was composed by ChatGPT, which had the option to decipher the solicitation. The researchers used ChatGPT to write code that let a robot arm use several wooden blocks to build the Microsoft logo in a different demonstration (Opens in a new window).

Even though the research indicates that ChatGPT has the potential to be utilized in robotics, there is still one major drawback to the approach: The chatbot can only write computer code for the robot based on the initial text-based request made by the human. For the AI program to generate appropriate code, a human engineer must thoroughly explain to ChatGPT how a robot's application programming interface works.

In their paper, Microsoft researchers provide some guidelines on how to write an effective prompt for ChatGPT when it comes to controlling robots. Where anybody can share instances of provoking systems for various mechanical technology classes," the group likewise fostered an open-source stage on GitHub (Opens in another window).

However, another limitation of the robot is that it must appear to be constantly connected to ChatGPT. On the other hand, the integration may herald a time when robots will be able to understand a wide variety of human voice commands.

"Have you ever considered instructing a robot with your own words in the same way that you would instruct a human?" If you could just tell your robot assistant, "Please warm my lunch and let it find the microwave on its own," wouldn't it be amazing? In the interim, the researchers are asking questions and advising others to control robots with ChatGPT with caution. They wrote, "We emphasize that these devices should not be given full control of the advanced mechanic's pipeline, especially for security basic applications." Due to the tendency of LLMs (large language models) to eventually generate incorrect responses, it is fairly important to ensure solution quality and safety with human supervision before executing the code on the robot.