How the AI chatbot ChatGPT can find bugs in computer code and fix bugs in computer code

The AI chatbot developed by OpenAI, ChatGPT finds bugs in computer code, as well as ChatGPT, fixes bugs using standard machine learning approaches – and performs even better when engaged in conversation. The AI chatbot ChatGPT can find bugs in computer code and ChatGPT fix bugs in computer code by using a standard machine learning approach. Read to know more.

The OpenAI chat room ChatGPT chatbot is very good at fixing software bugs, but its main advantage over other methods and AI models is its unique ability to converse with humans allowing it to improve the answer correctness.

Researchers from Johannes Gutenberg University Mainz and University College London compared OpenAI's ChatGPT to "standard automated programme repair techniques" as well as two deep-learning approaches to programme repairs: Codex, OpenAI's GPT-3-based model that underpins GitHub's Copilot paired programming and auto code-completion service, and CoCoNut from the University of Waterloo, Canada.

"We find that ChatGPT's bug fixing performance is competitive to the common deep learning approaches CoCoNut and Codex, and significantly better than the results reported for the standard programme repair approaches," the researchers write in a new arXiv paper, which New Scientist first spotted.

Although the ability of ChatGPT to solve coding problems is not novel, the researchers emphasise that its unique ability to converse with humans gives it a potential advantage over other approaches and models.

The researchers used the QuixBugs bug-fixing benchmark to evaluate ChatGPT's performance. Because they were developed before 2018, automated programme repair (APR) systems appear to be at a disadvantage.

ChatGPT is built on the transformer architecture, which was developed by Google, according to Meta's AI chief Yann LeCunn this week. Codex, Microsoft Research's CodeBERT, and Google's predecessor BERT are all based on Google's transformer method.

In examples for debugging code, OpenAI highlights ChatGPT's dialogue capability, where it can ask for clarifications and receive hints from a person to arrive at a better answer. It used Reinforcement Learning from Human Feedback to train the large language models that power ChatGPT (GPT-3 and GPT 3.5). (RLHF).

While ChatGPT's ability to discuss can help it arrive at a more correct answer, the researchers note that the quality of its suggestions is unknown. That's why they wanted to assess ChatGPT's bug-fixing capabilities.

ChatGPT was tested against QuixBugs 40 Python-only problems, and the researchers manually checked whether the suggested solution was correct or not. They repeated the query four times because the reliability of ChatGPT's answers is somewhat random, as a Wharton professor discovered after putting the chatbot through an MBA-style exam.

ChatGPT fixed 19 of the 40 Python bugs, matching CoCoNut (19) and Codex (19). (21). However, standard APR methods only addressed seven of the issues.

The researchers discovered that ChatGPT had a success rate of 77.5% with follow-up interactions.

However, the implications for developers are unclear. ChatGPT-generated answers were recently banned on Stack Overflow due to their low-quality but plausible-sounding nature. The Wharton professor discovered that ChatGPT can act as a "smart consultant" (one who produces elegant but often incorrect answers) and foster critical thinking in MBA students.