A New AI System “AlphaCode” to Assist Experienced Coders

In arduous glitches, DeepMind's AlphaCode outperforms many human coders

Human intelligence is able to solve problems that arise out of the blue because it uses critical thinking that is influenced by experience. The field of machine learning has made great strides in producing and comprehending textual material, but breakthroughs in problem solving are still confined to tackling relatively straightforward programming and math issues or obtaining and replicating already existing solutions. We developed a program-writing system called AlphaCode as part of DeepMind's effort to address the intelligence problem. By figuring out new challenges that call for the use of a combination of logic, algorithms, code, and natural language comprehension, AlphaCode was able to place itself among the top 54% of competitors in programming competitions. The performance they're able to get on some really difficult issues is "quite astounding," says Armando Solar-Lezama, director of the Massachusetts Institute of Technology's computer assisted programming division.

AlphaCode improves upon Codex, a method introduced in 2021 by nonprofit research center OpenAI, which had previously set the bar for authoring AI code. The lab has already created GPT-3, a "big language model" that has been trained on billions of words from digital books, Wikipedia articles, and other internet text sites. GPT-3 is good at mimicking and reading human text. OpenAI developed Codex by fine-tuning GPT-3 using more than 100 gigabytes of code from the Internet software repository Github. When given a commonplace description of what it must accomplish, such as counting the vowels in a string of text, the software can generate code. But when faced with challenging tasks, it performs poorly.

The designers of AlphaCode concentrated on resolving such challenging issues. They began by feeding a big language model many gigabytes of code from GitHub, similar to the Codex researchers, only to get it acquainted with coding syntax and conventions. Then, using tens of thousands of issues gathered from programming contests, they trained it to convert problem descriptions into code. A task might, for instance, instruct a computer to count the number of binary strings (sequences of ones and zeros) of length n that don't contain any consecutive zeros. AlphaCode produces potential code solutions (in Python or C++) in response to a brand-new challenge and eliminates the subpar ones. However, DeepMind used AlphaCode to generate up to more than 1 million candidates, whereas academics have previously used models like Codex to generate tens or hundreds of candidates.

AlphaCode first keeps just the 1% of programs that successfully complete test cases that go along with issues. It clusters the keepers according to how closely their outputs resemble fabricated inputs in order to further reduce the field. Then, starting with the largest cluster, it sends programs from each cluster one at a time until it decides on a successful one or reaches ten submissions (about the maximum that humans submit in the competitions). It can test a variety of programming strategies because submissions come from various clusters. According to Kevin Ellis, a computer scientist at Cornell University who specializes in AI coding, that is the stage in AlphaCode's method that is the most novel.

DeepMind engaged AlphaCode into online coding contests to better evaluate its abilities. The system excelled in competitions with at least 5000 participants, outperforming 45.7% of programmers. The researchers observed no significant code or logic duplication when they compared the programs to those in the training database. Ellis was surprised by the inventiveness it inspired.

According to Yujia Li, a computer scientist at DeepMind and co-author of the paper, AI coding may have uses beyond winning tournaments. It might do routine software tasks, freeing developers to work at a higher or more abstract level, or it might assist non-programmers in developing straightforward applications. There are further issues. Tens of billions of trillions of operations are needed for each problem in AlphaCode, which is computational capacity that only the biggest tech companies possess. Additionally, the issues from the online programming contests it resolved were specific and contained. However, handling huge code packages across several locations is a common requirement of real-world programming, according to Solar-Lezama, which necessitates a more comprehensive grasp of the software.