Emergent Abilities of LLMs Define how they Evolve Over Time: Study

The "emergent" capabilities of huge language models are being studied by AI researchers. This study clarifies the relationship between the scale of large language models and their "emergent" capabilities.

Large Language Models (LLMs) are the focus of attention and hype due to their magical abilities to produce lengthy passages of coherent text, perform tasks for which they were not trained, and converse (to some extent) about subjects that were previously regarded as inappropriate for computers.

But there is still plenty to understand about how LLMs function and don't function. Researchers from Google, Stanford University, DeepMind, and the University of North Carolina at Chapel Hill have just published a study that examines novel tasks that LLMs can perform as they expand and accumulate more training data. Large language models are a particularly fascinating case study since they have manifested extremely distinct emergence-related traits. LLMs are very large transformer neural networks that have been trained on hundreds of terabytes of text input and frequently span hundreds of billions of parameters. They can be used for many different things, including text production, answering questions, summarizing information, and more.

LLMs' capability for few-shot and zero-shot learning, or the ability to carry out activities that were not included in their training examples, is one of its intriguing characteristics. With the release of OpenAI's GPT-3 in 2020, few-shot learning in LLMs gained significant interest, and its scope and bounds have since been extensively researched.

Inspired by Anderson's work, Jacob Steinhardt, Professor at UC Berkeley, defined emergence as "when quantitative changes in a system result in qualitative changes in behavior." To be more specific, Rishi Bommasani, a Ph.D. candidate at Stanford University and a co-author of the paper, explained that emergent skills are those that are "not present in smaller models but are present in larger models. This distinguishes emergent skills from abilities that smoothly grow with scale: it is significantly more difficult to forecast when emergent abilities will arise ". The researchers searched for phase transitions, where model performance is near-random below a specific size threshold and significantly above random beyond that threshold, to find emerging talents in Large Language Models.

Model size (number of parameters), computation (FLOPs), and data size are some examples of scale metrics. The researchers' analysis focuses on computation and model size, but they emphasize that no single proxy can fully reflect all facets of scale. The study clarifies the relationship between the "emergent" skills of large language models and their magnitude.

The researchers evaluated several well-known LLM families in their investigation, including LaMDA, GPT-3, Gopher, Chinchilla, and PaLM. They also used challenges from TruthfulQA, Massive Multi-task Language Understanding (MMLU), and Word in Context (WiC), all benchmarks that are designed to test the limits of LLMs in tackling challenging language tasks. They selected several tasks from BIG-Bench, a crowd-sourced benchmark of over 200 tasks "that are believed to be beyond the capabilities of current language models." Additionally, the researchers went above and beyond to test the LLMs on multi-step computation, multi-step reasoning, and following multi-step instructions. With the now well-known few-shot prompting/in-context learning, GPT-3 is renowned for having introduced the truly distinctive initial wave of emergent abilities in LLMs, according to Bommasani. The largest models (i.e., the 175B model) could do reasonably well on some tasks because a task can be given in natural language here with a description and perhaps five or so samples of the input-output structure of the work. In other words, you could specify the task without using fine-tuning or gradient-based methods and needed significantly less task-specific data.

The study's findings demonstrate a strong correlation between scale and the creation of new skills. Every LLM family, which comes in various sizes, performs randomly or below randomly on jobs that are smaller than a specified threshold.