Single-machine training of 20 billion parameter large model: Cerebras breaks new record

This week, chip startup Cerebras announced a new milestone: training an NLP (natural language processing) artificial intelligence model with more than 10 billion parameters in a single computing device.

The volume of AI models trained by Cerebras reaches an unprecedented 20 billion parameters, all without scaling workloads across multiple accelerators. This work is enough to satisfy the most popular text-to-image AI generation model on the Internet - OpenAI's 12 billion parameter large model DALL-E.

#The most important thing about Cerebras’ new job is the reduced infrastructure and software complexity requirements. The chip provided by this company, Wafer Scale Engine-2 (WSE2), is, as the name suggests, etched on a single entire wafer of TSMC's 7 nm process, an area typically large enough to accommodate hundreds of mainstream chips - with a staggering 2.6 trillion transistors. , 850,000 AI computing cores and 40 GB integrated cache, and the power consumption after packaging is as high as 15kW.

The Wafer Scale Engine-2 is close to the size of a wafer and is larger than an iPad.

Although Cerebras’ single machine is already similar to a supercomputer in terms of size, the NLP model retaining up to 20 billion parameters in a single chip is still Significantly reduces the cost of training on thousands of GPUs, and the associated hardware and scaling requirements, while eliminating the technical difficulty of splitting models among them. The latter is "one of the most painful aspects of NLP workloads" and sometimes "takes months to complete," Cerebras said.

This is a customized problem that is unique not only to each neural network being processed, but also to the specifications of each GPU and the network that ties them together - These elements must be set up in advance before the first training session and are not portable across systems.

Cerebras’ CS-2 is a stand-alone supercomputing cluster that includes Wafer Scale Engine-2 chips, all Associated power, memory, and storage subsystems.

#What is the approximate level of 20 billion parameters? In the field of artificial intelligence, large-scale pre-training models are the direction that various technology companies and institutions are working hard to develop recently. OpenAI's GPT-3 is an NLP model that can write entire articles and do things that are enough to deceive human readers. Mathematical operations and translations with a staggering 175 billion parameters. DeepMind’s Gopher, launched late last year, raised the record number of parameters to 280 billion.

Recently, Google Brain even announced that it had trained a model with more than one trillion parameters, Switch Transformer.

"In the field of NLP, larger models have been proven to perform better. But traditionally, only a few companies have the resources and expertise to complete the decomposition of these large models. model, the hard work of distributing it across hundreds or thousands of graphics processing units," said Andrew Feldman, CEO and co-founder of Cerebras. "As a result, very few companies can train large NLP models - it is too expensive, time-consuming and unavailable for the rest of the industry."

Now, Cerebras' approach Can lower the application threshold of GPT-3XL 1.3B, GPT-J 6B, GPT-3 13B and GPT-NeoX 20B models, enabling the entire AI ecosystem to build large models in minutes and train on a single CS-2 system they.

#However, just like the clock speed of flagship CPUs, the number of parameters is only one factor in the performance of large models an indicator. Recently, some research has achieved better results while reducing parameters, such as Chinchilla proposed by DeepMind in April this year, which surpassed GPT-3 and Gopher in conventional cases with only 70 billion parameters.

The goal of this type of research is of course working smarter, not working harder. Therefore, Cerebras's achievements are more important than what people first see - this research gives us confidence that the current level of chip manufacturing can adapt to increasingly complex models, and the company said that systems with special chips as the core have the support " The ability of models with hundreds of billions or even trillions of parameters.

The explosive growth in the number of trainable parameters on a single chip relies on Cerebras’ Weight Streaming technology. This technology decouples computation and memory footprint, allowing memory to scale at any scale based on the rapidly growing number of parameters in AI workloads. This reduces setup time from months to minutes and allows switching between models such as GPT-J and GPT-Neo. As the researcher said: "It only takes a few keystrokes."

"Cerebras provides people with the ability to run large language models in a low-cost and convenient way, opening the door to artificial intelligence This is an exciting new era of intelligence. It provides organizations that cannot spend tens of millions of dollars with an easy and inexpensive way to compete in large models," said Dan Olds, chief research officer at Intersect360 Research. "We very much look forward to new applications and discoveries from CS-2 customers as they train GPT-3 and GPT-J-level models on massive data sets."

The above is the detailed content of Single-machine training of 20 billion parameter large model: Cerebras breaks new record. For more information, please follow other related articles on the PHP Chinese website!