13 billion parameters, 8 A100 training, UC Berkeley releases dialogue model Koala

Since Meta released and open sourced the LLaMA series of models, researchers from Stanford University, UC Berkeley and other institutions have carried out "second creation" on the basis of LLaMA, and have successively launched Alpaca, Vicuna and other " Alpaca" large model.

Alpaca has become the new leader in the open source community. Due to the abundance of "secondary creations", the English words for the biological alpaca genus are almost out of use, but it is also possible to name the large model after other animals.

Recently, UC Berkeley’s Berkeley Artificial Intelligence Institute (BAIR) released a conversation model Koala (literally translated as Koala) that can run on consumer-grade GPUs. Koala fine-tunes the LLaMA model using conversation data collected from the web.

Project address: https://bair.berkeley.edu/blog/2023/04/03/koala/

Koala has launched an online test demo:

• ##Demo address: https://chat.lmsys.org/?model=koala-13b
• Open source address: https://github.com/young-geng/EasyLM

Koala Overview

Like Vicuna, Koala also uses conversation data collected from the network to fine-tune the LLaMA model, with a focus on ChatGPT Public data of closed-source large model dialogues.

The research team stated that the Koala model is implemented in EasyLM using JAX/Flax and the Koala model is trained on a single Nvidia DGX server equipped with 8 A100 GPUs. It takes 6 hours to complete 2 epochs of training. The cost of such training is typically less than $100 on public cloud computing platforms.

The research team experimentally compared Koala with ChatGPT and Stanford University's Alpaca. The results showed that Koala-13B with 13 billion parameters can effectively respond to various user queries and generate Response is generally better than Alpaca's and is comparable to ChatGPT's performance in more than half of the cases.

The most important significance of Koala is that it shows that when trained on a higher quality data set, a model small enough to run locally can also achieve excellent performance similar to that of a large model . This means that the open source community should work harder to curate high-quality datasets, as this may lead to more secure, realistic, and powerful models than simply increasing the size of existing systems. From this perspective, Koala is a small but refined alternative to ChatGPT.

However, Koala is only a research prototype and still has significant flaws in content, security, and reliability, and should not be used for any purpose other than research.

Datasets and Training

The main hurdle in building a conversation model is managing the training data. Large conversation models such as ChatGPT, Bard, Bing Chat, and Claude all use proprietary datasets with extensive human annotations. To build Koala's training dataset, the research team collected and curated conversation data from the web and public datasets, which contain data shared publicly by users speaking to large language models such as ChatGPT.

Unlike other models that crawl as much network data as possible to maximize the data set, Koala focuses on collecting small high-quality data sets, including the question and answer part of public data sets, human Feedback (positive and negative) and dialogue with existing language models. Specifically, Koala's training data set includes the following parts:

ChatGPT distillation data:

• Publicly available chatGPT conversation data (ShareGPT);
• Human ChatGPT comparison corpus (HC3), which uses both human and ChatGPT responses from the HC3 dataset.

Open source data:

• Open Instruction Generalist (OIG);
• Dataset used by the Stanford Alpaca model;
• Anthropic HH ;
• OpenAI WebGPT;
• OpenAI Summarization.

Experimentation and Evaluation

This study conducted a manual evaluation comparing the generation of Koala-All with Koala-Distill, Alpaca and ChatGPT. The results are compared and the results are shown in the figure below. Among them, two different data sets are used for testing, one is Stanford's Alpaca test set, which includes 180 test queries (Alpaca Test Set), and the other is the Koala Test Set.

Overall, the Koala model is sufficient to demonstrate many features of LLM, while being small enough to facilitate fine-tuning or in situations where computing resources are limited. Use below. The research team hopes that the Koala model will become a useful platform for future academic research on large-scale language models. Potential research application directions may include:

• Safety and alignment: Koala allows further research on language models security and better alignment with human intent.
• Model Bias: Koala enables us to better understand bias in large language models, delve into quality issues in conversation datasets, and ultimately help improve the performance of large language models.
• Understanding large language models: Because Koala models can run on relatively cheap consumer-grade GPUs and perform a variety of tasks, Koala allows us to better examine and understand conversational language The internal structure of the model makes the language model more interpretable.

The above is the detailed content of 13 billion parameters, 8 A100 training, UC Berkeley releases dialogue model Koala. For more information, please follow other related articles on the PHP Chinese website!