Deployment of domestic ChatGPT only requires 6G of video memory! The ChatYuan model is open for download: the industry’s first functional dialogue open source Chinese large model

Some time ago, Yuanyu Intelligent Development Team trained a functional dialogue large model ChatYuan similar to ChatGPT, and opened a trial interface in the web version.

Now you can also deploy a ChatYuan on your own machine!

The model can be used in question and answer scenarios, and can conduct dialogues and various generation tasks based on context, including creative writing. It can also answer questions in fields such as law and COVID-19.

And supports zero-sample learning in all Chinese tasks. Users can use it by providing prompts. It supports nearly 30 kinds of Chinese tasks under the categories of text generation, information extraction and understanding. .

ChatYuan is further trained based on PromptCLUE-large combined with hundreds of millions of functional question and answer and multi-round dialogue data. The model parameters are 770 million, the video memory is about 6G, and a civilian graphics card can be loaded and used. , the model is currently open for download.

PromptCLUE is pre-trained on 100 billion token Chinese corpus, has learned a total of 1.5 trillion Chinese tokens, and conducted Prompt task-based training on hundreds of tasks.

For understanding tasks, such as classification, sentiment analysis, extraction, etc., the label system can be customized; for a variety of generation tasks, sampling can be freely generated.

How to use

1. Github

##Project address: https://github.com/clue-ai/ChatYuan

2. Huggingface

Project address: https://huggingface.co/ClueAI/ChatYuan- large-v1

#3. ModelScope

## Project address: https://modelscope.cn/models/ ClueAI/ChatYuan-large

Load model:

Using models for predictive inference methods:

##4. PaddlePaddle

##Project address: https://aistudio.baidu.com /aistudio/projectdetail/5404182

Model address: https://huggingface.co/ClueAI/ChatYuan-large-v1-paddle​

Use since Have data to train the model

1. Organize the data

Organize the data into a unified format, which consists of two parts: input ( Input) and output (Output).

For a single round of question and answer or generation:

Input: task description input text

Output: The text that the system needs to predict, such as the content after "Xiaoyuan" in the figure below.

For multiple rounds of dialogue:

Input: Enter the text above the task description

Output: The text that the system needs to predict, such as the content after "Xiaoyuan" in the figure below.

Single round dialogue data format:

##Multiple rounds of dialogue data format:

##2. Training model

For training code, please refer to:

Use pCLUE data set for training, prediction and Effect verification, pytorch implementation - online colab.

Github project: https://colab.research.google.com/drive/1lEyFhEfoc-5Z5xqpEKkZt_iMaojH1MP_ ?usp=sharingFor example

==========Example 1======= ===

##User:

What can you do

Xiao Yuan:

Hello! I am Yuanyu AI. I can answer your questions, write articles, assignments, translate, and I can also provide you with information on issues in the legal and other fields.

==========Example 2==========

User:

Write a poem about winter

Xiao Yuan:

The winter night is quiet and cold,

Clouds are floating in the sky,

On the frozen white snow,

Cold like snow.

Snowflakes melt into ice,

Like those snowflakes falling,

In the cold winter ,

Feel the joy of spring.

In the winter,

the wind and snow gradually subsided,

there was silence,

Bring happiness and warmth home.

==========Example 3==========

User:

During the internship in the preschool education major, there will be problems with students. Please propose improvement measures. 800 words

Xiao Yuan:

In the internship of preschool education majors, there will be some problems with students. , such as students' lack of professional knowledge, lack of cooperation awareness, lack of basic sense of responsibility, etc., we need to take measures to solve these problems.

First of all, during the internship process, we need to cultivate students' team awareness, strengthen cooperation between students and teachers, strengthen communication with parents, and improve their communication skills. In addition, we also need to strengthen students' independence, let students learn to think independently, learn to communicate with teachers and parents, and cultivate their independence.

In addition, in daily work, we must also provide students with appropriate opportunities and opportunities to have more contact with society and participate in more social activities, deepening their understanding of society and school. Understand and cultivate their initiative, positive and innovative spirit.

Finally, we also need to pay attention to the problems that students have during internships, especially students’ lack of cooperation, dependence, not knowing how to share, procrastination, etc. These students should cause us Pay attention to it and formulate reasonable improvement measures to improve these problems.

Technology and training process

ChatYuan is a model obtained by further training based on PromptCLUE. It removes text understanding and information extraction tasks, and strengthens the learning and training of question and answer, dialogue and various generative tasks; The addition of anti-interference data allows the model to ignore irrelevant context when necessary; the addition of learning from user feedback data enables the model to not only have certain general language understanding capabilities and specific task generation capabilities, but also better respond to user intentions. .

The learning process of PromptCLUE

1. Three major unifications: unified model framework (text-to-text), unified task form (prompt), unified application method (zero-shot/few-shot) (T0)
2. 大Scale pre-training: Based on the t5-large version, hundreds of G Chinese corpus were used, 1 million steps were trained, and 1.5 trillion Chinese word-level tokens were trained cumulatively
3. Large-scale task data: used 16 task types, hundreds of tasks, accumulated task data of 100 million levels
4. Hybrid pre-training: On the one hand, downstream tasks are used as pre-training corpus, and on the other hand, downstream tasks and pre-training corpus are trained together to reduce tasks Disaster forgetting and shortening the distance between pre-training and downstream tasks, better adapting to downstream tasks (ExT5)
5. Hybrid sampling: For many tasks with greatly different amounts of data, use all training batches within each training batch Tasks are sampled according to proportion, smooth sampling is performed according to the data volume of the task, and at the same time, the upper limit of the task data volume sampling pool is limited. Smooth sampling can reduce the harm of biased task training, and training within each batch can reduce the negative transfer of training between heterogeneous tasks (T5)
6. Phased training: On the one hand, it refers to the pre-training phase, Involves the phasing of training sequence length (128 and 512) to speed up pre-training (Bert); on the other hand, in the downstream training phasing, it involves changes in learning rate and sequence length and decreasing data volume restrictions on downstream tasks. Better adapt to different downstream tasks.
7. Increase the training of language model: refer to t5.1.1, in addition to using the Span Corrpution construction method for unsupervised training, and also using the prefix LM method to train to enhance the ability to generate tasks (LM adapted)
8. Increase the training of the encoder and decoder of the model: Construct Data_text and Data_target pre-training data corpora respectively based on the downstream task data, and add them to the pre-training to respectively enhance the model's encoder understanding ability and decoder generation ability (see UIE)
9. Reconstruct the model Chinese dictionary: Use sentencepiece to learn and build a model dictionary on Qianyi token, which is more in line with Chinese language habits

Follow-up work

The current version can carry out question and answer, dialogue and various creative writing or text generation. Compared with the online version, its intention understanding and generation capabilities still have a lot of room for improvement in some situations; it also cannot achieve reasoning well. or complex tasks. The existing version will then be further improved based on feedback.

The above is the detailed content of Deployment of domestic ChatGPT only requires 6G of video memory! The ChatYuan model is open for download: the industry’s first functional dialogue open source Chinese large model. For more information, please follow other related articles on the PHP Chinese website!