Chen Danqi's ACL academic report is here! Detailed explanation of the 7 major directions and 3 major challenges of the large model 'plug-in' database, 3 hours full of useful information

Tsinghua Yao Class AlumniChen Danqi gave a latest speech at ACL 2023!

The topic is still a very hot research direction recently-

(big)Language models like GPT-3 and PaLM, whether It is necessary to rely on retrieval to make up for its own shortcomings, so as to better implement the application.

In this speech, she and three other speakers jointly introduced several major research directions on this topic, including training methods, applications and challenges.

Picture

The response from the audience during the speech was also very enthusiastic. Many netizens raised their questions seriously, and several speakers tried their best to answer their questions. .

Picture

As for the specific effect of this speech? Some netizens directly said "recommend" to the comment area.

Picture

So, what exactly did they talk about in this 3-hour speech? What other places are worth listening to?

Why do large models need "plug-in" databases?

The core theme of this speech is "Retrieval-based Language Model", which contains two elements: Retrieval and Language Model.

From the definition, it refers to "plugging in" a data retrieval database to the language model, and using this database when performing inference (etc. operations) Perform a search and finally output based on the search results.

This type of plug-in data repository is also called a semi-parametric model or a non-parametric model.

Picture

The reason why we need to study this direction is because of (big) language models such as GPT-3 and PaLM , while showing good results, some troublesome "bugs" also appeared. There are three main problems:

1. The number of parameters is too large. If it is repeated based on new data, Training, the computational cost is too high;
2, poor memory (faced with long text, memorize the following and forget the above), over time it will cause hallucinations, and it is easy to Data leakage;
3, With the current amount of parameters, it is impossible to remember all the knowledge.

In this case, the external retrieval corpus was proposed, which is to "plug-in" a database for the large language model, so that it can answer questions by searching for information at any time, and because this database can be updated at any time, it can also Don't worry about the cost of retraining.

After introducing the definition and background, it is the specific architecture, training, multi-modality, application and challenges of this research direction.

In the architecture, it mainly introduces the content, retrieval method and retrieval "timing" based on the language model retrieval.

Specifically, this type of model mainly retrieves tokens, text blocks and entity words (entity mentions) . The methods and timing of using retrieval are also very diverse, and it is a very flexible type. model architecture.

Picture

In terms of training method, it focuses on independent training(independent training, language model and retrieval Model separate training) , continuous learning (sequential training) , multi-task learning (joint training) and other methods.

Picture

As for application, this type of model involves more, and can be used not only for code generation, It can be used for tasks such as classification and knowledge-intensive NLP, and can be used through methods such as fine-tuning, reinforcement learning, and retrieval-based prompt words.

Application scenarios are also very flexible, including long-tail scenarios, scenarios that require knowledge updates, and scenarios involving privacy and security. This type of model has a place to be used.

Of course, it’s not just about text. This type of model also has the potential for multimodal extension, which can be used for tasks other than text.

Picture

It sounds like this type of model has many advantages, but there are also some challenges based on the retrieval-based language model.

In his final "ending" speech, Chen Danqi highlighted several major problems that need to be solved in this research direction.

First, small language model (continuously expanding) Does a large database essentially mean that the number of parameters of the language model is still very large? How to solve this problem?

For example, although the parameter amount of this type of model can be very small, only 7 billion parameters, the plug-in database can reach 2T...

Picture

Second, the efficiency of similarity search. How to design algorithms to maximize search efficiency is currently a very active research direction.

Picture

Third, complete complex language tasks. Including open-ended text generation tasks and complex text reasoning tasks, how to use retrieval-based language models to complete these tasks is also a direction that requires continued exploration.

Picture

Of course, Chen Danqi also mentioned that these topics are not only challenges, but also research opportunities. Friends who are still looking for thesis topics, you can consider whether to add them to the research list~

It is worth mentioning that this speech is not a topic "out of thin air", 4 speakers I thoughtfully posted a link to the paper reference for the speech on the official website.

From model architecture, training methods, applications, multi-modality to challenges, if you are interested in any part of these topics, you can go to the official website to find the corresponding classic papers:

Picture

Answer the audience’s confusion on the spot

It was such a informative speech, and the four keynote speakers were not without background. They also patiently spoke during the speech. Questions raised by the audience were answered.

Let’s first talk about who the speakers are at Kangkang.

The first is Chen Danqi, assistant professor of computer science at Princeton University who led this speech.

Picture

She is one of the most popular young Chinese scholars in the field of computer science recently and is also an alumnus of Tsinghua Yao Class in 2008.

In the informatics competition circle, she is quite legendary-

CDQ divide and conquer algorithm is named after her. In 2008, she won an IOI gold medal on behalf of the Chinese team.

And her 156-page doctoral thesis "Neural Reading Comprehension and Beyond" once became very popular. Not only did it win the Stanford Best Doctoral Thesis Award that year, it also became a recent Stanford University award. One of the most popular graduation thesis of the decade.

Now, in addition to being an assistant professor of computer science at Princeton University, Chen Danqi is also the co-leader of the school’s NLP team and a member of the AIML team.

Her research interests mainly focus on natural language processing and machine learning, and she is interested in simple and reliable methods that are feasible, scalable and generalizable in practical problems.

Also from Princeton University, there is also Chen Danqi’s apprentice

Zhong Zexuan（Zexuan Zhong）.

Picture

Zhong Zexuan is a fourth-year doctoral student at Princeton University. I graduated with a master's degree from the University of Illinois at Urbana-Champaign, and my mentor was Xie Tao; I graduated from the Computer Science Department of Peking University with a bachelor's degree, and worked as an intern at Microsoft Research Asia, and my mentor was Nie Zaiqing.

His latest research focuses on extracting structured information from unstructured text, extracting factual information from pre-trained language models, analyzing the generalization ability of dense retrieval models, and developing language models suitable for retrieval-based training techniques.

In addition, the keynote speakers include Akari Asai and Sewon Min from Washington University.

Picture

Akari Asai is a fourth-year doctoral student at the University of Washington, majoring in natural language processing. He graduated from the University of Tokyo, Japan.

She is primarily interested in developing reliable and adaptable natural language processing systems to improve information acquisition.

Recently, her research focuses on general knowledge retrieval systems, efficient adaptive NLP models and other fields.

Picture

Sewon Min is a doctoral candidate in the Natural Language Processing Group at the University of Washington. During his doctoral studies, he worked at Meta AI He worked part-time as a researcher for four years and graduated from Seoul National University with a bachelor's degree.

Recently she focuses on language modeling, retrieval and the intersection of the two.

During the speech, the audience also enthusiastically raised many questions, such as why perplexity (perplexity) is used as the main indicator of the speech.

Picture

The speaker gave a detailed answer:

When comparing parameterized language models, the degree of confusion (PPL) is often used. But whether improvements in perplexity can be translated into downstream applications remains a research question.

Research has shown that perplexity has a good correlation with downstream tasks (especially generation tasks) , and perplexity usually provides very stable results, which can be Evaluation on large-scale evaluation data (Compared to downstream tasks, evaluation data is unlabeled, and downstream tasks may be affected by the sensitivity of cues and the lack of large-scale labeled data, resulting in unstable results) .

Picture

Some netizens raised this question:

About "The training cost of language model is high, and Introducing retrieval may solve this problem", did you just replace time complexity with space complexity(data storage)?

The answer given by the speaker is Aunt Jiang’s:

The focus of our discussion is how to reduce the language model to a smaller size. Small, thus reducing time and space requirements. However, data storage also actually adds additional overhead, which needs to be carefully weighed and studied, and we believe this is a current challenge.

Compared with training a language model with more than 10 billion parameters, I think the most important thing at present is to reduce the training cost.

Pictures

If you want to find the PPT of this speech, or the specific playback, you can go to the official website to have a look~

Official website: https://acl2023-retrieval-lm.github.io/

The above is the detailed content of Chen Danqi's ACL academic report is here! Detailed explanation of the 7 major directions and 3 major challenges of the large model 'plug-in' database, 3 hours full of useful information. For more information, please follow other related articles on the PHP Chinese website!