Google explores a new NLU task "Natural Language Assessment", let AI help you warm up before the formal interview!

"Question solving" can be said to run through life. Some questions can be solved alone, such as exam questions; but when it comes to interactive questions such as interviews, it is difficult for one person to solve them.

This kind of interaction is different from ordinary question and answer. It usually requires the "sparing partner" to answer questions in a specific situation and guide the user to think in order to achieve the ultimate goal.

For example, the interviewer will not only be responsible for asking questions, but also need to guide you to express your understanding of the problem and optional solutions. Such questions may also be open-ended, such as self-introduction.

The ultimate goal of AI is that everything that humans can do can be replaced by models, and this type of "interview trainer" is no exception.

But in the current field of natural language processing, this ability has not received enough attention and is technically very challenging.

Recently, Google introduced an important natural language understanding (NLU) capability, namely Natural Language Assessment (NLA), on its official blog, and discussed how it can be helpful in the context of education. .

Typical NLU tasks focus on the user’s intent, while NLA allows the answer to be evaluated from multiple perspectives.

In situations where users want to know how good their answer is, NLA can provide an analysis of how close the answer is to expectations.

In situations where there may not be a “right” answer, NLA can provide nuanced insights, including topicality, relevance, lengthy questions, and more.

The researchers developed the scope of NLA, proposed a practical model for performing topical NLA, and demonstrated how NLA can be used to help job candidates practice answering interview questions.

Overview of Natural Language Evaluation

The goal of NLA is to evaluate the answers given by users against a set of expectations.

For example, there is an NLA system that interacts with students. It has the following components:

• Ask a question to the student;
• Expectations define user expectations What to get in the answer. For example, a specific text answer or a set of topics that the user expects the answer to cover, and the answer needs to be concise.
• Answers provided by students;
• Evaluation results. Including correctness, missing information, being too specific or general, stylistic feedback, pronunciation, etc.
• Optional: context. For example, a paragraph in a book or article.

With NLA, both expectations for answers and evaluation of answers can be very broad, which makes interactions between teachers and students more expressive and detailed.

Questions with Specific Correct Answers

Even in cases where there is a definite correct answer, the answer can be evaluated with more nuance than simply correct or incorrect.

• Context: Harry Potter and the Philosopher's Stone
• Question: What is Hogwarts?
• Expectation: Hogwarts is a school of magic
• Answer: I'm not sure, but I think it is a school.

For the question and answer system, the above answer may be marked as incorrect due to the lack of key details "magic", because the user will think that the answer is not completely correct and does not make much sense.

NLA can provide more detailed understanding, such as determining that a student's answer is too general and the student is not confident enough about the answer.

This nuanced assessment, along with noticing the uncertainty students express, is important in helping students build skills in conversational settings.

TOPIC EXPECTATIONS

In many cases, the questioner does not expect a specific answer.

For example, if a student is asked an opinion question and there is no specific text expectation, the questioner is more concerned with the relevance and opinion of the answer. Perhaps the simplicity and fluency of the answer are also the questioner. within the scope of assessment.

• Question: Please introduce yourself. (Tell me a little about yourself?)
• Expectation: A collection of topics, which may include "education", "experience", "interest", etc.
• Answer: I am in Salina, California Si grew up and went to Stanford and majored in economics, but then got excited about the tech industry, so next I...

In this case, a useful evaluation output The user's answers will be mapped to a subset of the topics covered, possibly with markers for which parts of the text relate to which topic.

This is challenging from a natural language processing perspective because the answers can be long, the topics can be mixed, and each topic itself can be multifaceted.

Topic NLA model

In principle, topic NLA (Topicallity NLA) is a standard multi-classification task, and developers can easily train a classifier based on commonly used models.

But for NLA, there is very little training data available, and collecting training data for each question and topic is very costly and time-consuming.

Google’s solution is to decompose each topic into fine-grained components that can be identified using large language models (LLM) and subjected to simple general tuning.

The researchers mapped each topic to a list of potential questions and defined that if a sentence contained an answer to one of these potential questions, then it covered that topic.

For the topic of Experience, the model can choose some potential questions, such as:

• Where do you work?
• What did you study?
• ...

Under the topic of Interests, there are also some basic questions, such as

• What are you interested in?
• What do you like to do?
• ...

These basic questions are designed through an iterative manual process.

Importantly, because these questions are fine-grained enough, current language models can capture the semantics within these sentences (such as the difference between What and Where), and also allow developers to provide solutions for NLA subject tasks. A zero-shot setup: After the model is trained once, new questions and new topics can be continuously added, or existing topics can be expected to be adapted by modifying the basic content without collecting topic-specific data.

Help job seekers prepare for interviews

In order to explore the application scenarios of NLA, Google developers also worked with job seekers to develop a new tool, Interview Warmup. Help users prepare for interviews in fast-growing employment fields such as IT Support and User Experience Design.

The website provides a large number of questions so that job seekers can practice answering questions from industry experts at home to help them become more confident and comfortable in real-person interviews.

Google was also inspired by job seekers and came up with NLA research after understanding the difficulties in the interview process.

Interview Warmup does not rate or judge answers. It only provides users with an environment to practice alone and helps users improve themselves.

Every time a user answers an interview question, the answer is parsed sentence by sentence by the NLA model, and the user can then switch between different talking points to see which ones were found in their answer.

The researchers realized that there are many potential pitfalls when it comes to signaling to users that their feedback is "good," especially when the model only detects a limited set of topics.

Instead, the system puts the control in the hands of the user, using only machine learning to help users discover how to improve.

So far, the tool has helped a large number of job seekers from all over the world with great results, and the development team has recently expanded it to Africa, And plans to continue working with job seekers, iterating and making the tool more helpful to the millions of people looking for new jobs.

Natural Language Assessment (NLA) is a technically challenging and interesting research area.

NLA paves the way for new conversational applications that advance learning through nuanced evaluation and analysis of answers from multiple perspectives.

By working with the community, from job seekers and businesses to classroom teachers and students, NLA can identify situations where NLA has the potential to help users learn, engage and develop skills across a variety of subjects, in a responsible way Build applications that enable users to assess their abilities and find ways to improve.

Reference: https://ai.googleblog.com/2022/10/natural-language-assessment-new.html

The above is the detailed content of Google explores a new NLU task "Natural Language Assessment", let AI help you warm up before the formal interview!. For more information, please follow other related articles on the PHP Chinese website!