Google is so anxious that it supports DeepMind! 70 billion parameters Sparrow hard ChatGTP

Google’s counterattack has begun!

Recently, Microsoft has been attacking Google one after another - it just said that ChatGPT will be integrated into its own search engine Bing, and then it announced that ChatGPT plans to join the Office "family bucket". bomb.

Microsoft frequently makes dangerous moves, and Google naturally cannot sit idly by.

According to the information revealed by the founder of DeepMind in an exclusive interview with "Times", Google is likely to use Sparrow, which DeepMind previously launched, to respond to ChatGPT's provocation!

In fact, in the past, Google had several large models that had been brewing for a long time, such as LaMDA, Imagen, etc., which were said to have reached SOTA, but had never been made public.

It seems that Google has really been forced by Microsoft recently and is ready to lead its "children" out to meet people.

In response to ChatGPT, Google has the answer

Google’s hesitation once planted the seeds of regret for myself today.

Although ChatGPT is very expensive, burning 300,000 US dollars every day, people will always remember that on that day in December 2022, it was OpenAI’s ChatGPT that wrote One of the most significant moments in the history of large-scale language models is that people all over the world flooded servers to chat with robots, not Google.

In fact, before OpenAI launched ChatGPT, Google already had large-scale language models optimized for dialogue, such as LaMDA and Flamingo.

Even as early as 2020, Google launched Meena, an end-to-end neural network conversation model with 2.6 billion parameters. Its effects were said to be more reasonable and specific than all other SOTA chatbots at the time.

But as of now, Google has not applied these research results to any products.

According to its own statement, this is mainly due to considerations of "reputational risk". AI chatbots are very easy to be "toxic" - the data for AI training comes from the Internet, which is often full of prejudice, hatred and abuse, so the AI ​​will generate destructive content and answers that are inconsistent with the facts.

In mid-December, Google AI head Jeff Dean said at a plenary meeting that although Google has the technology and capabilities to make AI products, compared with "small startups," they Decisions must be made "more conservatively".

But now, Google, forced into a corner by Microsoft, may no longer be "conservative".

ChatGPT has become popular all over the Internet in the past two months, setting off a frenzy of national participation. In recent days, it has been reported that Microsoft has invested another 10 billion in OpenAI with a stroke of pen. dollar news.

Faced with the ongoing discussion of "Can ChatGPT replace the search engine?" and the fact that Microsoft's Office software is expected to undergo a comprehensive upgrade, now Google must Prove to the outside world that you can catch up with OpenAI or even surpass it.

Not long ago, DeepMind CEO and founder Demis Hassabis said in an interview with "Times" that DeepMind's chatbot Sparrow may enter the "testing" stage later this year ( private beta).

Hassabis said that the reason for delaying the release of Sparrow is that DeepMind hopes to make Sparrow go further in terms of reinforcement learning-based functions, which is what ChatGPT lacks.

DeepMind, as the most important AI research institution under Google, has been providing Google with advanced new technologies and integrating them into consumer products.

It seems that Google is going to make a big move.

Previously, Yann LeCun, the head of Meta AI, also expressed support for Google: "Google has more experience in using large language models for search than other companies (including OpenAI ).》

Who is DeepMind’s Sparrow?

Like ChatGPT, the Sparrow model proposed by DeepMind in September 2022 also adopts a reinforcement learning (RL) framework based on human feedback.

But the difference is that Sparrow can "Google it".

##Paper address: https://arxiv.org/abs/2209.14375

After all, Sparrow was originally designed to chat with users and use Google to search for relevant information as supporting evidence when answering questions.

Of course, in order to ensure that the behavior of the model is safe, its behavior must also be constrained.

Therefore, the researchers identified an initial set of simple rules for the model, such as "Don't make hateful or insulting remarks," "Don't impersonate or pretend to be a real person," etc. .

Afterwards, induce Sparrow to violate these rules through small talk. The conversations involved can be used to train a separate "rule model" to show when Sparrow's behavior violates which rules.

From the results, when a potentially dangerous behavior is detected, such as when a user asks how to hotwire a car, the Sparrow model Hui said that he was trained not to provide advice on any illegal behavior.

According to participants, when asked a factual question, Sparrow generally provided credible answers and had 78 % of the data is supported by evidence, which is a great improvement compared to the baseline model.

#Can DeepMind help Google get out of trouble?

Whether Sparrow is qualified to compete with ChatGPT

In recent years, the focus of artificial intelligence research is usually to use more parameters to get better performance. But DeepMind has greatly reduced the size of the Chinchilla language model.

As the basis of Sparrow, the number of parameters of Chinchilla is only a fraction of GPT-3 - 70 billion vs 175 billion.

At the same time, DeepMind has doubled the amount of Chinchilla’s training data several times – 1.3 trillion tokens.

The result is that this "mini version" of the artificial intelligence model performs better than its predecessors in almost all speech tasks, including of course the old rival GPT-3.

It is not difficult to see from the success of Chinchilla that small artificial intelligence models trained with large amounts of data can also achieve high performance.

Moreover, the biggest advantage of small models is that not only are they cheaper to run, but they can also be optimized for specific use cases with very little additional data.

This may prevent users from using it too happily and burning all the company's money.

Well, I’m talking about OpenAI.

However, the now popular ChatGPT is based on the more advanced GPT-3.5.

When OpenAI launched the latest version of GPT-3.5 text-davinci-003, OpenAI once listed its advantages:

1. Generate higher quality: Provide clearer, more engaging, and more persuasive content.

2. Can handle more complex instructions: Can use its capabilities more creatively.

3. Get better at generating longer content: Able to accomplish tasks that were previously difficult to achieve.

So, the question now actually becomes: Can Chinchilla be used against GPT-3.5?

「Really fragrant」

Step by step to get to today, looking back at Google’s action layout in recent years, the management probably has mixed feelings. .

As early as early December, when ChatGPT first exploded in popularity, there were “whistleblowers” ​​at Google who felt the crisis. Some employees raised their concerns at the all-hands meeting - the rapid rise of ChatGPT may cause Google to lose its competitive advantage in the field of AI.

Sridhar Ramaswamy, the former head of the Google advertising team, also said that if ChatGPT continues to be so popular, netizens will no longer click on Google links with ads.

You know, in 2021, the advertising business earned Google $208 billion, accounting for 81% of Alphabet’s total revenue.

At that time, Google executives did not realize what kind of crisis the birth of ChatGPT would cause them.

Google executives at the time stated this: Google would not launch a competitor to ChatGPT because Google had greater "reputational risk" than startups such as OpenAI. Moreover, "chatbots are not yet ready to replace search engines."

But just ten days later, the attitude of the executives changed drastically.

Pichai instructed some teams to switch directions and develop AI products

CEO Pichai issued a "Red Code" urgently, participated in several meetings around Google's AI strategy, and gave key instructions: Google's multiple teams need to focus on solving the threat ChatGPT poses to the company's search engine business.

Teams across Google’s Research, Trust & Safety and other divisions have been directed to shift gears and instead begin assisting with the development and launch of AI prototypes and products.

In fact, Google’s concerns are understandable.

A robot will spread lies, spread hate speech, and even give people the illusion that it is "conscious". Burn money and provide a testing ground for such a robot. OpenAI's attempt It's actually quite bold.

So far, OpenAI’s test response has been quite good. Someone has found a way forward, and it seems that Google is also about to enter the market.

Trapped in the "Innovator's Dilemma"

Decoder reporter Matthias Bastian believes that in addition to concerns about "reputational risk", there are more reasonable reasons Yes, Google may be stuck in the "innovator's dilemma."

This term was proposed by American scholar Clayton Christensen in 1997 to describe a situation where it is difficult for mature companies to adopt methods that disrupt traditional markets. new technologies or business models.

These mature companies often hold the most important resources and have sufficient scientific research capabilities. However, their existing customer base and fixed internal processes can keep them away from breakthrough innovation.

And when smaller, more nimble competitors enter and take over the market (yes, this refers to OpenAI), it may eventually lead to the demise of established large companies.

Most of Google’s revenue comes from online advertising

Now, Google's core search engine business is growing and profitable (remember the $208 billion above).

It can be said that every search results page is printing money for Google.

The new search chatbot (search chatbot) means a new monetization strategy. Can it make Google profitable like a search engine?

Of course Google can launch a chatbot as successful as ChatGPT, but if its profit is lower than that of a search engine, Google will face losses.

Digital marketing expert Tim Peter pointed out on Twitter that Microsoft’s advantage is that it can subsidize Bing through other revenue sources. The cost of ChatGPT

Behind ChatGPT is Microsoft, the financial backer, and Microsoft has abundant sources of income and can currently support ChatGPT to continue burning like this.

But Google is different. Almost all of its revenue comes from advertising. Therefore, without advertising revenue, the value of the Google search engine would be significantly reduced.

##Microsoft’s fiscal year revenue by division from 2012 to 2022

What's more, the reason why search engines are so profitable is that users continue to click on links. In the process, Google charges merchants huge advertising fees.

But if users can get answers directly through chatbots without having to search in the list, the paid business ecosystem of search engines will be directly destroyed.

Does Google dare to risk smashing this cake worth hundreds of billions of dollars?

Now, Google has a smooth transition way out: launch a commercial chatbot through its DeepMind, and create a Google search by itself opponents (who are also OpenAI's opponents), so that you can be a hedge.

This way Google can appease shareholders and have a response when outsiders doubt that chatbots will replace search engines.

Looking at it this way, the Sparrow beta version to be released in 2023 has a really big mission.

The above is the detailed content of Google is so anxious that it supports DeepMind! 70 billion parameters Sparrow hard ChatGTP. For more information, please follow other related articles on the PHP Chinese website!