The father of LSTM once again challenged LeCun: Your five points of 'innovation' were all copied from me! But unfortunately, 'I can't read it back'

Recently, Jürgen Schmidhuber, the father of LSTM, had a disagreement with LeCun again!

In fact, students who were a little familiar with this grumpy old man before knew that there had been unpleasantness between the maverick Jürgen Schmidhuber and several big names in the machine learning community.

Especially when "those three people" won the Turing Award together, but Schmidhuber did not, the old man became even more angry...

After all, Schmidhuber has always believed that these current ML leaders, such as Bengio, Hinton, LeCun, including the father of "GAN" Goodfellow and others, many of their so-called "pioneering achievements" were first proposed by him. came out, and these people didn't mention him at all in the paper.

To this end, Schmidhuber once wrote a special article to criticize the review article "Deep Learning" published by Bengio, Hinton, and LeCun in Nature in 2015.

Mainly talking about the results in this article, which things were mentioned first by him, and which things were mentioned first by other seniors. Anyway, it was not the three authors who mentioned it first.

Why are they arguing again?

Going back to the cause of this incident, it was actually a tweet sent by LeCun in September.

The content is a response to Professor David Chalmers' question: "What is the most important intellectual breakthrough (new idea) in AI in the past ten years?"

On October 4th, Schmidhuber wrote an article on his blog angrily: Most of these five "best ideas" came from my laboratory, and they were proposed in a long time. Far earlier than the "10 years" time point.

In the article, Schmidhuber listed six pieces of evidence in detail to support his argument.

But probably because too few people saw it, Schmidhuber tweeted again on November 22 to stir up this "cold rice" again Read it again.

However, compared to the last time, which was quite a heated argument, LeCun didn’t even pay attention to it this time...

The father of LSTM presents "six major evidences"

1. "Self-supervised learning" that automatically generates labels through neural networks (NN):It goes back at least to my work in 1990-91.

(I) Self-supervised object generation in a recurrent neural network (RNN) via predictive coding to learn to compress data sequences at multiple time scales and levels of abstraction.

Here, an "automaton" RNN learns the pre-task of "predicting the next input" and sends unexpected observations in the incoming data stream as targets to the "analyzer" block machine" RNN, which learns higher-level regularities and subsequently refines its acquired predictive knowledge back into an automaton through appropriate training objectives.

This greatly facilitates the downstream deep learning task of sequence classification that was previously unsolvable.

(II) Self-supervised annotation generation via GAN-type intrinsic motivation, where a world model NN learns to predict adversarial, annotation generation The behavioral consequences of the experimentally invented controller NN.

In addition, the term "self-supervision" has already appeared in the title of the paper I published in 1990.

But, this word was also used in an earlier (1978) paper...

2. "ResNets": is actually the Highway Nets I proposed early on. But LeCun thinks that the intelligence of ResNets is "not deep", which makes me very sad.

Before I proposed Highway Nets, feedforward networks only had a few dozen layers (20-30 layers) at most, and Highway Nets was the first truly deep feedforward neural network, with Hundreds of layers.

In the 1990s, my LSTM brought essentially infinite depth to supervised recursive NNs. In the 2000s, LSTM-inspired Highway Nets brought depth to feedforward NNs.

As a result, LSTM has become the most cited NN in the 20th century, and Highway Nets (ResNet) is the most cited NN in the 21st century.

It can be said that they represent the essence of deep learning, and deep learning is about the depth of NN.

3. "Gating->Attention->Dynamic Connected Graph": It can be traced back to at least my Fast Weight Programmers and Key from 1991-93 -Value Memory Networks (the "Key-Value" is called "FROM-TO").

In 1993, I introduced the term "attention" as we use it today.

However, it is worth noting that the first multiplication gate in NN can be traced back to Ivakhnenko & Lapa’s deep learning machine in 1965.

4. "Differentiable memory": It can also be traced back to my Fast Weight Programmers or Key-Value Memory Networks in 1991.

Separate storage and control like in traditional computers, but in an end-to-end differential, adaptive, fully neural way (rather than in a hybrid way).

5. "Replacement equivariant module, such as multi-head self-attention->Transformer": I published it in 1991 Transformer with linearized self-attention. The corresponding term "internal spotlights of attention" dates back to 1993.

6. "GAN is the best machine learning concept in the past 10 years" ​

You mentioned The principle of this GAN (2014) was actually proposed by me in 1990 in the name of artificial intelligence curiosity.

The last time was a few months ago

In fact, this is no longer the relationship between Schmidhuber and LeCun There was a dispute for the first time this year.

In June and July, the two had a back-and-forth quarrel about an outlook report on the "Future Direction of Autonomous Machine Intelligence" published by LeCun.

On June 27, Yann LeCun published the paper "A Path Towards Autonomous Machine Intelligence" that he had been saving for several years, calling it "a work that points to the future development direction of AI."

This paper systematically talks about the issue of "how machines can learn like animals and humans" and is more than 60 pages long.

LeCun said that this article is not only his thoughts on the general direction of AI development in the next 5-10 years, but also what he plans to research in the next few years, and hopes to inspire the AI ​​community. More people come to study together.

Schmidhuber learned about the news about ten days in advance, got the paper, and immediately wrote an article to refute it.

According to Schmidhuber’s own blog post, this is what happened at the time:

On June 14, 2022, a science media released news that LeCun would release a report on June 27, and sent me a draft of the report (it was still in the confidentiality period at the time), and I was asked to comment.

I wrote a review telling them that this was basically a replica of our previous work, which was not mentioned in LeCun's article.

However, my comments fell on deaf ears.

In fact, long before his article was published, we had proposed LeCun’s so-called “main original contribution” in this article. Most of the content mainly includes:

(1) "Cognitive architecture, in which all modules are separable and many modules are trainable" (we proposed it in 1990 ).

(2) "Predicting hierarchical structures of world models, learning representations at multiple abstraction levels and multiple time scales" (we proposed in 1991).

(3) "A self-supervised learning paradigm that produces representations that are simultaneously informative and predictable" (Our model has been used in reinforcement learning and world building since 1997 Modeled)

(4) Predictive model "for hierarchical planning under uncertainty", including gradient-based neural subgoal generator (1990), abstract concepts Spatial reasoning (1997), neural networks that “learn to act primarily through observation” (2015), and learning to think (2015) were all proposed by us first.

On July 14th, Yann LeCun responded, saying that discussions must be constructive. He said this:

I don’t want to get into a situation. In this useless debate about "who invented a certain concept", you don't want to delve into the 160 references listed in your response article. I think a more constructive approach would be to identify 4 publications that you think may contain ideas and methods from the 4 contributions I listed.

As I said at the beginning of the paper, there are many concepts that have been around for a long time and neither you nor I are the inventors of them: for example, the concept of fine-tunable world models , which can be traced back to early optimization control work.

Training the World Model Using neural networks to learn system recognition of world models, this idea dates back to the late 1980s, by Michael Jordan, Bernie Widrow, Robinson & Fallside, Kumpathi Narendra, Paul Werbos The work being done precedes your work.

In my opinion, this straw man answer seems to be LeCun changing the subject and avoiding the issue of taking credit for others in his so-called "main original contribution".

I replied on July 14th:

Regarding what you said about "something neither you nor I invented": your paper claims , using neural networks for system identification can be traced back to the early 1990s. However, in your previous response you seemed to agree with me that the first papers on this appeared in the 1980s.

As for your "main original contribution", they actually used the results of my early work.

(1) Regarding the "cognitive architecture in which all modules are differentiable and many modules are trainable" you proposed, "through intrinsic motivation Driving behavior": ​

# I proposed a differentiable architecture for online learning and planning in 1990. This was the first control with "intrinsic motivation" It is used to improve the world model, which is both generative and adversarial; the 2014 GAN cited in your article is a derivative version of this model.

(2) About your proposed "hierarchical structure of predictive world models that learn representations at multiple abstraction levels and time scales": ​

This was made possible by my 1991 Neural History Compressor. It uses predictive coding to learn hierarchical internal representations of long sequence data in a self-supervised manner, greatly facilitating downstream learning. Using my 1991 neural network refinement procedure, these representations can be collapsed into a single recurrent neural network (RNN).

(3) About your "self-supervised learning paradigm in control, producing representations that are both informative and predictable": ​

This point was made in the system I proposed to build in 1997. Rather than predicting all the details of future inputs, it can ask arbitrary abstract questions and give computable answers in what you call a "representation space." In this system, two learning models named "left brain" and "right brain" select opponents with maximum rewards to engage in zero-sum games, and occasionally bet on the results of such computational experiments.

(4) Regarding your hierarchical planning predictive differentiable model that can be used under uncertainty, your article says this: ​

"One unanswered question is how the configurator learns to decompose a complex task into a series of sub-goals that can be completed by the agent alone. I will leave this question to the future Investigation."

Don’t talk about the future. In fact, I published this article more than 30 years ago:

a The controller neural network is responsible for obtaining additional command inputs in the form of (start, target). An estimator neural network is responsible for learning to predict the expected cost from start to goal. A subgoal generator based on a fine-tunable recurrent neural network sees this (start, goal) input and learns a sequence of minimal-cost intermediate subgoals via gradient descent using an estimator neural network.

# (5) You also emphasized the neural network that “learns behavior mainly through observation”. We actually solved this problem very early, in this article in 2015, which discussed the general problem of reinforcement learning (RL) in partially observable environments.

World model M may be good at predicting some things but uncertain about others. Controller C maximizes its objective function by learning to query through a self-invented sequence of questions (activation patterns) and interpret answers (more activation patterns).

C can benefit from learning to extract any kind of algorithmic information from M, such as for hierarchical planning and reasoning, leveraging passive observations encoded in M, etc.

The above is the detailed content of The father of LSTM once again challenged LeCun: Your five points of 'innovation' were all copied from me! But unfortunately, 'I can't read it back'. For more information, please follow other related articles on the PHP Chinese website!