Continuous reversals! DeepMind was questioned by the Russian team: How can we prove that neural networks understand the physical world?

Recently there has been another controversy in the scientific community. The protagonist of the story is a Science paper published by DeepMind's research center in London in December 2021. Researchers found that neural networks can be used to train and build models that are better than before. More accurate electron density and interaction maps can effectively solve the systematic errors in traditional functional theory.

Paper link: https://www.science.org/doi/epdf/10.1126/science.abj6511

The DM21 model proposed in the article accurately simulates complex systems such as hydrogen chains, charged DNA base pairs, and binary transition states. For the field of quantum chemistry, it can be said that it has opened up a feasible technical route to accurate universal functions.

DeepMind researchers also released the code of the DM21 model to facilitate reproduction by peers.

Warehouse link: https://github.com/deepmind/deepmind-research

Logically speaking, the papers and codes are public and published in top journals. The experimental results and research conclusions are basically reliable.

But eight months later, eight researchers from Russia and South Korea also published a scientific review in Science. They believed that there were problems in DeepMind’s original research, namelyThe training set and the test set may have overlapping parts, resulting in incorrect experimental conclusions.

Paper link: https://www.science.org/doi/epdf/10.1126/science.abq3385

If the suspicion is true, then DeepMind’s paper, which is known as a major technological breakthrough in the chemical industry, may be attributed to data leakage# for the improvements made in neural networks ##.

However, DeepMind responded quickly. On the same day the comment was published, it immediately wrote a reply to express its opposition and strong condemnation: the views they raised were either incorrect, Either is irrelevant to the main conclusions of the paper and the assessment of the overall quality of DM21.

Paper link: https://www.science.org/doi/epdf/10.1126/science.abq4282

The famous physicist Feynman once said, Scientists must prove themselves wrong as soon as possible. Only in this way can progress be made.

Although the outcome of this discussion has not been finalized and the Russian team has not published further rebuttal articles, the incident may have a more profound impact on research in the field of artificial intelligence. : That is, how to prove that the neural network model you have trained truly understands the task, rather than just memorizing the pattern? Research Question

Chemistry is the central science of the 21st century (convinced), such as designing new materials with specified properties, such as producing clean electricity or developing high temperatures Superconductors all require the simulation of electrons on a computer.

Electrons are subatomic particles that control how atoms combine to form molecules. They are also responsible for the flow of electricity in solids. Understanding the location of electrons within a molecule can go a long way toward explaining its structure and properties. and reactivity.

In 1926, Schrödinger proposed the Schrödinger equation, which can correctly describe the quantum behavior of the wave function. But using this equation to predict electrons in a molecule is insufficient because all electrons repel each other, and it is necessary to track the probability of each electron's position, which is a very complicated task even for a small number of electrons.

A major breakthrough came in the 1960s, when Pierre Hohenberg and Walter Kohn realized there was no need to track each electron individually. Instead, knowing the probability of any electron being in each position (i.e., the electron density) is enough to accurately calculate all interactions.

After proving the above theory, Kohn won the Nobel Prize in Chemistry, thus creating density functional theory (density functional theory, DFT)

Despite DFT proving that mapping exists, for more than 50 years the exact nature of the mapping between electron density and interaction energy, the so-called density functional, has remained unknown and must be solved approximately.

DFT is essentially a method of solving the Schrödinger equation, and its accuracy depends on its exchange-correlation part. Although DFT involves a certain degree of approximation, it is the only practical way to study how and why matter behaves in a certain way at the microscopic level, and has therefore become one of the most widely used techniques in all fields of science.

Over the years, researchers have proposed more than 400 approximate functions of varying degrees of accuracy, but all of these approximations suffer from systematic errors because they fail to capture some of the key mathematics of the exact functional characteristic.

When it comes to learning approximate functions, isn’t this what neural networks do?

In this paper, DeepMind trains a neural network DM 21 (DeepMind 21), successfully learned a functional without systematic errors, which can avoid delocalization errors and spin symmetry breaking, and can better describe a wide range of chemical reaction categories.

In principle, any chemical and physical process involving charge movement is prone to delocalization errors, and any process involving bond breaking is prone to occurrence. Spin symmetry broken. While charge movement and bond breaking are at the heart of many important technical applications, these problems can also lead to numerous qualitative failures in describing the functional groups of the simplest molecules, such as hydrogen.

The model is built using a multi-layer perceptron (MLP), and the input is the local and non-local images of the occupied Kohn-Sham (KS) orbit. local features.

The objective function contains two: one is the regression loss used to learn the exchange correlation energy itself, and the other is to ensure that the function derivative can be used in the self-consistent field after training. , SCF) calculated gradient regularization term.

For the regression loss, the researchers used a fixed-density data set representing the reactants and products of 2235 reactions, and trained the network to map from these densities to Highly accurate reaction energies, with 1161 training reactions representing atomization, ionization, electron affinity and intermolecular binding energies of small main group H-Kr molecules and 1074 reactions representing key FC and FS densities of H-Ar atoms .

The trained model DM21 is able to run self-consistently on all reactions of the large main family benchmark, producing more accurate molecular densities.

Really SOTA or data leak?

When DeepMind trains DM21, the data used is a fractional charge system, such as a hydrogen atom with half an electron.

To demonstrate the superiority of DM21, the researchers tested it on a set of stretched dimers, called the bond-breaking benchmark (BBB) ​​set. For example, two hydrogen atoms far apart have a total of one electron.

Experimental results found that the DM21 functional showed excellent performance on the BBB test set, surpassing all the classic DFT functionals tested so far and DM21m (same as DM21 training, but on There are no fractional charges in the training set).

Then DeepMind claimed in the paper: DM21 has understood the physical principles behind the fractional charge system.

But if you look closely, you will find that in the BBB group, all dimers become very similar to the system in the training group. In fact, due to the localized nature of electroweak interactions, atomic interactions are only strong over short distances, beyond which the two atoms behave essentially as if they were not interacting.

Michael Medvedev, research group leader at the Zelinsky Institute of Organic Chemistry of the Russian Academy of Sciences, explains that in some ways neural networks are like humans Likewise, they prefer to get the right answer for the wrong reason. So it's not hard to train a neural network, but it's hard to prove that it has learned the laws of physics rather than just memorized the correct answers.

Therefore, the BBB test set is not a suitable test set: it does not test DM21's understanding of fractional electronic systems, a thorough analysis of the other four evidences of DM21's handling of such systems Nor is a conclusive conclusion drawn: only its good accuracy on the SIE4x4 set may be reliable.

Russian researchers also believe that the use of fractional charge systems in the training set is not the only novelty in DeepMind's work. Their idea of ​​introducing physical constraints into neural networks through training sets, and the method of giving physical meaning through training on the correct chemical potentials, may be widely used in the construction of neural network DFT functionals in the future.

DeepMind Response

Regarding the Comment paper’s claim that DM21’s ability to predict fractional charge (FC) and fractional spin (FS) conditions outside the training set is not found in the paper This was demonstrated based on approximately 50% overlap of the training set with the bondbreaking benchmark BBB, as well as the effectiveness and accuracy of other generalization examples.

DeepMind disagrees with this analysis and believes that the points made are either incorrect or irrelevant to the main conclusions of the paper and the assessment of the overall quality of DM21, as BBB is not included in the paper Only example of FC and FS behavior shown.

The overlap between the training set and the test set is a research issue worthy of attention in machine learning: memory means that a model can be trained by copying Concentrated examples perform better on the test set.

Gerasimov believes that the performance of DM21 on the BBB (containing dimers at finite distances) can be achieved by replicating the output of FC and FS systems (i.e., atoms at infinite separation limits with dimers match) is well explained.

To demonstrate that DM21 generalizes beyond the training set, DeepMind researchers also considered H2 (cationic dimer) and H2 (neutral dimeric For the prototype BBB example of aggregates), it can be concluded that the exact exchange-correlation function is non-local; returning a constant memorized value can lead to significant errors in BBB predictions as distance increases.

The above is the detailed content of Continuous reversals! DeepMind was questioned by the Russian team: How can we prove that neural networks understand the physical world?. For more information, please follow other related articles on the PHP Chinese website!