Using diffusion model to generate network parameters, LeCun praises You Yang's team's new research

If you have been shocked by the videos generated by Sora, then you have seen the huge potential of diffusion models in visual generation. Of course, the potential of the diffusion model does not stop here. It also has promising application prospects in many other fields. For more cases, please refer to the recent report on this site "The technology behind Explosion Sora" Review of the latest development directions of diffusion models》.

Recently, research conducted by You Yang’s team at the National University of Singapore, the University of California, Berkeley, and Meta AI Research discovered a new application of the diffusion model: used to generate model parameters for neural networks.

• Paper address: https://arxiv.org/pdf/2402.13144.pdf

• Project Address: https://github.com/NUS-HPC-AI-Lab/Neural-Network-Diffusion

• Paper title: Neural Network Diffusion

This approach seems to make it possible to easily generate new models using existing neural networks! Yann LeCun appreciates this and shares it. The generated model not only maintains the performance of the original model, but may even surpass it.

Diffusion models were originally derived from the concept of non-equilibrium thermodynamics. In 2015, Jascha Sohl-Dickstein et al. in their paper "Deep Unsupervised Learning using Nonequilibrium Thermodynamics" first used a diffusion process to gradually remove noise from the input, resulting in clear images.

Subsequent research work such as DDPM and DDIM optimized the diffusion model and gave its training paradigm the distinct characteristics of forward and reverse processes.

At that time, the quality of images generated by diffusion models was not yet ideal.

GuidedDiffusion This work conducts extensive ablation studies and discovers a better architecture; this pioneering work begins to allow diffusion models to surpass GAN-based methods in image quality. Later models such as GLIDE, Imagen, DALL·E 2 and Stable Diffusion can already generate photorealistic images.

Although diffusion models have achieved great success in the field of vision generation, their potential in other fields is relatively underexploited.

This recent study by the National University of Singapore, the University of California, Berkeley, and Meta AI Research has discovered an amazing ability of the diffusion model: generating high-performance model parameters.

You must know that this task is fundamentally different from traditional visual generation tasks! The parameter generation task focuses on creating neural network parameters that perform well on a given task. Researchers have previously explored this task from a priori and probabilistic modeling perspectives, such as stochastic neural networks and Bayesian neural networks. However, no one has previously studied the use of diffusion models to generate parameters.

As shown in Figure 1, carefully observing the training process and diffusion model of the neural network, we can find that the diffusion-based image generation method and the stochastic gradient descent (SGD) learning process have some things in common: 1) The neural network Both the training process and the reverse process of the diffusion model can be viewed as the process of converting from random noise/initialization to a specific distribution; 2) By adding noise multiple times, high-quality images and high-performance parameters can be degraded to a simple distribution, such as Gaussian distribution.

The team proposed a new method for parameter generation based on the above observations: neural network diffusion, abbreviated as p-diff, where p refers to parameters.

The idea of ​​this method is very straightforward, which is to use the standard implicit diffusion model to synthesize the parameter set of the neural network, because the diffusion model can convert a given random distribution into a specific distribution.

Their approach is simple: use a combination of an autoencoder and a standard latent diffusion model to learn high-performance parameter distributions.

First, for a subset of model parameters trained using the SGD optimizer, an autoencoder is trained to extract latent representations of these parameters. Then, a standard latent diffusion model is used to synthesize latent representations starting from noise. Finally, the trained autoencoder is used to process the synthesized latent representation to obtain new high-performance model parameters.

This new method exhibits these two characteristics: 1) On multiple data sets and architectures, its performance is comparable to its training data (i.e., the model trained by the SGD optimizer) within seconds , and even surpass it; 2) The generated model is quite different from the trained model, which shows that the new method can synthesize new parameters instead of memorizing training samples.

Neural network diffusion

#Introduction to diffusion model

Diffusion model usually consists of forward and reverse processes , these processes form a multi-step chain process and can be indexed by time steps.

Forward process. Given a sample x_0 ∼ q(x), the forward process is to gradually add Gaussian noise in T steps to obtain x_1, x_2...x_T.

Reverse process. Unlike the forward process, the goal of the backward process is to train a denoising network that can recursively remove noise in x_t. The process is the reverse of multiple steps, where t decreases from T all the way down to 0.

Overview of Neural Network Diffusion Methods

Neural Network Diffusion (p-diff) The goal of this new method is to generate high-performance parameters based on random noise. As shown in Figure 2, this method consists of two processes: parameter autoencoder and parameter generation.

Given a set of trained high-performance models, first select a subset of its parameters and flatten them into a one-dimensional vector.

After that, an encoder is used to extract the implicit representations of these vectors, and a decoder is responsible for reconstructing the parameters based on these implicit representations.

Then, a standard latent diffusion model is trained to synthesize this latent representation based on random noise.

After training, p-diff can be used to generate new parameters through such a chain process: random noise → reverse process → trained decoder → generated parameters.

Experiment

The team gave detailed experimental settings in the paper, which can help other researchers reproduce their results. For details, please see the original paper. We will update it here. Focus on its results and ablation studies.

Results

Table 1 is a comparison of results with two baseline methods on 8 datasets and 6 architectures.

Based on these results, the following observations can be made: 1) In most experimental cases, the new method can achieve results comparable to or better than the two baseline methods. This shows that the newly proposed method can efficiently learn the distribution of high-performance parameters and generate better models based on random noise. 2) The new method performs well on multiple different data sets, which shows that this method has good generalization performance.

Ablation research and analysis

Table 2(a) shows the results of different training data sizes (i.e., the number of original models) Influence. As can be seen, the performance difference between the best results for different numbers of original models is actually not that big.

To study the effectiveness of p-diff at other normalization layer depths, the team also explored the performance of new methods for synthesizing other shallow parameters. To guarantee an equal number of BN parameters, the team implemented the newly proposed method for three sets of BN layers (which are located between layers of different depths). The experimental results are shown in Table 2(b). It can be seen that the performance (best accuracy) of the new method is better than that of the original model at all depths of BN layer settings.

The purpose of noise enhancement is to improve the robustness and generalization ability of trained autoencoders. The team conducted ablation studies on the application of noise enhancement to input parameters and implicit representations. The results are shown in Table 2(c).

Previously, experiments have evaluated the effectiveness of new methods in synthesizing a subset of model parameters (i.e., batch normalization parameters). So we can't help but ask: Can the overall parameters of the model be synthesized using this method?

To answer this question, the team conducted experiments using two small architectures: MLP-3 and ConvNet-3. Among them, MLP-3 contains three linear layers and ReLU activation function, and ConvNet-3 contains three convolutional layers and one linear layer. Unlike the previously mentioned training data collection strategy, the team trained these architectures from scratch based on 200 different random seeds.

Table 3 gives the experimental results, where the new method is compared with two baseline methods (original method and ensemble method). It reports the comparison of results and number of parameters of ConvNet-3 on CIFAR-10/100 and MLP-3 on CIFAR-10 and MNIST.

These experiments demonstrate the effectiveness and generalization ability of the new method in synthesizing the overall model parameters, which means that the new method achieves performance comparable to or better than the baseline method. . These results can also demonstrate the practical application potential of the new method.

But the team also stated in the paper that it is currently unable to synthesize the overall parameters of large architectures such as ResNet, ViT and ConvNeXt. This is mainly limited by the limits of GPU memory.

As for why this new method can effectively generate neural network parameters, the team also tried to explore and analyze the reasons. They trained ResNet-18 from scratch using 3 random seeds and visualized its parameters as shown in Figure 3.

They obtained heat maps of parameter distributions of different layers through the min-max normalization method. Based on the visualization results of the convolutional layer (Conv.-layer2) and the fully connected layer (FC-layer18), it can be seen that certain parameter patterns do exist in these layers. By learning these patterns, the new method can generate high-performance neural network parameters.

p-diff Is it purely based on memory?

p-diff seems to be able to generate neural network parameters, but does it generate parameters or just remember them? The team did some research on this and compared the differences between the original model and the generated model.

In order to make a quantitative comparison, they proposed a similarity index. Simply put, this indicator determines the similarity between two models by calculating the Intersection over Union (IoU) ratio of their incorrect prediction results. Then they did some comparative studies and visualizations based on that. The comparison results are shown in Figure 4.

Figure 4(a) reports the similarity comparison between the original model and the p-diff model, which involves 4 comparison schemes.

It can be seen that the difference between the generated models is much larger than the difference between the original models. In addition, the maximum similarity between the original model and the generated model is also lower than the similarity between the original models. This is enough to show that p-diff can generate new parameters that are different from its training data (i.e. the original model).

The team also compared the new method with fine-tuned models and models with added noise. The results are shown in Figure 4(b).

It can be seen that it is difficult for the fine-tuned model and the model with added noise to surpass the original model. Furthermore, the similarity between the fine-tuned model or the model with added noise and the original model is very high, indicating that these two operating methods cannot obtain completely new and high-performance models. However, the models generated by the new method show various similarities and better performance than the original model.

The team also compared implicit representations. The results are shown in Figure 4(c). As can be seen, p-diff can generate a completely new latent representation, while adding noise methods only interpolate around the latent representation of the original model.

The team also visualized the trajectory of the p-diff process. Specifically, they plotted the parameter trajectories generated at different time steps of the inference phase. Figure 5(a) shows 5 trajectories (using 5 different random noise initializations). The red center in the figure is the average parameter of the original model, and the gray area is its standard deviation (std).

#As the time steps increase, the generated parameters will be closer to the original model as a whole. However, it can also be seen that the end points of these trajectories (orange triangles) are still some distance from the average parameters. In addition, the shapes of these five trajectories are also very diverse.

Finally, the team studied the impact of the number of original models (K) on the diversity of the generated models. Figure 5(b) visually shows the maximum similarity between the original model and the generated model for different K. Specifically, they generated 50 models by continuing to generate parameters until the generated 50 models performed better than 76.5% in all cases.

It can be seen that when K=1, the similarity is very high and the range is narrow, indicating that the model generated at this time basically memorizes the parameters of the original model. As K increases, the similarity range also becomes larger, indicating that the new method can generate different parameters than the original model.

The above is the detailed content of Using diffusion model to generate network parameters, LeCun praises You Yang's team's new research. For more information, please follow other related articles on the PHP Chinese website!