Soft Diffusion: Google's new framework correctly schedules, learns and samples from a universal diffusion process

We know that score-based models and denoising diffusion probability models (DDPM) are two powerful types of generative models that generate samples by inverting the diffusion process. These two types of models have been unified into a single framework in the paper "Score-based generative modeling through stochastic differential equations" by Yang Song and other researchers, and are widely known as diffusion models.

At present, the diffusion model has achieved great success in a series of applications including image, audio, video generation and solving inverse problems. In the paper "Elucidating the design space of diffusionbased generative models", researchers such as Tero Karras analyzed the design space of the diffusion model and identified three stages, namely i) selecting the scheduling of the noise level, ii) selecting the network parameters. ization (each parameterization generates a different loss function), iii) design the sampling algorithm.

Recently, in an arXiv paper "Soft Diffusion: Score Matching for General Corruptions" jointly conducted by Google Research and UT-Austin, several researchers believe that the diffusion model still has a Important step: corruption. Generally speaking, corruption is a process of adding noise of different amplitudes, and for DDMP also requires rescaling. Although there have been attempts to use different distributions for diffusion, a general framework is still lacking. Therefore, the researchers proposed a diffusion model design framework for a more general damage process.

Specifically, they proposed a new training objective called Soft Score Matching and a novel sampling method, Momentum Sampler. Theoretical results show that for damage processes that satisfy regularity conditions, Soft Score MatchIng is able to learn their scores (i.e., likelihood gradients) that diffusion must transform any image into any image with non-zero likelihood.

In the experimental part, the researchers trained the model on CelebA and CIFAR-10. The model trained on CelebA achieved the SOTA FID score of the linear diffusion model - 1.85. At the same time, the model trained by the researchers is significantly faster than the model trained using the original Gaussian denoising diffusion.

##Paper address: https://arxiv.org/pdf/2209.05442.pdf

Method Overview

Generally speaking, diffusion models generate images by inverting a damage process that gradually increases noise. The researchers show how to learn to invert diffusion involving linear deterministic degradation and stochastic additive noise.

#Specifically, the researchers demonstrated a framework for using a more general damage model to train a diffusion model, which consists of three parts, each for new training objectives. Soft Score Matching, novel sampling method Momentum Sampler, and scheduling of damage mechanisms.

Let’s first look at the training target Soft Score Matching. The name is inspired by soft filtering, which is a photography term that refers to a filter that removes fine details. It learns the fraction of a conventional linear damage process in a provable way, also incorporates a filtering process into the network, and trains the model to predict images after damage that match diffusion observations.

This training objective can prove that the score is learned as long as diffusion assigns non-zero probability to any clean, corrupted image pair. Additionally, this condition is always satisfied when additive noise is present in the damage.

Specifically, the researchers explored the damage process in the following form.

In the process, the researchers discovered that noise has both empirical (i.e., better results) and theoretical (i.e., for learning fractions) benefits. Very important. This also becomes a key difference from Cold Diffusion, a concurrent work that reverses deterministic corruption.

The second is the sampling method Momentum Sampling. The researchers demonstrated that the choice of sampler has a significant impact on the quality of the generated samples. They proposed Momentum Sampler for inverting a universal linear damage process. The sampler uses convex combinations of damage with different diffusion levels and is inspired by momentum methods in optimization.

This sampling method is inspired by the continuous formulation of the diffusion model proposed in the paper by Yang Song et al. above. The algorithm for Momentum Sampler is shown below.

The following figure visually shows the impact of different sampling methods on the quality of the generated samples. The image sampled with Naive Sampler on the left seems repetitive and lacks detail, while the Momentum Sampler on the right significantly improves the sampling quality and FID score.

The last thing is scheduling. Even if the type of degradation is predefined (like blurring), deciding how much to damage at each diffusion step is not trivial. The researchers propose a principled tool to guide the design of damage processes. To find the schedule, they minimize the Wasserstein distance between distributions along the path. Intuitively, researchers want a smooth transition from a completely corrupted distribution to a clean distribution.

Experimental Results

The researchers evaluated the proposed method on CelebA-64 and CIFAR-10, both of which are standard baselines for image generation. The main purpose of the experiment is to understand the role of damage type.

The researchers first tried to use blur and low-amplitude noise for damage. The results show that their proposed model achieves SOTA results on CelebA, i.e., an FID score of 1.85, outperforming all other methods that only add noise and possibly rescale the image. In addition, the FID score obtained on CIFAR-10 is 4.64, which is competitive even though it does not reach SOTA.

In addition, on the CIFAR-10 and CelebA data sets, the researcher's method also performed better on another indicator, sampling time. Another added benefit is significant computational advantages. Deblurring (almost no noise) appears to be a more efficient manipulation compared to image generation denoising methods.

The graph below shows how the FID score changes with the Number of Function Evaluations (NFE). As can be seen from the results, our model can achieve the same or better quality than the standard Gaussian denoising diffusion model using significantly fewer steps on the CIFAR-10 and CelebA datasets.

The above is the detailed content of Soft Diffusion: Google's new framework correctly schedules, learns and samples from a universal diffusion process. For more information, please follow other related articles on the PHP Chinese website!