Apple’s “matryoshka”-style diffusion model reduces the number of training steps by 70%!

A latest research by Apple has greatly improved the performance of diffusion models on high-resolution images.

Using this method, the number of training steps for images with the same resolution is reduced by more than 70%.

At the resolution of 1024×1024, the picture quality is directly full, and the details are clearly visible.

Apple named this achievement MDM. DM is the abbreviation of Diffusion Model, and the first M stands for Matryoshka.

Just like a real matryoshka doll, MDM nests a low-resolution process within a high-resolution process, and it is nested in multiple layers.

The high- and low-resolution diffusion processes are performed simultaneously, which greatly reduces the resource consumption of the traditional diffusion model in the high-resolution process.

For a 256×256 resolution image, in an environment with a batch size of 1024, the traditional diffusion model needs to be trained for 1.5 million steps, while the MDM only requires 390,000, a decrease of more than 70%.

In addition, MDM adopts end-to-end training and does not rely on specific data sets and pre-trained models. It improves speed while still ensuring generation quality and is flexible to use.

Not only can you draw high-resolution images, but you can also synthesize 16×256² videos.

#Some netizens commented that Apple finally connected text to images.

So, how exactly does MDM’s “matryoshka” technology work?

Combining overall and progressive

Before starting training, the data needs to be preprocessed. High-resolution images will be resampled using a certain algorithm. Get different resolution versions.

Then we use these data of different resolutions for joint UNet modeling. The small UNet handles low resolution and is nested into the large UNet that handles high resolution.

Through cross-resolution connections, features and parameters can be shared between UNets of different sizes.

#MDM training is a step-by-step process.

Although the modeling is jointly performed, the training process will not be performed for high resolution at the beginning, but will gradually expand from low resolution.

This can avoid a huge amount of calculations, and also allows the pre-training of low-resolution UNet to accelerate the high-resolution training process.

During the training process, higher-resolution training data will be gradually added to the overall process, allowing the model to adapt to the progressively increasing resolution and smoothly transition to the final high-resolution process.

However, overall, after the high-resolution process is gradually added, MDM training is still an end-to-end joint process.

In joint training at different resolutions, loss functions at multiple resolutions participate in parameter updates together, avoiding the accumulation of errors caused by multi-stage training.

Each resolution has a corresponding reconstruction loss of the data item. The losses of different resolutions are weighted and merged. In order to ensure the generation quality, the low-resolution loss has a larger weight.

In the reasoning phase, MDM also adopts a strategy that combines parallelism and progression.

In addition, MDM also uses a pre-trained image classification model (CFG) to guide the generated samples to be optimized in a more reasonable direction, and adds noise to low-resolution samples to make them closer to high-resolution samples. The distribution of the sample.

So, what is the effect of MDM?

Fewer parameters rival SOTA

In terms of images, on the ImageNet and CC12M data sets, MDM's FID (the lower the value, the better the effect) and CLIP performance are significantly better than the ordinary diffusion model.

FID is used to evaluate the quality of the image itself, and CLIP illustrates the degree of matching between the image and text instructions.

Compared with SOTA models such as DALL E and IMAGEN, the performance of MDM is also very close, but the training parameters of MDM are far less than these models.

Not only is it better than the ordinary diffusion model, the performance of MDM also exceeds that of other cascade diffusion models.

Ablation experiment results show that the more steps of low-resolution training, the more obvious the MDM effect enhancement; on the other hand, the more nesting levels, the same results will be achieved. The CLIP score requires fewer training steps.

The selection of CFG parameters is a result of the trade-off between FID and CLIP after multiple tests (a high CLIP score corresponds to an increase in CFG strength).

The above is the detailed content of Apple’s “matryoshka”-style diffusion model reduces the number of training steps by 70%!. For more information, please follow other related articles on the PHP Chinese website!