HKUST & MSRA Research: Regarding image-to-image conversion, Finetuning is all you need

Many content production projects require converting simple sketches into realistic pictures, which involves image-to-image translation, which uses deep generative model learning Conditional distribution of natural images given input.

The basic concept of image-to-image conversion is to utilize pre-trained neural networks to capture natural image manifolds. Image transformation is similar to traversing the manifold and locating feasible input semantic points. The system pre-trains the synthetic network using many images to provide reliable output from any sampling of its latent space. Through the pre-trained synthetic network, downstream training adapts user input to the model’s latent representation.

Over the years, we have seen many task-specific methods reach SOTA level, but current solutions still struggle to create high-fidelity images for real-world use.

In a recent paper, researchers from Hong Kong University of Science and Technology and Microsoft Research Asia believe that for image-to-image conversion, pre-training is All you need . Previous methods require specialized architecture design and training a single transformation model from scratch, making it difficult to generate complex scenes with high quality, especially when paired training data is insufficient.

Therefore, the researchers treat each image-to-image translation problem as a downstream task and introduce a simple general framework that adopts a pre-trained diffusion model to adapt to various image-to-image translations. They called the proposed pre-trained image-to-image translation model PITI (pretraining-based image-to-image translation). In addition, the researchers also proposed to use adversarial training to enhance texture synthesis in diffusion model training, and combine it with normalized guided sampling to improve the generation quality.

Finally, the researchers conducted extensive empirical comparisons on various tasks on challenging benchmarks such as ADE20K, COCO-Stuff, and DIODE, demonstrating that PITI-synthesized images display unprecedented realism and fidelity. Spend.

• Paper link: https://arxiv.org/pdf/2205.12952.pdf
• Project homepage: https://tengfei-wang .github.io/PITI/index.html

GAN is dead, the diffusion model lives on

The author did not use the GAN that performed best in a specific field, but used the diffusion model , synthesizes a wide variety of images. Second, it should generate images from two types of latent codes: one that describes visual semantics and another that adjusts for image fluctuations. Semantic, low-dimensional latent is critical for downstream tasks. Otherwise, it would be impossible to transform the modal input into a complex latent space. Given this, they used GLIDE, a data-driven model that can generate different images, as a pretrained generative prior. Since GLIDE uses latent text, it allows for a semantic latent space.

Diffusion and score-based methods demonstrate generation quality across benchmarks. On class-conditional ImageNet, these models compete with GAN-based methods in terms of visual quality and sampling diversity. Recently, diffusion models trained with large-scale text-image pairings have shown surprising capabilities. A well-trained diffusion model can provide a general generative prior for synthesis.

Framework

The author can use the pretext task to pre-train on large amounts of data and develop a very meaningful latent space to predict images statistics.

For downstream tasks, they conditionally fine-tune the semantic space to map task-specific environments. The machine creates believable visuals based on pre-trained information.

The author recommends using semantic input to pre-train the diffusion model. They use text-conditioned,image-trained GLIDE model. The Transformer network encodes text input and outputs tokens for the diffusion model. As planned, it makes sense for the text to be embedded in the space.

The picture above is the author’s work. Pretrained models improve image quality and diversity compared to techniques from scratch. Since the COCO dataset has numerous categories and combinations, the basic approach cannot provide beautiful results with a compelling architecture. Their method can create rich details with precise semantics for difficult scenes. Pictures illustrate the versatility of their approach.

Experiments and Impact

Table 1 shows that the performance of the method proposed in this study is always better than other models. Compared with the leading OASIS, PITI achieves significant improvements in FID in mask-to-image synthesis. Furthermore, the method also shows good performance in sketch-to-image and geometry-to-image synthesis tasks.

# Figure 3 shows the visualization results of this study on different tasks. Experiments show that compared with the method of training from scratch, the pre-trained model significantly improves the quality and diversity of generated images. The methods used in this study can produce vivid details and correct semantics even for challenging generation tasks.

The research also conducted a user study on mask-to-image synthesis on COCO-Stuff on Amazon Mechanical Turk, with 3000 results from 20 participants ticket. Participants were given two images at a time and asked to vote on which one was more realistic. As shown in Table 2, the proposed method outperforms the model from scratch and other baselines by a large extent.

Conditional image synthesis creates high-quality pictures that meet certain conditions. The fields of computer vision and graphics use it to create and manipulate information. Large-scale pretraining improves image classification, object recognition, and semantic segmentation. What is unknown is whether large-scale pretraining is beneficial for general generation tasks.

Energy usage and carbon emissions are key issues in image pre-training. Pre-training is energy-intensive, but only required once. Conditional fine-tuning allows downstream tasks to use the same pre-trained model. Pre-training allows generative models to be trained with less training data, improving image synthesis when data is limited due to privacy issues or expensive annotation costs.

The above is the detailed content of HKUST & MSRA Research: Regarding image-to-image conversion, Finetuning is all you need. For more information, please follow other related articles on the PHP Chinese website!