A new paradigm for text and image editing, a single model enables multi-text guided image editing

Brief Overview of the Paper

#Research related to image editing using text is very hot, and many recent studies are based on denoising diffusion models to improve However, few scholars continue to pay attention to GAN-related research. This article is based on the classic StyleGAN and CLIP and proposes a semantic modulation module, so that only a single model is needed for different texts to perform text-image editing.

This article first uses the existing encoder to convert the image to be edited into the latent code w in the W^ semantic space of StyleGAN, and then uses the proposed semantic modulation module to encode the latent code Perform adaptive modulation. The semantic modulation module includes semantic alignment and semantic injection modules. It first aligns the semantics between the text encoding and the latent encoding of GAN through the attention mechanism, and then injects the text information into the aligned latent encoding to ensure that the Cain encoding owns the text. Information thereby achieving the ability to edit images using text.

Different from the classic StyleCLIP model, our model does not need to train a separate model for each text. One model can respond to multiple texts to effectively edit images, so we The model becomes FFCLIP-Free Form Text-Driven Image Manipulation. At the same time, our model has achieved very good results on the classic church, face and car data sets.

• ##Paper address: https://arxiv.org/pdf/2210.07883.pdf
• Github address: https://github.com/KumapowerLIU/FFCLIP

Background and inspiration

Recently, free text prompts describing user intent have been used to edit the StyleGAN latent space for image editing operations [1, 2]. Taking as input a sentence (e.g., ‘Blue’) or a phrase (e.g., ‘Man aged 10’), these methods edit the described image attributes accordingly by modulating the latent encoding in the StyleGAN latent space.

Precise text-image editing relies on an accurate latent mapping between StyleGAN’s visual semantic space and CLIP’s textual semantic space. For example, when the text prompt is "surprise", we first identify its related semantic subspace (i.e. "expression", because surprise belongs to the attribute of expression) in the visual semantic space. After finding the semantic subspace corresponding to the text, the text will tell us the direction in which the latent encoding changes, from the current expression to the surprise expression. Pioneering studies such as TediGAN [1] and StyleCLIP [2] empirically predefined which latent visual subspace corresponds to the target textual hint embedding (i.e., specific attribute selection in TediGAN and grouping mapping in StyleCLIP). This empirical recognition constrains that given a text prompt, they must train a corresponding editing model.

Different text cues require different models to modulate the latent codes in the latent visual subspace of StyleGAN. Although the global orientation method in StyleCLIP does not employ such a process, parameter adjustments and editing orientations are manually predefined. For this reason, we have reason to explore how to automatically find the implicit visual semantic subspace through explicit text, so that a single model can handle multiple texts.

In this paper, we propose FFCLIP-Free Form CLIP, which can automatically find the corresponding visual subspace for different texts. FFCLIP consists of several semantic modulation modules that take as input the latent encoding w^ and the text encoding e in the StyleGAN latent space W^.

The semantic modulation module consists of a semantic alignment module and a semantic injection module. The semantic alignment module takes the text encoding e as the query and the latent encoding w as the key and value. Then we calculate cross-attention in the position and channel dimensions respectively, resulting in two attention maps. Then we use linear transformation to transform the current visual space into the subspace corresponding to the text, where the linear transformation parameters (i.e., translation and scaling parameters) are calculated based on these two attention maps. Through this alignment, we can automatically find the corresponding visual subspace for each text. Finally, the semantic injection module [3] modifies the latent code in the subspace by following another linear transformation.

From an FFCLIP perspective, [1, 2] neutron space empirical selection is a special form of our linear transformation in the semantic alignment module. Their group selection operation is similar to the binary values ​​of our scaling parameters to indicate the usage of each position dimension of w. On the other hand, we observe that the semantics of W^ space are still entangled, and empirical design cannot find an accurate mapping between the latent space of StyleGAN and the textual semantic space of CLIP. Instead, the scaling parameter in our semantic alignment module adaptively modifies the latent code w to map different textual cue embeddings. The alignment is then further improved via our translation parameters. We evaluate our method on benchmark datasets and compare FFCLIP with state-of-the-art methods. The results show that FFCLIP is able to generate more reasonable content while conveying user intent.

FFCLIP

Figure 1 shows our overall framework. FFCLIP first obtains the latent encoding of images and texts through the pre-trained GAN inversion encoder and text encoder. The latent encoding of the image is w in the previously mentioned StyleGAN visual semantic space W^, and the text encoding is e_t . Like StyleCLIP, we use the e4e GAN inversion encoder [4] and the text encoder in CLIP to obtain the corresponding latent encoding respectively. Then we use e_t and w as the input of the modulation module and output the offset Δw of w. Finally, add Δw to the original w and put it into the pre-trained StyleGAN to get the corresponding result.

##Figure 1: Overall framework diagram

Figure 2 below is our semantic modulation module. In the semantic alignment module (Semantic Alignment), we can clearly see that we set Δw to Key and Value and set e_t to Query to calculate two attention maps. The sizes of these two attention maps are 18×1 respectively. and 512×512. Then we use the 18×1 attention map as the scaling coefficient S in the linear transformation. Our process of calculating the attention map is as follows:

At the same time, we After multiplying the 512×512 attention map by Value, the translation coefficient T in the explicit transformation is obtained through the Pooling operation. Our process of calculating the attention map is as follows:

After we have the translation and scaling coefficients, we can find the phase for the current text e_t through linear transformation For the corresponding visual subspace, the calculation steps are as follows:

Midterm x_i is the output result of our i-th semantic modulation module. Since the size of Δw is 18×512, the attention maps of 18×1 and 512×512 are calculated in the two dimensions of position and channel of Δw respectively. This operation is similar to Dual Attention [5].

Figure 2: Semantic modulation module

We can obtain the visual subspace corresponding to the text through the above operations. Then we use a method similar to AdaIN to inject text information into this space to obtain the final result. We call this operation the semantic injection module (Semantic Injection). The implementation steps of the entire module are as follows:

In the end, a total of 4 semantic modulation modules were stacked in our FFCLIP, and finally the final offset Δw was obtained.

Experimental results

##Figure 3: Visual comparison chart

As shown in Figure 3, we made a visual comparison with StyleCLIP [1], TediGAN [2] and HairCLIP [3]: it can be seen that FFCLIP can better reflect the semantics of the text , and generate more realistic edited images. At the same time, the corresponding numerical comparison results are shown in the table below. Our method can achieve the best results in both objective and subjective values.

Table 1: Numerical comparison

At the same time, our method also shows very good robustness. FFCLIP has not seen word combinations during training but uses single words for training. However, in testing, it can perform image processing based on the semantics of word groups very well. Edit, the visual effect is shown in Figure 4.

##Figure 4: Phrase editing

For more experimental results and ablation experiments, please see the original text.

Summary

In this paper we propose FFCLIP, a new method for efficient image editing that can target different texts but only requires a single model. The motivation of this article is that existing methods match the current text and the semantic subspace of GAN based on existing experience, so an editing model can only handle one text prompt. We improve latent mapping through alignment and injected semantic modulation. It facilitates one editing model to handle multiple text prompts. Experiments on multiple datasets demonstrate that our FFCLIP effectively produces semantically relevant and visually realistic results.

The above is the detailed content of A new paradigm for text and image editing, a single model enables multi-text guided image editing. For more information, please follow other related articles on the PHP Chinese website!