Adding special effects only requires one sentence or a picture. The company Stable Diffusion has used AIGC to play new tricks.

I believe that many people have already understood the charm of generative AI technology, especially after experiencing the AIGC outbreak in 2022. Text-to-image generation technology represented by Stable Diffusion was once popular all over the world, and countless users poured in to express their artistic imagination with the help of AI...

Compared with image editing, video Editing is a more challenging topic, requiring synthesizing new actions rather than just modifying the visual appearance, while also maintaining temporal consistency.

There are many companies exploring this track. Some time ago, Google released Dreamix to apply text conditional video diffusion model (VDM) to video editing.

Recently, Runway, a company that participated in the creation of Stable Diffusion, launched a new artificial intelligence model "Gen-1", which uses any style specified by applying text prompts or reference images. Can convert existing videos into new videos.

Paper link: https://arxiv.org/pdf/2302.03011.pdf

Project homepage: https://research.runwayml.com/gen1

In 2021, Runway and the University of Munich Researchers collaborated to build the first version of Stable Diffusion. Then Stability AI, a UK startup, stepped in to fund the computational expenses needed to train the model on more data. In 2022, Stability AI brings Stable Diffusion into the mainstream, transforming it from a research project into a global phenomenon.

Runway said it hopes Gen-1 can do for video what Stable Diffusion has done for images.

“We’ve seen an explosion of image generation models,” said Cristóbal Valenzuela, CEO and co-founder of Runway. "I really believe that 2023 will be the year of video."

Specifically, Gen-1 supports several editing modes:

1. Stylization. Transfer the style of any image or prompt to every frame of your video.

2. Storyboard. Turn your model into a fully stylized and animated rendering.

3. Mask. Isolate topics in videos and modify them using simple text prompts.

4. Rendering. Turn textureless rendering into photorealistic output by applying input images or prompts.

5. Customization. Unleash the full power of Gen-1 by customizing your model for higher-fidelity results.

In a demo posted on the company’s official website, it shows how Gen-1 can smoothly change video styles. Let’s take a look at a few examples.

For example, to turn "people on the street" into "clay puppets", you only need one line of prompt:

Or turn "books stacked on the table" into "cityscape at night":

From "running on the snow" to "walking on the moon":

The young girl, in seconds Become an ancient sage:

#

Paper Details

Visual effects and video editing are ubiquitous in the contemporary media landscape. As video-centric platforms gain popularity, the need for more intuitive and powerful video editing tools increases. However, due to the temporal nature of video data, editing in this format is still complex and time-consuming. State-of-the-art machine learning models show great promise in improving the editing process, but many methods have to strike a balance between temporal consistency and spatial detail.

Generative methods for image synthesis have recently experienced a phase of rapid growth in quality and popularity due to the introduction of diffusion models trained on large-scale datasets. Some text-conditional models, such as DALL-E 2 and Stable Diffusion, enable novice users to generate detailed images with just a text prompt. Latent diffusion models provide efficient methods for generating images by compositing in a perceptually compressed space.

In this paper, the researchers propose a controllable structure- and content-aware video diffusion model on unsubtitled video and paired text-image data. trained on large-scale data sets. We chose to use monocular depth estimation to represent structure and embeddings predicted by a pre-trained neural network to represent content.

This method provides several powerful control modes during its generation process: First, similar to image synthesis models, the researchers train the model to make inferred video content, such as its appearance or style, matching a user-supplied image or text prompt (Figure 1). Second, inspired by the diffusion process, the researchers applied an information masking process to the structural representation to be able to select how well the model supports a given structure. Finally, we tune the inference process through a custom guidance method inspired by classification-free guidance to achieve control over the temporal consistency of generated segments.

Overall, the highlights of this study are as follows:

• By introducing a temporal layer into the pre-trained image model, and Joint training on images and videos extends the latent diffusion model to the field of video generation;
• proposes a structure- and content-aware model that modifies videos under the guidance of sample images or text . Editing occurs entirely within inference time, requiring no additional training or preprocessing for each video;
• demonstrates complete control over time, content, and structural consistency. This study shows for the first time that joint training on image and video data enables inference time to control temporal consistency. For structural consistency, training at different levels of detail in the representation allows the desired settings to be selected during inference;
• In a user study, our method More popular than several other methods;
• The trained model can be further customized by fine-tuning on a small set of images to produce more accurate videos of a specific subject.

Method

For research purposes it will be helpful to consider a video from both a content and structure perspective. By structure, here we mean features that describe its geometry and dynamics, such as the shape and position of its bodies, and their temporal changes. For content, it is defined here as features that describe the appearance and semantics of a video, such as the color and style of objects and the lighting of the scene. The goal of the Gen-1 model is to edit the content of a video while preserving its structure.

In order to achieve this goal, the researcher learned the generative model p (x|s, c) of video x, whose conditions are structural representation (represented by s) and content representation ( Represented by c). They infer the shape representation s from the input video and modify it based on the text prompt c describing the edit. First, the implementation of the generative model as a conditional latent video diffusion model is described, and then, the choice of shape and content representations is described. Finally, the optimization process of the model is discussed.

The model structure is shown in Figure 2.

Experiment

To evaluate the method, the researchers used DAVIS videos and various materials. To automatically create the editing prompt, the researchers first ran a subtitle model to obtain a description of the original video content, and then used GPT-3 to generate the editing prompt.

Qualitative research

As shown in Figure 5, the results prove that the method in this article is effective on some different inputs good performance.

##User Research

Researcher also A user study was conducted using Amazon Mechanical Turk (AMT) on an evaluation set of 35 representative video editing prompts. For each sample, 5 annotators were asked to compare the fidelity of video editing prompts between the baseline method and our method ("Which video better represents the provided edited subtitles?"), and then randomly Presented sequentially, with majority vote used to determine final outcome.

The results are shown in Figure 7:

##Quantitative Evaluation

Figure 6 shows the results of each model using the consistency and prompt consistency indicators of this article's framework. The performance of the model in this paper tends to surpass the baseline model in both aspects (i.e., it is higher in the upper right corner of the figure). The researchers also noticed that there is a slight tradeoff for increasing the intensity parameter in the baseline model: greater intensity scaling means higher prompt consistency at the cost of lower frame consistency. They also observed that increasing structural scaling leads to higher prompt consistency because the content becomes no longer determined by the input structure.

Customization

Figure 10 shows a model with different numbers of customization steps and different levels of structural dependencies. ts example. The researchers observed that customization increases fidelity to the character's style and appearance, so that, despite using driven videos of characters with different characteristics, combined with higher ts values, accurate animation effects can be achieved.

The above is the detailed content of Adding special effects only requires one sentence or a picture. The company Stable Diffusion has used AIGC to play new tricks.. For more information, please follow other related articles on the PHP Chinese website!