The magic of Google's new 'AI Director' is that it can change the protagonist of the video with just one sentence, which is amazing, and the picture quality is also very good.

This article is reprinted with the authorization of AI New Media Qubit (public account ID: QbitAI). Please contact the source for reprinting.

Google has created a new "AI director" , can even change the protagonist of the video in one sentence.

Look, a little bear is dancing on the green grass.

Are all the bears these days so artistic? ?

No, No, No! What was originally on the grass was actually a monkey.

To change from a monkey to a bear, just say one sentence to this AI:

A little bear dances to the beat of the music Dance, twisting his whole body. （A bear dancing and jumping to upbeat music, moving his whole body）

In addition to "magically modifying" videos, this AI named Dreamix can also Turn static pictures into animations - can be done in just one sentence.

For example, show this AI a "turtle swimming photo" and tell it:

A turtle was photographed underwater, and a shark was approaching from behind. （Underwater shot of a sea turle with a shark apporching from behind）

Good guy, one sentence not only makes the turtle swim Got up and added a shark out of thin air.

This effect made many onlookers like it.

Some people even assert that AIGC will continue to set off a craze in the next two years, even crazier than the development during the millennium.

Directed the video with just one sentence

It will be praised once it is released. How is this AI? You might as well look at more of his "director" works to get a feel for it.

First of all, in terms of changing video roles, this is the original field:

This is the AI A field set on fire:

This is a human hand writing:

This is an AI-generated robot hand writing:

It is also based on a video of a human writing. If the prompt sentence is replaced by "a human hand is drawing a circle", there will be a difference. The generation effect:

And in terms of static image changing to animation, the original image is a foggy jungle:

The AI ​​added a running unicorn to the forest, and the camera zoomed out according to sentence prompts.

There is also such a river valley scenery picture:

AI not only made the stream flow, but also added bathing buffaloes to the shore and flying birds to the sky.

#After seeing this, some people may feel that it is a bit lacking: the animation has been made, but the picture quality has also been sacrificed a lot.

Then you might as well show the AI ​​a few more pictures.

For example, show the AI ​​7 photos of toy fire alarms in one breath:

and then let it generate a video based on a sentence, The picture quality will be much clearer now.

As for how this "AI director" does it, Google said that the key lies in the "old friend" Diffusion model （Diffusion Model）.

The diffusion model is also the core of the popular AIGC painting tool DALL·E 2.

Google researchers pointed out that in fact, there has been a similar "text-generated video" AI before, but if the video diffusion model is only fine-tuned on the input video, it will limit the extent of motion changes.

What is different about this AI is that:

The team uses a "Mixed Target". In addition to fine-tuning the original target, it will also Frame sets are fine-tuned.

They adopted a specialized attention mechanism in deep learning: Masked Temporal Attention, which helps the model focus on specific parts of the input information and ignore other irrelevant parts.

——This improves the model's ability to process sequence data, generates more diverse dynamics in the video, and the effect is more natural.

With the help of the diffusion model and Masked Temporal Attention, for changing the video protagonist, the input has actually been omitted - just perform Fine-tuned, the fidelity of the results is pretty good too.

The above is the detailed content of The magic of Google's new 'AI Director' is that it can change the protagonist of the video with just one sentence, which is amazing, and the picture quality is also very good.. For more information, please follow other related articles on the PHP Chinese website!