


Another Alibaba paper called "Dance Whole Job" caused a sensation after AnimateAnyone
Now, just upload a photo of your face and describe it with a simple sentence, you can be anywhere Let’s dance!
For example, the dance video of "Cleaning the Glass" below:
Picture
All you need to do is upload a portrait photo , and fill in the corresponding prompt information
In the golden leaves of autumn, a girl is smiling and dancing in a light blue dress
As the prompts change, the background and clothes of the character will also Change accordingly. For example, we can change a few more sentences:
A girl is smiling and dancing in a wooden house. She is wearing a sweater and trousers
A girl is smiling and dancing in Times Square, Wearing a dress-like white shirt, long sleeves, and long pants.
Picture
This is Ali's latest research - DreaMoving, which focuses on letting anyone dance at any time and anywhere.
Pictures
And not only real people, but also cartoon and animation characters can be held~
Picture
As soon as the project came out, it also attracted the attention of many netizens. Some people called "Unbelievable" after seeing the effect~
Picture
So how is this result achieved? How was this research conducted?
Principle behind
Although the advent of text-to-video (T2V) models such as Stable Video Diffusion and Gen2, has made great progress in the field of video generation A major breakthrough, but there are still many challenges
For example, in terms of data sets, there is currently a lack of open source human dance video data sets and difficulty in obtaining corresponding precise text descriptions, which makes it difficult for models to generate diverse Sexuality, frame consistency, and longer videos have become challenges
And in the field of human-centered content generation, the personalization and controllability of the generated results are also key factors.
Picture
In order to deal with these two challenges, the Alibaba team first started to process the data set
The researchers first collected it from the Internet About 1000 high quality human dance videos. Then, they cut these videos into about 6,000 short videos (8 to 10 seconds each) to ensure that there are no transitions and special effects in the video clips, which is conducive to the training of the temporal model
In addition, in order to generate For the text description of the video, they used Minigpt-v2 as the video captioner (video captioner), specifically the "grounding" version. The instruction is to describe the frame in detail.
By generating subtitles based on the key frame center frame, the theme and background content of the video clip can be accurately described
In terms of framework, the Alibaba team proposed a tool called DreaMoving based on Stable Diffusion model.
It is mainly composed of three neural networks, including Denoising U-Net (Denoising U-Net), Video Control Network (Video ControlNet) and Content Guider (Content Guider).
picture
Among them, Video ControlNet is an image control network injected into the Motion Block after each U-Net block, processing the control sequence (pose or depth) into an additional temporal residual
Denoising U-Net is A derived Stable-Diffusion U-Net with motion blocks for video generation.
The Content Guider transmits the input text prompts and appearance expressions (such as faces) to the content embedding.
Through such operations, DreaMoving is able to generate high-quality, high-fidelity videos given the input of a guidance sequence and a simple content description (such as text and reference images)
Picture
But unfortunately, there is currently no open source code for the DreaMoving project.
For those who are interested in this, you can pay attention first and wait for the release of the open source code~
Please refer to the following link: [1]https://dreamoving.github.io/dreamoving /[2]https://arxiv.org/abs/2312.05107[3]https://twitter.com/ProperPrompter/status/1734192772465258499[4]https://github.com/dreamoving/dreamoving-project
The above is the detailed content of Ali innovates again: you can realize the dance of 'Cleaning the Glass' with a sentence and a human face, and the costume and background can be switched freely!. For more information, please follow other related articles on the PHP Chinese website!

简单易懂的PyCharm项目打包方法分享随着Python的流行,越来越多的开发者使用PyCharm作为Python开发的主要工具。PyCharm是功能强大的集成开发环境,它提供了许多方便的功能来帮助我们提高开发效率。其中一个重要的功能就是项目的打包。本文将介绍如何在PyCharm中简单易懂地打包项目,并提供具体的代码示例。为什么要打包项目?在Python开发

PyCharm是一款功能强大的Python集成开发环境,提供了丰富的开发工具和环境配置,让开发者能够更高效地编写和调试代码。在使用PyCharm进行Python项目开发的过程中,有时候我们需要将项目打包成可执行的EXE文件,以便在没有安装Python环境的计算机上运行。本文将介绍如何使用PyCharm将项目转换为可执行的EXE文件,同时给出具体的代码示例。首

如何在iOS17中的iPhone上制作GroceryList在“提醒事项”应用中创建GroceryList非常简单。你只需添加一个列表,然后用你的项目填充它。该应用程序会自动将您的商品分类,您甚至可以与您的伴侣或扁平伙伴合作,列出您需要从商店购买的东西。以下是执行此操作的完整步骤:步骤1:打开iCloud提醒事项听起来很奇怪,苹果表示您需要启用来自iCloud的提醒才能在iOS17上创建GroceryList。以下是它的步骤:前往iPhone上的“设置”应用,然后点击[您的姓名]。接下来,选择i

作为一个技术博主,了不起比较喜欢各种折腾,之前给大家介绍过ChatGPT接入微信,钉钉和知识星球(如果没看过的可以翻翻前面的文章),最近再看开源项目的时候,发现了一个ChatGPTWebUI项目。想着刚好之前没有将ChatGPT接入过WebUI,有了这个开源项目可以拿来使用,真是不错,下面是实操的安装步骤,分享给大家。安装官方在Github的项目文档上提供了很多中的安装方式,包括手动安装,docker部署,以及远程部署等方法,了不起在选择部署方式的时候,一开始为了简单想着

react启动项目报错的解决办法:1、进入项目文件夹,启动项目并查看报错信息;2、执行“npm install”或“npm install react-scripts”命令;3、执行“npm install @ant-design/pro-field --save”命令。

PyCharm是一款功能强大的Python集成开发环境(IDE),提供了丰富的功能帮助开发者更高效地编写和管理Python项目。在使用PyCharm开发项目的过程中,有时候我们需要删除一些不再需要的项目以释放空间或清理项目列表。本文将详细介绍如何在PyCharm中删除项目,并提供具体的代码示例。如何删除项目打开PyCharm,进入项目列表界面。在项目列表中,

IDEA(IntelliJIDEA)是一款强大的集成开发环境,可以帮助开发人员快速高效地开发各种Java应用程序。在Java项目开发中,使用Maven作为项目管理工具能够帮助我们更好地管理依赖库、构建项目等。本文将详细介绍如何在IDEA中创建一个Maven项目的基本步骤,同时提供具体的代码示例。步骤一:打开IDEA并创建新项目打开IntelliJIDEA

从零开始,快速上手PyCharm项目打包技巧概述:在Python开发中,将项目打包成可执行文件是非常重要的一步。它可以方便地分享和分发项目,而无需安装Python解释器和依赖包。PyCharm作为一个功能强大的Python集成开发环境,提供了快速上手项目打包的技巧和工具。本文将介绍如何利用PyCharm从零开始打包你的Python项目,并提供具体的代码示例。


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

PhpStorm Mac version
The latest (2018.2.1) professional PHP integrated development tool

Safe Exam Browser
Safe Exam Browser is a secure browser environment for taking online exams securely. This software turns any computer into a secure workstation. It controls access to any utility and prevents students from using unauthorized resources.

SublimeText3 English version
Recommended: Win version, supports code prompts!

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)
