search
HomeTechnology peripheralsAIAli innovates again: you can realize the dance of 'Cleaning the Glass' with a sentence and a human face, and the costume and background can be switched freely!

Another Alibaba paper called "Dance Whole Job" caused a sensation after AnimateAnyone

Now, just upload a photo of your face and describe it with a simple sentence, you can be anywhere Let’s dance!

For example, the dance video of "Cleaning the Glass" below:

Ali innovates again: you can realize the dance of Cleaning the Glass with a sentence and a human face, and the costume and background can be switched freely!Picture

All you need to do is upload a portrait photo , and fill in the corresponding prompt information

In the golden leaves of autumn, a girl is smiling and dancing in a light blue dress

As the prompts change, the background and clothes of the character will also Change accordingly. For example, we can change a few more sentences:

A girl is smiling and dancing in a wooden house. She is wearing a sweater and trousers

A girl is smiling and dancing in Times Square, Wearing a dress-like white shirt, long sleeves, and long pants.

Ali innovates again: you can realize the dance of Cleaning the Glass with a sentence and a human face, and the costume and background can be switched freely!Picture

This is Ali's latest research - DreaMoving, which focuses on letting anyone dance at any time and anywhere.

Ali innovates again: you can realize the dance of Cleaning the Glass with a sentence and a human face, and the costume and background can be switched freely!Pictures

And not only real people, but also cartoon and animation characters can be held~

Ali innovates again: you can realize the dance of Cleaning the Glass with a sentence and a human face, and the costume and background can be switched freely! Picture

As soon as the project came out, it also attracted the attention of many netizens. Some people called "Unbelievable" after seeing the effect~

Ali innovates again: you can realize the dance of Cleaning the Glass with a sentence and a human face, and the costume and background can be switched freely! Picture

So how is this result achieved? How was this research conducted?

Principle behind

Although the advent of text-to-video (T2V) models such as Stable Video Diffusion and Gen2, has made great progress in the field of video generation A major breakthrough, but there are still many challenges

For example, in terms of data sets, there is currently a lack of open source human dance video data sets and difficulty in obtaining corresponding precise text descriptions, which makes it difficult for models to generate diverse Sexuality, frame consistency, and longer videos have become challenges

And in the field of human-centered content generation, the personalization and controllability of the generated results are also key factors.

Ali innovates again: you can realize the dance of Cleaning the Glass with a sentence and a human face, and the costume and background can be switched freely!Picture

In order to deal with these two challenges, the Alibaba team first started to process the data set

The researchers first collected it from the Internet About 1000 high quality human dance videos. Then, they cut these videos into about 6,000 short videos (8 to 10 seconds each) to ensure that there are no transitions and special effects in the video clips, which is conducive to the training of the temporal model

In addition, in order to generate For the text description of the video, they used Minigpt-v2 as the video captioner (video captioner), specifically the "grounding" version. The instruction is to describe the frame in detail.

By generating subtitles based on the key frame center frame, the theme and background content of the video clip can be accurately described

In terms of framework, the Alibaba team proposed a tool called DreaMoving based on Stable Diffusion model.

It is mainly composed of three neural networks, including Denoising U-Net (Denoising U-Net), Video Control Network (Video ControlNet) and Content Guider (Content Guider).

Ali innovates again: you can realize the dance of Cleaning the Glass with a sentence and a human face, and the costume and background can be switched freely!picture

Among them, Video ControlNet is an image control network injected into the Motion Block after each U-Net block, processing the control sequence (pose or depth) into an additional temporal residual

Denoising U-Net is A derived Stable-Diffusion U-Net with motion blocks for video generation.

The Content Guider transmits the input text prompts and appearance expressions (such as faces) to the content embedding.

Through such operations, DreaMoving is able to generate high-quality, high-fidelity videos given the input of a guidance sequence and a simple content description (such as text and reference images)

Ali innovates again: you can realize the dance of Cleaning the Glass with a sentence and a human face, and the costume and background can be switched freely!Picture

But unfortunately, there is currently no open source code for the DreaMoving project.

For those who are interested in this, you can pay attention first and wait for the release of the open source code~

Please refer to the following link: [1]https://dreamoving.github.io/dreamoving /[2]https://arxiv.org/abs/2312.05107[3]https://twitter.com/ProperPrompter/status/1734192772465258499[4]https://github.com/dreamoving/dreamoving-project

The above is the detailed content of Ali innovates again: you can realize the dance of 'Cleaning the Glass' with a sentence and a human face, and the costume and background can be switched freely!. For more information, please follow other related articles on the PHP Chinese website!

Statement
This article is reproduced at:51CTO.COM. If there is any infringement, please contact admin@php.cn delete
分享PyCharm项目打包的简易方法分享PyCharm项目打包的简易方法Dec 30, 2023 am 09:34 AM

简单易懂的PyCharm项目打包方法分享随着Python的流行,越来越多的开发者使用PyCharm作为Python开发的主要工具。PyCharm是功能强大的集成开发环境,它提供了许多方便的功能来帮助我们提高开发效率。其中一个重要的功能就是项目的打包。本文将介绍如何在PyCharm中简单易懂地打包项目,并提供具体的代码示例。为什么要打包项目?在Python开发

PyCharm实用技巧:将项目转换为可执行EXE文件PyCharm实用技巧:将项目转换为可执行EXE文件Feb 23, 2024 am 09:33 AM

PyCharm是一款功能强大的Python集成开发环境,提供了丰富的开发工具和环境配置,让开发者能够更高效地编写和调试代码。在使用PyCharm进行Python项目开发的过程中,有时候我们需要将项目打包成可执行的EXE文件,以便在没有安装Python环境的计算机上运行。本文将介绍如何使用PyCharm将项目转换为可执行的EXE文件,同时给出具体的代码示例。首

制作 iPhone 上 iOS 17 提醒应用程序中的购物清单的方法制作 iPhone 上 iOS 17 提醒应用程序中的购物清单的方法Sep 21, 2023 pm 06:41 PM

如何在iOS17中的iPhone上制作GroceryList在“提醒事项”应用中创建GroceryList非常简单。你只需添加一个列表,然后用你的项目填充它。该应用程序会自动将您的商品分类,您甚至可以与您的伴侣或扁平伙伴合作,列出您需要从商店购买的东西。以下是执行此操作的完整步骤:步骤1:打开iCloud提醒事项听起来很奇怪,苹果表示您需要启用来自iCloud的提醒才能在iOS17上创建GroceryList。以下是它的步骤:前往iPhone上的“设置”应用,然后点击[您的姓名]。接下来,选择i

基于开源的 ChatGPT Web UI 项目,快速构建属于自己的 ChatGPT 站点基于开源的 ChatGPT Web UI 项目,快速构建属于自己的 ChatGPT 站点Apr 15, 2023 pm 07:43 PM

作为一个技术博主,了不起比较喜欢各种折腾,之前给大家介绍过ChatGPT​接入微信,钉钉和知识星球(如果没看过的可以翻翻前面的文章),最近再看开源项目的时候,发现了一个ChatGPTWebUI项目。想着刚好之前没有将ChatGPT​接入过WebUI,有了这个开源项目可以拿来使用,真是不错,下面是实操的安装步骤,分享给大家。安装官方在Github​的项目文档上提供了很多中的安装方式,包括手动安装,docker​部署,以及远程部署等方法,了不起在选择部署方式的时候,一开始为了简单想着

react启动项目报错怎么办react启动项目报错怎么办Dec 27, 2022 am 10:36 AM

react启动项目报错的解决办法:1、进入项目文件夹,启动项目并查看报错信息;2、执行“npm install”或“npm install react-scripts”命令;3、执行“npm install @ant-design/pro-field --save”命令。

PyCharm教程:如何在PyCharm中移除项目?PyCharm教程:如何在PyCharm中移除项目?Feb 24, 2024 pm 05:54 PM

PyCharm是一款功能强大的Python集成开发环境(IDE),提供了丰富的功能帮助开发者更高效地编写和管理Python项目。在使用PyCharm开发项目的过程中,有时候我们需要删除一些不再需要的项目以释放空间或清理项目列表。本文将详细介绍如何在PyCharm中删除项目,并提供具体的代码示例。如何删除项目打开PyCharm,进入项目列表界面。在项目列表中,

基础教程:使用IDEA创建Maven项目基础教程:使用IDEA创建Maven项目Feb 19, 2024 pm 04:43 PM

IDEA(IntelliJIDEA)是一款强大的集成开发环境,可以帮助开发人员快速高效地开发各种Java应用程序。在Java项目开发中,使用Maven作为项目管理工具能够帮助我们更好地管理依赖库、构建项目等。本文将详细介绍如何在IDEA中创建一个Maven项目的基本步骤,同时提供具体的代码示例。步骤一:打开IDEA并创建新项目打开IntelliJIDEA

快速掌握PyCharm项目打包的基础知识快速掌握PyCharm项目打包的基础知识Dec 30, 2023 pm 01:17 PM

从零开始,快速上手PyCharm项目打包技巧概述:在Python开发中,将项目打包成可执行文件是非常重要的一步。它可以方便地分享和分发项目,而无需安装Python解释器和依赖包。PyCharm作为一个功能强大的Python集成开发环境,提供了快速上手项目打包的技巧和工具。本文将介绍如何利用PyCharm从零开始打包你的Python项目,并提供具体的代码示例。

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
Repo: How To Revive Teammates
4 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
4 weeks agoBy尊渡假赌尊渡假赌尊渡假赌

Hot Tools

PhpStorm Mac version

PhpStorm Mac version

The latest (2018.2.1) professional PHP integrated development tool

Safe Exam Browser

Safe Exam Browser

Safe Exam Browser is a secure browser environment for taking online exams securely. This software turns any computer into a secure workstation. It controls access to any utility and prevents students from using unauthorized resources.

SublimeText3 English version

SublimeText3 English version

Recommended: Win version, supports code prompts!

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)