search
HomeTechnology peripheralsAIVideo can be reconstructed in 14 seconds and characters can be changed. Meta speeds up video synthesis by 44 times.

Meta’s new video synthesis framework has brought us some surprises


Regarding today’s artificial intelligence development level For example, it is not difficult to transfer text-based pictures, pictures-based videos, and image/video style transfer.

Generative AI is gifted with the ability to effortlessly create or modify content. Image editing, in particular, has experienced significant growth, driven by text-to-image diffusion models pre-trained on billion-scale datasets. This wave has spawned a plethora of image editing and content creation apps.

Based on the achievements of image-based generative models, the next challenge area must be to add a "time dimension" to it, so as to achieve relaxed and creative videos edit.

A straightforward strategy is to use an image model to process the video frame by frame. However, generative image editing inherently has high variability—even based on the same textual prompts. Countless ways to edit a given image. If each frame is edited independently, it will be difficult to maintain temporal consistency.

In a recent paper, researchers from the Meta GenAI team proposed Fairy - a "simple adaptation" of the image editing diffusion model, which greatly enhances the The performance of AI in video editing.

The following is Fairy’s editing video effect display:

Video can be reconstructed in 14 seconds and characters can be changed. Meta speeds up video synthesis by 44 times.

Video can be reconstructed in 14 seconds and characters can be changed. Meta speeds up video synthesis by 44 times.

Video can be reconstructed in 14 seconds and characters can be changed. Meta speeds up video synthesis by 44 times.

Video can be reconstructed in 14 seconds and characters can be changed. Meta speeds up video synthesis by 44 times.

##Fairy generates a 120-frame 512×384 video (4 seconds, 30 FPS) in just 14 seconds, which is at least 44 times faster than previous methods. A comprehensive user study involving 1000 generated samples confirmed that the proposed method generates high quality and significantly outperforms existing methods.

How did you do it?

According to the paper, Fairy is based on the concept of cross-frame attention based on anchor points. This mechanism can implicitly propagate diffusion features across frames and ensure temporal consistency. and high-fidelity composite effects. Fairy not only solves the limitations of previous models in terms of memory and processing speed, but also improves temporal consistency through a unique data augmentation strategy that makes the model equivalent to an affine transformation of the source and target images.

Video can be reconstructed in 14 seconds and characters can be changed. Meta speeds up video synthesis by 44 times.

  • Paper address: https://arxiv.org/pdf/2312.13834.pdf
  • Project homepage: https://fairy-video2video.github.io/

Method

Fairy revisits the previous tracking-and-propagation paradigm in the context of diffusion model characteristics. In particular, this study uses correspondence estimation to bridge cross-frame attention, allowing the model to track and propagate intermediate features within the diffusion model.

The cross-frame attention map can be interpreted as a similarity measure to evaluate the correspondence between tokens in each frame. The features of a semantic region will Higher attention is allocated to similar semantic regions in other frames, as shown in Figure 3 below.

Therefore, the current feature representation is refined and propagated by focusing on the weighted sum of similar regions between frames, thereby effectively minimizing feature differences between frames.

Video can be reconstructed in 14 seconds and characters can be changed. Meta speeds up video synthesis by 44 times.

A series of operations produce an anchor-based model, which is the core component of Fairy.

In order to ensure the temporal consistency of the generated videos, this study sampled K anchor frames to extract diffusion features, and the extracted features were defined as a set to be propagated to global features of consecutive frames. This study replaces the self-attention layer with cross-frame attention for the cached features of the anchor frame when each new frame is generated. Through cross-frame attention, tokens in each frame adopt features that exhibit similar semantic content in the anchor frame, thereby enhancing consistency.

Video can be reconstructed in 14 seconds and characters can be changed. Meta speeds up video synthesis by 44 times.

Experimental evaluation

In the experimental part, the researchers mainly based on command-based image editing model to implement Fairy and replace the model's self-attention with cross-frame attention. They set the number of anchor frames to 3. The model can accept inputs of different aspect ratios and rescale the longer size input resolution to 512, keeping the aspect ratio unchanged. The researchers edit all frames of the input video without downsampling. All calculations are distributed across 8 A100 GPUs.

Qualitative evaluation

The researcher first showed Fairy’s qualitative results, as shown below As shown in 5, Fairy can edit different themes.

Video can be reconstructed in 14 seconds and characters can be changed. Meta speeds up video synthesis by 44 times.

Video can be reconstructed in 14 seconds and characters can be changed. Meta speeds up video synthesis by 44 times.

In Figure 6 below, the researcher shows that Fairy can perform different types of editing according to text instructions, including stylization, role Changes, partial editing, attribute editing, etc.

Video can be reconstructed in 14 seconds and characters can be changed. Meta speeds up video synthesis by 44 times.

Figure 9 below shows that Fairy can convert source characters into different target characters according to instructions.

Video can be reconstructed in 14 seconds and characters can be changed. Meta speeds up video synthesis by 44 times.

Video can be reconstructed in 14 seconds and characters can be changed. Meta speeds up video synthesis by 44 times.

Quantitative evaluation

The researcher is below The overall quality comparison results are shown in Figure 7, where the videos generated by Fairy are more popular.

Video can be reconstructed in 14 seconds and characters can be changed. Meta speeds up video synthesis by 44 times.

Figure 10 below shows the visual comparison results with the baseline model.

Video can be reconstructed in 14 seconds and characters can be changed. Meta speeds up video synthesis by 44 times.

For more technical details and experimental results, please refer to the original paper.

The above is the detailed content of Video can be reconstructed in 14 seconds and characters can be changed. Meta speeds up video synthesis by 44 times.. For more information, please follow other related articles on the PHP Chinese website!

Statement
This article is reproduced at:机器之心. If there is any infringement, please contact admin@php.cn delete
AI Therapists Are Here: 14 Groundbreaking Mental Health Tools You Need To KnowAI Therapists Are Here: 14 Groundbreaking Mental Health Tools You Need To KnowApr 30, 2025 am 11:17 AM

While it can’t provide the human connection and intuition of a trained therapist, research has shown that many people are comfortable sharing their worries and concerns with relatively faceless and anonymous AI bots. Whether this is always a good i

Calling AI To The Grocery AisleCalling AI To The Grocery AisleApr 30, 2025 am 11:16 AM

Artificial intelligence (AI), a technology decades in the making, is revolutionizing the food retail industry. From large-scale efficiency gains and cost reductions to streamlined processes across various business functions, AI's impact is undeniabl

Getting Pep Talks From Generative AI To Lift Your SpiritGetting Pep Talks From Generative AI To Lift Your SpiritApr 30, 2025 am 11:15 AM

Let’s talk about it. This analysis of an innovative AI breakthrough is part of my ongoing Forbes column coverage on the latest in AI including identifying and explaining various impactful AI complexities (see the link here). In addition, for my comp

Why AI-Powered Hyper-Personalization Is A Must For All BusinessesWhy AI-Powered Hyper-Personalization Is A Must For All BusinessesApr 30, 2025 am 11:14 AM

Maintaining a professional image requires occasional wardrobe updates. While online shopping is convenient, it lacks the certainty of in-person try-ons. My solution? AI-powered personalization. I envision an AI assistant curating clothing selecti

Forget Duolingo: Google Translate's New AI Feature Teaches LanguagesForget Duolingo: Google Translate's New AI Feature Teaches LanguagesApr 30, 2025 am 11:13 AM

Google Translate adds language learning function According to Android Authority, app expert AssembleDebug has found that the latest version of the Google Translate app contains a new "practice" mode of testing code designed to help users improve their language skills through personalized activities. This feature is currently invisible to users, but AssembleDebug is able to partially activate it and view some of its new user interface elements. When activated, the feature adds a new Graduation Cap icon at the bottom of the screen marked with a "Beta" badge indicating that the "Practice" feature will be released initially in experimental form. The related pop-up prompt shows "Practice the activities tailored for you!", which means Google will generate customized

They're Making TCP/IP For AI, And It's Called NANDAThey're Making TCP/IP For AI, And It's Called NANDAApr 30, 2025 am 11:12 AM

MIT researchers are developing NANDA, a groundbreaking web protocol designed for AI agents. Short for Networked Agents and Decentralized AI, NANDA builds upon Anthropic's Model Context Protocol (MCP) by adding internet capabilities, enabling AI agen

The Prompt: Deepfake Detection Is A Booming BusinessThe Prompt: Deepfake Detection Is A Booming BusinessApr 30, 2025 am 11:11 AM

Meta's Latest Venture: An AI App to Rival ChatGPT Meta, the parent company of Facebook, Instagram, WhatsApp, and Threads, is launching a new AI-powered application. This standalone app, Meta AI, aims to compete directly with OpenAI's ChatGPT. Lever

The Next Two Years In AI Cybersecurity For Business LeadersThe Next Two Years In AI Cybersecurity For Business LeadersApr 30, 2025 am 11:10 AM

Navigating the Rising Tide of AI Cyber Attacks Recently, Jason Clinton, CISO for Anthropic, underscored the emerging risks tied to non-human identities—as machine-to-machine communication proliferates, safeguarding these "identities" become

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

SublimeText3 Linux new version

SublimeText3 Linux new version

SublimeText3 Linux latest version

VSCode Windows 64-bit Download

VSCode Windows 64-bit Download

A free and powerful IDE editor launched by Microsoft

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

Dreamweaver Mac version

Dreamweaver Mac version

Visual web development tools

WebStorm Mac version

WebStorm Mac version

Useful JavaScript development tools