"Famous Scenes from Huaguo Mountain" has a high-definition version, and NTU proposes a video super-resolution framework Upscale-A-Video

Diffusion models have achieved remarkable success in image generation, but there are still challenges in applying them to video super-resolution. Video super-resolution requires output fidelity and temporal consistency, which is complicated by the inherent stochasticity of diffusion models. Therefore, effectively applying diffusion models to video super-resolution remains a challenging task.

The research team from Nanyang Technological University S-Lab proposed a text-guided latent diffusion framework called Upscale-A-Video for video super-resolution. The framework ensures temporal consistency through two key mechanisms. First, on a local scale, it integrates temporal layers into U-Net and VAE-Decoder to maintain the consistency of short sequences. Second, at a global scale, the framework introduces a flow-guided recurrent latent propagation module that propagates and fuses latents throughout the sequence without training, thus enhancing the overall video stability. The proposal of this framework provides a new solution for video super-resolution, with better temporal consistency and overall stability.

Paper address: https://arxiv.org/abs/2312.06640

Through the diffusion paradigm, Upscale-A-Video gains great flexibility sex. It allows the use of text prompts to guide texture creation, and noise levels can be adjusted to balance fidelity and quality between recovery and generation. This feature allows the technology to fine-tune details while maintaining the meaning of the original content, resulting in more precise results.

Experimental results show that Upscale-A-Video outperforms existing methods on synthetic and real-world benchmarks, presenting impressive visual realism and temporal consistency.

Let’s look at a few specific examples first. For example, with the help of Upscale-A-Video, “Famous Scenes from Huaguo Mountain” has a high-definition version:

Compared to StableSR, Upscale-A-Video makes the squirrel hair texture in the video clearly visible:

Method Introduction

Some studies optimize image diffusion models to adapt to video tasks by introducing temporal consistency strategies. These strategies include the following two methods: first, fine-tuning video models through temporal layers, such as 3D convolution and temporal attention, to improve video processing performance. Second, zero-shot mechanisms, such as cross-frame attention and flow-guided attention, are used to tune in the pre-trained model to improve performance on video tasks. The introduction of these methods enables the image diffusion model to better handle video tasks, thereby improving the effect of video processing.

Although these solutions significantly improve video stability, two main issues remain:

• Current methods operate in U-Net feature or latent space Difficulty maintaining low-level consistency, issues like texture flickering persist.

• Existing temporal layers and attention mechanisms can only impose constraints on short local input sequences, limiting their ability to ensure global temporal consistency in longer videos.

To solve these problems, Upscale-A-Video adopts a local-global strategy to maintain temporal consistency in video reconstruction, focusing on fine-grained textures and overall consistency. On local video clips, this study explores using additional temporal layers on video data to fine-tune a pre-trained image ×4 super-resolution model.

Specifically, within the latent diffusion framework, this study first fine-tunes U-Net using integrated 3D convolution and temporal attention layers, and then uses video conditional input and 3D convolution to tune VAE decoding device. The former significantly achieves structural stability of local sequences, and the latter further improves low-level consistency and reduces texture flickering. On a global scale, this study introduces a novel, training-free flow-guided recurrent latent propagation module that performs frame-by-frame propagation and latent fusion in both directions during inference, promoting the overall stability of long videos.

Upscale-A-Video models can utilize text prompts as optional conditions to guide the model to produce more realistic and higher-quality details, as shown in Figure 1.

Upscale-A-Video divides the video into segments and processes them using U-Net with temporal layers to achieve intra-segment consistency. A recurrent latent propagation module is used to enhance inter-segment consistency during user-specified global refinement diffusion. Finally, a fine-tuned VAE decoder reduces flicker artifacts and achieves low-level consistency.

Experimental results

Upscale-A-Video achieves SOTA performance on existing benchmarks, Demonstrates superior visual realism and temporal consistency.

Quantitative assessment. As shown in Table 1, Upscale-A-Video achieves the highest PSNR in all four synthetic datasets, indicating its excellent reconstruction capabilities.

Qualitative assessment. The study shows the visual results for synthetic and real-world videos in Figures 4 and 5 respectively. Upscale-A-Video significantly outperforms existing CNN and diffusion-based methods in both artifact removal and detail generation.

The above is the detailed content of "Famous Scenes from Huaguo Mountain" has a high-definition version, and NTU proposes a video super-resolution framework Upscale-A-Video. For more information, please follow other related articles on the PHP Chinese website!