Using machine learning to reconstruct faces in videos

Translator|Cui Hao

Reviser|Sun Shujuan

Opening Chapter

##A project from China and Britain Collaborative research devises a new way to recreate faces in videos. This technology can enlarge and reduce facial structure with high consistency and no trace of artificial trimming.

Generally speaking, this transformation of facial structure is achieved through traditional CGI methods, which rely on detailed and expensive motion capping, rigging and texturing procedure to completely reconstruct the face.

Different from traditional methods, CGI in the new technology is integrated into the neural pipeline as a parameter for 3D facial information and serves as the basis for the machine learning workflow.

The author pointed out:

"Our goal is to deform and edit the facial contours based on natural faces in the real world. , thereby generating high-quality portrait reshaping videos [results]. This technology can be used for visual effect applications such as facial beautification and facial exaggeration.

Although consumers have been able to use 2D faces since the advent of Photoshop distortion technology (and led to a subculture of facial distortion and body dysmorphia), but it is still a difficult technology to achieve facial reconstruction for video without using CGI.

Mark Zuckerberg’s facial size expands and shrinks due to new technology

Body reshaping is currently a hot topic in the field of computer vision, mainly because of its potential in fashion e-commerce , for example: making people look taller and more skeletally diverse, but there are still some challenges.

Similarly, changing the shape of faces in videos in a convincing way has been at the core of the researchers' work, Although the implementation of this technology has been affected by artificial processing and other limitations. As a result, the new product migrates the previously studied capabilities from static expansion to dynamic video output.

The new system is equipped with AMD Ryzen 9 3950X Training is performed on a desktop PC with 32GB of memory, and motion maps are generated using OpenCV’s optical flow algorithm and smoothed through the StructureFlow framework; Facial Alignment Network (FAN) component for feature estimation, also used in the popular deepfakes component package Medium; Working with Ceres Solver to solve facial optimization problems.

Example of using the new system to enlarge the face

The title of this paper is Parametric Reshaping of Portraits in Videos, whose authors are three researchers from Zhejiang University and one researcher from the University of Bath.

About Face

In the new system, videos are extracted into image sequences, starting with faces Build a base model. Then connect representative subsequent frames to build consistent personality parameters along the entire image running direction (i.e., the direction of the video frame).

Face The architectural process of the deformation system

Next, according to the calculation expression, the shaping parameters implemented by linear regression are generated. Then the 2D mapping of the facial contour is constructed through the signed distance function (SDF) before and after facial reshaping. .

Finally, the output video is subjected to morphing optimization for content recognition.

Facial parameterization

This process utilizes the 3D Morphable Face Model (3DMM) , it is a face synthesis auxiliary tool based on neural and GAN, and is also suitable for deepfake detection systems.

Example from 3D Morphable face Model (3DMM) - a parametric prototype face used in a new project. Top left, iconic application on 3DMM surface. Top right, 3D mesh vertices of isomap. The lower left corner shows the feature fit; the bottom middle picture, the isomap of the extracted facial texture; and the lower right corner, the final fit and shape

The workflow of the new system will take into account occlusion situations, such as when an object moves away from the line of sight. This is also one of the biggest challenges for deepfake software, as FAN landmarks can barely account for these situations and their translation quality tends to degrade as faces are avoided or occluded.

The new system avoids the above problems by defining "contour energy" that matches the boundaries of 3D faces (3DMM) and 2D faces (defined by FAN landmarks).

Optimization

The application scenario of this system is real-time deformation, such as real-time transformation of face shape in a video chat filter. Currently, frameworks cannot achieve this, so providing the necessary computing resources to enable "real-time" deformation becomes a significant challenge.

According to the assumptions of the paper, the latency of each frame operation of the 24fps video relative to the material per second in the pipeline is 16.344 seconds. At the same time, for feature estimation and 3D facial deformation, it is also accompanied by one hit (respectively 321 ms and 160 ms).

As a result, optimization has made key progress in reducing latency. Since joint optimization across all frames would significantly increase system overhead, and optimization of the initialization style (assuming consistent speaker characteristics throughout) may lead to anomalies, the authors adopted a sparse mode to calculate coefficients at realistic intervals of sampled frames.

Joint optimization is then performed on this subset of frames, resulting in a leaner reconstruction process.

Facial Surface

The morphing technology used in this project is an adaptation of the author’s 2020 work Deep Shapely Portraits (DSP).

Deep Shapely Portraits, 2020 submission to ACM Multimedia. The paper was led by researchers from the Zhejiang University-Tencent Joint Laboratory for Game and Intelligent Graphics Innovation Technology

The authors observed that “we extend this method from reshaping a single image to reshaping an entire image sequence.”

Testing

The paper points out that there is no comparable historical data to evaluate the new method. Therefore, the authors compared their curved video output frames with static DSP output.

Testing the new system against static images from Deep Shapely Portraits

The author pointed out that due to the use of sparse mapping, the DSP method will have traces of artificial modification— —The new framework solves this problem through dense mapping. Furthermore, the paper argues that videos produced by DSP lack smoothness and visual coherence.

The authors pointed out:

“The results show that our method can stably and coherently generate reshaped portrait videos, while image-based methods can easily lead to obvious flickering artifacts (artificial Traces of modification)."

Translator introduction

Cui Hao, 51CTO community editor, senior architect, has 18 years of software development and architecture experience, and 10 years of distributed architecture experience. Formerly a technical expert at HP. He is willing to share and has written many popular technical articles with more than 600,000 reads. Author of "Principles and Practice of Distributed Architecture".

Original title: ​​Restructuring Faces in Videos With Machine Learning​​, author: Martin Anderson

The above is the detailed content of Using machine learning to reconstruct faces in videos. For more information, please follow other related articles on the PHP Chinese website!