search
HomeTechnology peripheralsAIUse 2D images to create a 3D human body. You can wear any clothes and change your movements.

Thanks to the differentiable rendering provided by NeRF, recent 3D generative models have achieved stunning results on stationary objects. However, in a more complex and deformable category such as the human body, 3D generation still poses great challenges. This paper proposes an efficient combined NeRF representation of the human body, enabling high-resolution (512x256) 3D human body generation without the use of super-resolution models. EVA3D has significantly surpassed existing solutions on four large-scale human body data sets, and the code has been open source.

Use 2D images to create a 3D human body. You can wear any clothes and change your movements.


  • ##Thesis name: EVA3D: Compositional 3D Human Generation from 2D image Collections
  • Paper address: https://arxiv.org/abs/2210.04888
  • Project homepage: https://hongfz16.github.io/projects/EVA3D.html
  • Open source code: https://github.com/hongfz16/EVA3D
  • Colab Demo: https://colab.research.google. com/github/hongfz16/EVA3D/blob/main/notebook/EVA3D_Demo.ipynb
  • Hugging Face Demo: https://huggingface.co/spaces/hongfz16/EVA3D


Use 2D images to create a 3D human body. You can wear any clothes and change your movements.


Use 2D images to create a 3D human body. You can wear any clothes and change your movements.


#Background

Use the differentiable rendering algorithm provided by NeRF, three-dimensional generation algorithm, such as EG3D, StyleSDF, in the generation of static object categories It has already had very good results. However, compared with categories such as faces or CAD models, the human body is more complex in appearance and geometry, and the human body is deformable, so learning to generate 3D human bodies from 2D images is still a very difficult task. Researchers have made some attempts on this task, such as ENARF-GAN and GNARF, but limited by inefficient human expression, they cannot achieve high-resolution generation, so the generation quality is also very low.

In order to solve this problem, this paper proposes an efficient combined 3D human body NeRF representation to achieve high-resolution (512x256) 3D human body GAN training and generation. The human NeRF representation proposed in this article and the three-dimensional human GAN training framework will be introduced below.

Efficient Human NeRF Representation

The human NeRF proposed in this article is based on the parametric human model SMPL, which provides convenient control of human posture and shape. When doing NeRF modeling, as shown in the figure below, this article divides the human body into 16 parts. Each part corresponds to a small NeRF network for local modeling. When rendering each part, this paper only needs to reason about the local NeRF. This sparse rendering method can also achieve native high-resolution rendering with lower computing resources.

For example, when rendering a human body whose body and action parameters are inverse linear blend skinning), convert the sampling points in posed space into canonical space. Then it is calculated that the sampling points in the Canonical space belong to the bounding box of one or several local NeRFs, and then the NeRF model is inferred to obtain the color and density corresponding to each sampling point; when a certain sampling point falls into multiple local NeRFs In the overlapping area, each NeRF model will be inferred, and multiple results will be interpolated using the window function; finally, this information will be used for light integration to obtain the final rendering.

Use 2D images to create a 3D human body. You can wear any clothes and change your movements.

Three-dimensional human body GAN framework

Based on the proposed efficient human NeRF expression, this article implements a three-dimensional human body GAN training framework. In each training iteration, this paper first samples an SMPL parameter and camera parameters from the data set, and randomly generates a Gaussian noise z. Using the human body NeRF proposed in this article, this article can render the sampled parameters into a two-dimensional human body picture as a fake sample. Using real samples in the data set, this article conducts adversarial training of GAN.

Use 2D images to create a 3D human body. You can wear any clothes and change your movements.

Extremely imbalanced data sets

Two-dimensional human body data sets, such as DeepFashion, are usually It is prepared for two-dimensional vision tasks, so the posture diversity of the human body is very limited. To quantify the degree of imbalance, this paper counts the frequency of model face orientations in DeepFashion. As shown in the figure below, the orange line represents the distribution of face orientations in DeepFashion. It can be seen that it is extremely unbalanced, which makes it difficult to learn three-dimensional human body representation. To alleviate this problem, we propose a sampling method guided by human posture to flatten the distribution curve, as shown by the other colored lines in the figure below. This allows the model during training to see more diverse and larger angle images of the human body, thereby helping to learn three-dimensional human geometry. We conducted an experimental analysis of the sampling parameters. As can be seen from the table below, after adding the human posture guidance sampling method, although the image quality (FID) will be slightly reduced, the learned three-dimensional geometry (Depth) is significantly better.

High-quality generation results

The following figure shows some EVA3D generation results. EVA3D can randomly sample human body appearance, and can control rendering camera parameters and human postures. and body shape.

Use 2D images to create a 3D human body. You can wear any clothes and change your movements.

This paper conducts experiments on four large-scale human data sets, namely DeepFashion, SHHQ, UBCFashion, and AIST . This study compares the state-of-the-art static 3D object generation algorithm EG3D with StyleSDF. At the same time, the researchers also compared the algorithm ENARF-GAN specifically for 3D human generation. In the selection of indicators, this article takes into account the evaluation of rendering quality (FID/KID), the accuracy of human body control (PCK) and the quality of geometry generation (Depth). As shown in the figure below, this article significantly surpasses previous solutions in all data sets and all indicators.

Use 2D images to create a 3D human body. You can wear any clothes and change your movements.

Application potential

Finally, this article also shows some application potential of EVA3D. First, the study tested differencing in the latent space. As shown in the figure below, this article is able to make smooth changes between two three-dimensional people, and the intermediate results maintain high quality. In addition, this article also conducted experiments on GAN inversion. The researchers used Pivotal Tuning Inversion, an algorithm commonly used in two-dimensional GAN ​​inversion. As shown in the right figure below, this method can better restore the appearance of the reconstructed target, but a lot of details are lost in the geometric part. It can be seen that the inversion of three-dimensional GAN ​​is still a very challenging task.

Use 2D images to create a 3D human body. You can wear any clothes and change your movements.

Conclusion

This paper proposes the first high-definition three-dimensional human NeRF generation algorithm EVA3D, and only needs It can be trained using 2D human body image data. EVA3D achieves state-of-the-art performance on multiple large-scale human datasets and shows potential for application on downstream tasks. The training and testing codes of EVA3D have been open sourced, and everyone is welcome to try it!

The above is the detailed content of Use 2D images to create a 3D human body. You can wear any clothes and change your movements.. For more information, please follow other related articles on the PHP Chinese website!

Statement
This article is reproduced at:51CTO.COM. If there is any infringement, please contact admin@php.cn delete
如何在 Windows 11 中清除桌面背景最近的图像历史记录如何在 Windows 11 中清除桌面背景最近的图像历史记录Apr 14, 2023 pm 01:37 PM

<p>Windows 11 改进了系统中的个性化功能,这使用户可以查看之前所做的桌面背景更改的近期历史记录。当您进入windows系统设置应用程序中的个性化部分时,您可以看到各种选项,更改背景壁纸也是其中之一。但是现在可以看到您系统上设置的背景壁纸的最新历史。如果您不喜欢看到此内容并想清除或删除此最近的历史记录,请继续阅读这篇文章,它将帮助您详细了解如何使用注册表编辑器进行操作。</p><h2>如何使用注册表编辑

如何在电脑上下载 Windows 聚光灯壁纸图像如何在电脑上下载 Windows 聚光灯壁纸图像Aug 23, 2023 pm 02:06 PM

窗户从来不是一个忽视美学的人。从XP的田园绿场到Windows11的蓝色漩涡设计,默认桌面壁纸多年来一直是用户愉悦的源泉。借助WindowsSpotlight,您现在每天都可以直接访问锁屏和桌面壁纸的美丽、令人敬畏的图像。不幸的是,这些图像并没有闲逛。如果您爱上了Windows聚光灯图像之一,那么您将想知道如何下载它们,以便将它们作为背景保留一段时间。以下是您需要了解的所有信息。什么是WindowsSpotlight?窗口聚光灯是一个自动壁纸更新程序,可以从“设置”应用中的“个性化&gt

如何在Python中使用图像语义分割技术?如何在Python中使用图像语义分割技术?Jun 06, 2023 am 08:03 AM

随着人工智能技术的不断发展,图像语义分割技术已经成为图像分析领域的热门研究方向。在图像语义分割中,我们将一张图像中的不同区域进行分割,并对每个区域进行分类,从而达到对这张图像的全面理解。Python是一种著名的编程语言,其强大的数据分析和数据可视化能力使其成为了人工智能技术研究领域的首选。本文将介绍如何在Python中使用图像语义分割技术。一、前置知识在深入

如何在Windows上使用PowerToys批量调整图像大小如何在Windows上使用PowerToys批量调整图像大小Aug 23, 2023 pm 07:49 PM

那些必须每天处理图像文件的人经常不得不调整它们的大小以适应他们的项目和工作的需求。但是,如果要处理的图像太多,则单独调整它们的大小会消耗大量时间和精力。在这种情况下,像PowerToys这样的工具可以派上用场,除其他外,可以使用其图像调整大小器实用程序批量调整图像文件的大小。以下是设置图像调整器设置并开始使用PowerToys批量调整图像大小的方法。如何使用PowerToys批量调整图像大小PowerToys是一个多合一的程序,具有各种实用程序和功能,可帮助您加快日常任务。它的实用程序之一是图像

2D图像脑补3D人体,衣服随便搭,还能改动作2D图像脑补3D人体,衣服随便搭,还能改动作Apr 11, 2023 pm 02:31 PM

得益于 NeRF 提供的可微渲染,近期的三维生成模型已经在静止物体上达到了很惊艳的效果。但是在人体这种更加复杂且可形变的类别上,三维生成依旧有很大的挑战。本文提出了一个高效的组合的人体 NeRF 表达,实现了高分辨率(512x256)的三维人体生成,并且没有使用超分模型。EVA3D 在四个大型人体数据集上均大幅超越了已有方案,代码已开源。论文名称:EVA3D: Compositional 3D Human Generation from 2D image Collections论文地址:http

新视角图像生成:讨论基于NeRF的泛化方法新视角图像生成:讨论基于NeRF的泛化方法Apr 09, 2023 pm 05:31 PM

新视角图像生成(NVS)是计算机视觉的一个应用领域,在1998年SuperBowl的比赛,CMU的RI曾展示过给定多摄像头立体视觉(MVS)的NVS,当时这个技术曾转让给美国一家体育电视台,但最终没有商业化;英国BBC广播公司为此做过研发投入,但是没有真正产品化。在基于图像渲染(IBR)领域,NVS应用有一个分支,即基于深度图像的渲染(DBIR)。另外,在2010年曾很火的3D TV,也是需要从单目视频中得到双目立体,但是由于技术的不成熟,最终没有流行起来。当时基于机器学习的方法已经开始研究,比

无需下游训练,Tip-Adapter大幅提升CLIP图像分类准确率无需下游训练,Tip-Adapter大幅提升CLIP图像分类准确率Apr 12, 2023 pm 03:25 PM

论文链接:https://arxiv.org/pdf/2207.09519.pdf代码链接:https://github.com/gaopengcuhk/Tip-Adapter一.研究背景对比性图像语言预训练模型(CLIP)在近期展现出了强大的视觉领域迁移能力,可以在一个全新的下游数据集上进行 zero-shot 图像识别。为了进一步提升 CLIP 的迁移性能,现有方法使用了 few-shot 的设置,例如 CoOp 和 CLIP-Adapter,即提供了少量下游数据集的训练数据,使得 CLIP

一键抹去瑕疵、褶皱:深入解读达摩院高清人像美肤模型ABPN一键抹去瑕疵、褶皱:深入解读达摩院高清人像美肤模型ABPNApr 12, 2023 pm 12:25 PM

随着数字文化产业的蓬勃发展,人工智能技术开始广泛应用于图像编辑和美化领域。其中,人像美肤无疑是应用最广、需求最大的技术之一。传统美颜算法利用基于滤波的图像编辑技术,实现了自动化的磨皮去瑕疵效果,在社交、直播等场景取得了广泛的应用。然而,在门槛较高的专业摄影行业,由于对图像分辨率以及质量标准的较高要求,人工修图师还是作为人像美肤修图的主要生产力,完成包括匀肤、去瑕疵、美白等一系列工作。通常,一位专业修图师对一张高清人像进行美肤操作的平均处理时间为 1-2 分钟,在精度要求更高的广告、影视等领域,该

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
Repo: How To Revive Teammates
1 months agoBy尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
4 weeks agoBy尊渡假赌尊渡假赌尊渡假赌

Hot Tools

SAP NetWeaver Server Adapter for Eclipse

SAP NetWeaver Server Adapter for Eclipse

Integrate Eclipse with SAP NetWeaver application server.

EditPlus Chinese cracked version

EditPlus Chinese cracked version

Small size, syntax highlighting, does not support code prompt function

Dreamweaver Mac version

Dreamweaver Mac version

Visual web development tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

VSCode Windows 64-bit Download

VSCode Windows 64-bit Download

A free and powerful IDE editor launched by Microsoft