Xiaozha spent a lot of money! Meta has developed an AI model specifically for the Metaverse

​Artificial intelligence will become the backbone of the virtual world.

Artificial intelligence can be combined with a variety of related technologies in the metaverse, such as computer vision, natural language processing, blockchain and digital twins.

#In February, Zuckerberg showed off what the Metaverse would look like at Inside The Lab, the company’s first virtual event. He said the company is developing a new series of generative AI models that will allow users to generate their own virtual reality avatars simply by describing them.

Zuckerberg announced a series of upcoming projects, such as Project CAIRaoke, a fully end-to-end neural model for building on-device voice assistants that can help users communicate with their voice assistants more naturally . Meanwhile, Meta is working hard to build a universal speech translator that provides direct speech-to-speech translation for all languages.

A few months later, Meta fulfilled their promise. However, Meta isn't the only tech company with skin in the game. Companies such as NVIDIA have also released their own self-developed AI models to provide a richer Metaverse experience.

Open source pre-trained Transformer (OPT-175 billion parameters)

GAN verse 3D

GANverse 3D is developed by NVIDIA AI Research and is A model that uses deep learning to process 2D images into 3D animated versions, a tool described in a research paper published at ICLR and CVPR last year, can produce simulations faster and at a lower cost.

This model uses StyleGAN to automatically generate multiple views from a single image. The application can be imported as an extension to NVIDIA Omniverse to accurately render 3D objects in virtual worlds. Omniverse launched by NVIDIA helps users create simulations of their final ideas in a virtual environment.

The production of 3D models has become a key factor in building the metaverse. Retailers such as Nike and Forever21 have set up their virtual stores in the Metaverse to drive e-commerce sales.

Visual Acoustic Matching Model (AViTAR)

​

Meta’s Reality Lab team collaborated with the University of Texas to build an artificial intelligence model that to improve the sound quality of metaspace. This model helps match audio and video in a scene. It transforms audio clips to make them sound like they were recorded in a specific environment. The model uses self-supervised learning after extracting data from random online videos. Ideally, users should be able to view their favorite memories on their AR glasses and hear the exact sounds produced by the actual experience.

Meta AI has released AViTAR as open source, along with two other acoustic models, which is very rare considering that sound is an often overlooked part of the metaverse experience.

Visually Impacted Vibration Reduction (VIDA)

The second acoustic model released by Meta AI is used to remove reverberation in acoustics.

The model is trained on a large-scale dataset with various realistic audio renderings from 3D models of homes. Reverb not only reduces the quality of the audio, making it difficult to understand, but it also improves the accuracy of automatic speech recognition.

​

VIDA is unique in that it uses audio for observation as well as visual cues. Improving on typical audio-only approaches, VIDA can enhance speech and identify voices and speakers.

Visual Voice (VisualVoice)

VisualVoice, the third acoustic model released by Meta AI, can extract speech from videos. Like VIDA, VisualVoice is trained on audio-visual cues from unlabeled videos. The model has automatically separated speech.

This model has important application scenarios, such as making technology for the hearing-impaired, enhancing the sound of wearable AR devices, transcribing speech from online videos in noisy environments, etc.

Audio2Face

Last year, Nvidia released an open beta of Omniverse Audio2Face to generate AI-driven facial animations to match any voiceover. This tool simplifies the long and tedious process of animating games and visual effects. The app also allows users to issue commands in multiple languages.

At the beginning of this year, Nvidia released an update to the tool, adding features such as BlendShape Generation to help users create a set of blendhapes from a neutral avatar. Additionally, the functionality of a streaming audio player has been added, allowing streaming of audio data using text-to-speech applications. Audio2Face sets up a 3D character model that can be animated with audio tracks. The audio is then fed into a deep neural network. Users can also edit characters in post-processing to change their performance. ​

The above is the detailed content of Xiaozha spent a lot of money! Meta has developed an AI model specifically for the Metaverse. For more information, please follow other related articles on the PHP Chinese website!