Killing like crazy! Google rolls video to speech, and realistic sound effects make AI videos farewell to silence!

The great situation that the AI ​​​​circle is blooming everywhere has surprised the melon-eating public.

# In the past few days, the other side of the ocean has been crazy!

#Luma’s excitement hasn’t passed yet, last night Runway released a king bomb-Gen-3 Alpha. (For details, please go to: Runway version Sora released: high fidelity, super consistency, Gen-3 Alpha shocked netizens)

What was even more unexpected was that when I woke up, Google DeepMind also had new news, quietly announcing the progress of video-to-speech (V2A) technology.

Although this feature is not yet open to the public, judging from the official video demo, the effect is quite smooth. At the same time, Google DeepMind emphasized that all examples were jointly created by V2A technology and their most advanced generative video model Veo.

Audio prompt: An exciting horror movie soundtrack, footsteps echoing on the concrete. (Cinematic, thriller, horror film, music, tension, ambience, footsteps on concrete)

In the dark abandoned warehouse, a man in black walks slowly like a ghost. There is weird music and footsteps, and the atmosphere is full of terror.

Audio prompt: The wolf howls in the moonlight. (Wolf howling at the moon)

As soon as the video demo came out, Qing Yishui in the comment area asked: When will it be available?

Some netizens hope that the open source community will become a cyber bodhisattva and copy Google’s technology.

In fact, not long after Google DeepMind was officially announced, ElevenLabs, the "leader" in the field of AI audio, stepped in and open sourced a project for automatic dubbing of uploaded videos. , which can generate suitable sound effects for videos.

Link:

https://elevenlabs.io/docs/api-reference/how-to-use-text-to-sound-effects

Nowadays, the competition in the AI ​​​​circle has become fierce. The pursuit of each other by large and small manufacturers will create a more fair competitive environment. Once these technologies mature, the AI ​​​​video field will There are endless possibilities.

##AI Video Farewell to Silent Movies

It is known that video generation models are developing at an astonishing rate. However, whether it is Sora, which shocked the world at the beginning of the year, or the recent Keling, Luma, and Gen-3 Alpha, they are all "silent movies" without exception.

And Google DeepMind’s video-to-audio (V2A) technology makes synchronous audio-visual generation possible. It can combine video pixels and natural language text cues to generate rich voiceovers for on-screen action.

In terms of technical application, V2A technology can be combined with video generation models such as Veo to create dialogues with dramatic soundtracks, realistic sound effects, or matching video characters and styles. lens.

It can also generate audio tracks for archival materials, silent films and other traditional images, broadening creative possibilities.

Audio prompt: Cute baby dinosaurs chirp in the jungle, accompanied by the sound of cracking eggshells. (Cute baby dinosaur chirps, jungle ambience, egg cracking)Audio prompts: The sound of cars skidding and engines roaring, accompanied by angelic electronic music. (cars skidding, car engine throttling, angelic electronic music) Audio prompt: At sunset, the melodious harmonica sounds on the grassland. (a slow mellow harmonica plays as the sun goes down on the prairie)

V2A technology is capable of generating an unlimited number of audio tracks for any video input. Users can choose to define "positive cues" to guide the generation of desired sounds, or "negative cues" to avoid undesired sounds.

This flexibility gives users more control over audio output, allowing them to quickly try different audio outputs and choose the best match.

Audio prompt: A spaceship is speeding in the vast space, stars are passing around it, flying at high speed, full of science fiction feeling. (A spaceship hurtles through the vastness of space, stars streaking past it, high speed, Sci-fi) Audio prompt: Ethereal cello atmosphere Audio prompt: A spaceship shuttles through the vast space at high speed, with stars passing quickly around it, giving it a sci-fi feel. (A spaceship hurtles through the vastness of space, stars streaking past it, high speed, Sci-fi)

The working principle behind it

The research team tried autoregressive and diffusion methods to discover the most scalable AI architecture. Diffusion methods give the most realistic and engaging results on audio generation for synchronizing video and audio information.

#The V2A system first encodes the video input into a compressed representation, then a diffusion model iteratively refines the audio from random noise. This process is guided by visual input and given natural language cues, producing synchronized, realistic audio that is tightly aligned with the cues. Finally, the audio output is decoded into an audio waveform and combined with the video data.

To generate higher quality audio and guide the model to generate specific sounds, the research team added more information during the training process, including AI-generated annotations that describe the sounds in detail and dialogue text.

Through training on video, audio and additional annotations, the technology learns to associate specific audio events with various visual scenes, while responding to annotations or text provided Information.

Google emphasizes that their technology is different from existing video-to-audio solutions because it understands raw pixels and adding text hints is optional. In addition, the system does not require manual alignment of generated sound and video, greatly simplifying the creative process.

#However, Google’s technology is not perfect, and they are still working hard to solve some bugs. For example, the quality of the video input directly affects the quality of the audio output, and artifacts or distortions in the video can cause audio quality to degrade.

At the same time, they are also optimizing the lip sync function.

V2A technology attempts to generate speech from input text and synchronize it with the character's mouth movements, but if the video model is not adjusted accordingly for the text content, This may cause the mouth shape and speech to be out of sync. They are improving this technology to make lip sync more natural.

Audio prompt: Music, Transcript "This turkey looks amazing, I'm so hungry." 'm so hungry”)

Perhaps due to the many social problems caused by deep forgery technology, Google DeepMind is full of desire to survive and continues to promise to develop and deploy AI responsibly. Technology, V2A technology will undergo rigorous safety assessment and testing before being made available to the public.

Additionally, they have integrated the SynthID toolkit into V2A research to add a watermark to all AI-generated content to prevent misuse of the technology.

^{Reference link:}

^{https://deepmind.google/ discover/blog/generating-audio-for-video/}

^{https://x.com/GoogleDeepMind/status/1802733643992850760}

The above is the detailed content of Killing like crazy! Google rolls video to speech, and realistic sound effects make AI videos farewell to silence!. For more information, please follow other related articles on the PHP Chinese website!