Google AI singer is coming! AudioLM can compose music and songs by simply listening for a few seconds.

Image generation model rolled up! Video generative models rolled up!

The next one is the audio generation model.

Recently, the Google research team launched an AI model for speech generation-AudioLM.

With just a few seconds of audio prompts, it can not only generate high-quality, coherent speech, but also piano music.

Paper address: https://www.php.cn/link/b6b3598b407b7f328e3129c74ca8ca94

AudioLM is a high-quality audio generation framework with long-term consistency , maps the input audio into a string of discrete tokens, and transforms the audio generation task into a language modeling task.

Existing audio taggers have to make a trade-off between audio generation quality and stable long-term structure.

In order to solve this contradiction, Google adopts a "hybrid tokenization" solution, using discretized activation of pre-trained mask language models, and using discrete codes generated by neural audio codecs to achieve high-quality synthesis.

AudioLM model can learn to generate natural and coherent continuous words based on short prompts, when trained on speech, without any recording or annotation. It achieves continuous speech that is grammatically smooth and semantically reasonable, while maintaining the identity and intonation of the speaker.

In addition to speech, AudioLM can also generate coherent piano music without even needing to train on any musical symbols.

From text to piano music: two major issues

In recent years, language models trained on massive text corpora have shown their excellent generative capabilities, realizing open dialogue, Machine translation, and even common sense reasoning, can also model signals other than text, such as natural images.

The idea of ​​AudioLM is to leverage these advances in language modeling to generate audio without training on annotated data.

But this requires facing two problems.

First, the data rate for audio is much higher and the unit sequence is longer. For example, a sentence contains dozens of character representations, but after being converted into an audio waveform, it generally contains hundreds of thousands of values.

In addition, there is a one-to-many relationship between text and audio. The same sentence can be presented by different speakers with different styles, emotional content, and context.

To overcome these two challenges, AudioLM utilizes two audio tags.

First, semantic tags are extracted from w2v-BERT, a self-supervised audio model.

These tags capture both local dependencies (such as speech in speech, local melodies in piano music) and global long-term structures (such as language syntax and semantic content in speech, harmony and rhythm in piano music) while heavily downsampling the audio signal in order to model long sequences.

However, the fidelity of the audio reconstructed from these tokens is not high.

To improve sound quality, in addition to semantic tagging, AudioLM also utilizes acoustic tags generated by the SoundStream neural codec to capture the details of the audio waveform (such as speaker characteristics or recording conditions) for high-quality synthesis.

How to train?

AudioLM is an audio-only model trained without any symbolic representation of text or music.

It models audio sequences hierarchically by chaining multiple Transformer models (one for each stage) from semantic tagging to fine acoustic tagging.

Each stage is trained to predict the next token based on the last token, just like training a language model.

The first stage performs this task on semantic tags to model the high-level structure of the audio sequence.

In the second stage, by connecting the entire semantic tag sequence with the past rough tags, and feeding the two as conditions to the rough model, and then predicting the future mark.

This step simulates acoustic properties, such as speaker characteristics or timbre in music.

In the third stage, a fine acoustic model is used to process the coarse acoustic signal, thereby adding more details to the final audio.

Finally, the acoustic markers are fed into the SoundStream decoder to reconstruct the waveform.

After training is completed, AudioLM can be adjusted on a few seconds of audio, which allows it to generate continuous audio.

In order to demonstrate the general applicability of AudioLM, the researchers tested it through 2 tasks in different audio fields.

The first is Speech continuation. This model retains the prompted speaker characteristics and prosody, while also outputting new content that is grammatically correct and semantically consistent.

The second is Piano continuation, which generates piano music that is consistent with the cues in terms of melody, harmony, and rhythm.

As shown below, all the sounds you hear after the gray vertical lines are generated by AudioLM.

To verify the effectiveness, the researchers asked human raters to listen to short audio clips to determine whether they were original recordings of human speech or recordings generated by AudioLM.

According to the collected ratings, it can be seen that AudioLM has a success rate of 51.2%, which means that the speech generated by this AI model is difficult for ordinary listeners to distinguish from real speech.

Rupal Patel, who studies information and language science at Northeastern University, said that previous work using artificial intelligence to generate audio could only capture these nuances if they were explicitly annotated in the training data.

In contrast, AudioLM automatically learns these features from the input data and also achieves high-fidelity results.

With the emergence of multi-modal ML models such as GPT3 and Bloom (text generation), DALLE and Stable Diffusion (image generation), RunwayML and Make-A-Video (video generation), content creation and creativity Work is changing.

The future world is a world generated by artificial intelligence.

Reference materials:

​​https://www.php.cn/link/c11cb55c3d8dcc03a7ab7ab722703e0a​​

​​https ://www.php.cn/link/b6b3598b407b7f328e3129c74ca8ca94​​

​​https://www.php.cn/link/c5f7756d9f92a8954884ec415f79d120​​

​​https://www.php.cn/link/9b644ca9f37e3699ddf2055800130aa9​​

The above is the detailed content of Google AI singer is coming! AudioLM can compose music and songs by simply listening for a few seconds.. For more information, please follow other related articles on the PHP Chinese website!