The first large-scale TTS model that supports mixed speaking of Mandarin and dialects: Henanese, Shanghainese, and Cantonese can be spoken smoothly

The AIxiv column is a column where this site publishes academic and technical content. In the past few years, the AIxiv column of this site has received more than 2,000 reports, covering top laboratories from major universities and companies around the world, effectively promoting academic exchanges and dissemination. If you have excellent work that you want to share, please feel free to contribute or contact us for reporting. Submission email: liyazhou@jiqizhixin.com; zhaoyunfeng@jiqizhixin.com

Since the emergence of GPT-4o in 2024, companies in the industry have invested huge resources in the research and development of TTS large models. In recent months, large Chinese speech synthesis models have sprung up, such as chattts, seedtts, cosyvoice, etc.

Although the current large-scale speech synthesis model is almost indistinguishable from real people in Chinese Mandarin, in the face of China's complicated dialects, TTS large-scale models have rarely been involved in training a unified speech synthesis of various Chinese dialects. Large models are a very challenging task.

Industry Pain Points and Technical Bottlenecks

Currently, speech synthesis large model technology has made significant progress in the field of Mandarin, but its development in the field of dialects is very slow. China has dozens of major dialects, each with unique phonetic features and grammatical structures, which makes training a large TTS model covering various dialects extremely complex.

Most of the existing large TTS models focus on Mandarin and cannot meet the diverse speech synthesis needs. In addition, the scarcity of dialect corpora and the lack of high-quality annotation data further increase the technical difficulty.

Technological innovation and breakthrough of Giant Network AI Lab

In order to solve the above problems, algorithm experts and linguists in the Giant Network AI Lab team worked together to build a comprehensive system based on the Chinese dialect system. 20 dialects, over 200,000 hours of Mandarin and dialect data sets. Through this huge data set, we trained the first large-scale TTS model that supports multiple Mandarin dialects - Bailing-TTS. Bailing-TTS can not only generate high-quality Mandarin speech, but also generate a variety of dialect speech including Henanese, Shanghainese, Cantonese, etc.

• ArXiv: https://arxiv.org/pdf/2408.00284
• Homepage: https://giantailab.github.io/bailingtts_tech_report/index.html
• Paper title: Bailing- TTS: Chinese Dialectal Speech Synthesis Towards Human-like Spontaneous Representation

The following is the synthesis effect of Bailing-TTS Henan dialect:

Text 1:

Bianshui flows eastward Infinite spring, the Sui family palace has become dust. Pedestrians should not go up to the long embankment to look out; the wind blows and the flowers and flowers worry about killing people.

Generate voice 1:

Text 2:

I also have many hobbies. It’s nice to listen to Henan opera, and the accent is very exciting to listen to. When I have nothing to do, I can go out for a walk and take in the beautiful scenery of Henan. Fortunately, I can make some fun things, such as the braised noodles and spicy soup. Don't tell me, it's okay if I make them myself.

Generated Voice 2:

Let me listen to the effect of zero-sample cloning in Mandarin:

Prompt 1: Young-Male Generated 1: This question, hmm, From another perspective, is it also a good thing for us? Prompt 2: Boy-Male Prompt 2: Hey, tomorrow is the weekend again, let’s go watch a movie together. Prompt 3: Elderly-Female

Generation 3: Speaking of our past, ah, I couldn’t finish it in three days and three nights. Prompt 4: Toddler-Female Generation 4: Oh, this is what you are talking about. I picked this up when I went to the beach.

We have adopted a number of innovative technologies to achieve this goal:

1. Unified dialect token specifications: We have unified the token specifications of various dialects and unified the tokens of Mandarin and various dialects There is partial overlap to provide basic pronunciation skills using Mandarin. This enables us to achieve high-quality dialect speech synthesis under limited data conditions.

2. Refined Token Alignment Technology: We propose a refined token-wise alignment technology based on large-scale multi-modal pre-training.

3. Hierarchical Mixed Expert Architecture: We design a hierarchical hybrid expert architecture for learning unified representations for multiple Chinese dialects and specific representations for each dialect.

4. Hierarchical reinforcement learning enhancement strategy: We propose a hierarchical reinforcement learning strategy to further enhance the dialect expression ability of the TTS model by combining basic training strategies and advanced training strategies.

Implementation details

                                                                                                                                                                                        ^{Figure 1 Overall architecture of Bailing-TTS}

1. Refined Token alignment based on large-scale multi-modal pre-training

In order to achieve refined alignment of text and voice tokens, We propose a multi-stage, multi-modal pre-training learning framework.

In the first stage, we use an unsupervised sampling strategy to conduct rough training on a large-scale data set. In the second stage, we adopt a refined sampling strategy to conduct fine-grained training on high-quality dialect datasets. This method can effectively capture the fine-grained correlation between text and speech and promote the alignment of the two modalities.

2. Based on the hierarchical mixed expert Transformer network structure

In order to train a unified TTS model suitable for multiple Chinese dialects, we designed a hierarchical mixed expert network structure and multi-stage multi-dialect tokens Learning Strategies.

First, we propose a specially designed hybrid expert architecture for learning unified representations for multiple Chinese dialects and specific representations for each dialect. Then, we inject dialect tokens into different levels of the TTS model through a fusion mechanism based on cross-attention to improve the model's multi-dialect expression capabilities.

3. Hierarchical reinforcement learning enhancement strategy

We propose a hierarchical reinforcement learning strategy to further enhance the TTS model by combining basic strategy training and advanced training strategies. Dialect expression ability. The basic training strategy supports the exploration of high-quality dialect speech expressions, and the advanced training strategy strengthens the speech characteristics of different dialects on this basis, thereby achieving high-quality speech synthesis in multiple dialects.

                                                           

Bailing-TTS has reached a level closer to real people in terms of robustness, generation quality and naturalness in Mandarin and multiple dialects . In Table 1 Bailing-TTS's test results in Chinese general calls and dialects

In the actual application scenario evaluation, Baling-TTS has achieved good results. In Table 2 Bailing-TTS Test results of the test result of the speaker of Chinese general calls, dialects and dialects. The multi-dialect TTS large model has been applied in many practical scenarios. For example, dubbing NPCs in games, dubbing dialects in video creation, etc. Through this technology, game and video content can be closer to regional culture, improving users’ sense of immersion and experience.

In the future, with the further development of end-to-end voice interaction large models, this technology will show greater potential in areas such as dialect culture protection and game AI NPC dialect interaction. In the dialect protection scenario, by supporting voice interaction in multiple dialects, the next generation can easily learn, inherit, and protect Chinese dialects, allowing Chinese dialect culture to have a long history. In the game scene, intelligent NPCs that can speak dialects and can interact with voice will further enhance the expressiveness of game content.

Giant Network AI Lab will continue to be committed to promoting the innovation and application of this technology, bringing users a smarter and more convenient voice interaction experience.

Team introduction

Makmal AI Giant telah ditubuhkan pada tahun 2022. Ia adalah aplikasi teknologi kecerdasan buatan dan institusi penyelidikan yang bergabung dengan Giant Network. Komited dalam bidang penjanaan kandungan AIGC (imej/teks/audio/video/model 3D, dsb.), merealisasikan penghasilan dan penciptaan kandungan pintar yang komprehensif, dan mempromosikan inovasi permainan. Pada masa ini, makmal itu telah membina saluran paip pengeluaran industri AI pautan penuh dalam Giant Pada masa yang sama, ia telah menyelesaikan pendaftaran model menegak besar pertama (GiantGPT) dalam industri permainan dan merupakan yang pertama dimasukkan ke dalam komersil. permohonan.

The above is the detailed content of The first large-scale TTS model that supports mixed speaking of Mandarin and dialects: Henanese, Shanghainese, and Cantonese can be spoken smoothly. For more information, please follow other related articles on the PHP Chinese website!