Hot ChatTTS broke through the open source voice ceiling and gained 9k stars in 3 days

Is this what the future of communication between people will look like?

Recently, a text-to-speech project called ChatTTS has become popular, attracting great attention from everyone. In just three days, it has gained 9.2k stars on GitHub.

Project address: https://github.com/2noise/ChatTTS/tree/main

The author himself also stated on x that ChatTTS has achieved a breakthrough Open source ceiling. However, what is currently open source is only the base model, which has not been supervised and fine-tuned by SFT.

This project converts text into speech. The effect is as follows: ChatTTS can not only speak Chinese, but also hold English. Supporting some fine-grained control, it allows you to add laughter, pauses between speaking, and modal particles, making it very playable. It can reproduce the out-of-print voices of people who have passed away. If you want to hear Steve Jobs’ development conference again, you can do so at any time. Listening to it imitating Swift's timbre, both the intonation and the changes in tone are very close to her, and there is almost no AI flavor at all. I can also speak both Chinese and English. With my half-English and half-Chinese accent, I am brave enough to break into the Liuzi circle. ChatTTS’s language ability has reached the next level.

##The above audio comes from station B: https://www.bilibili.com/video/BV1zn4y1o7iV/?share_source=copy_web&vd_source=983ec32a3036bb1cf2699e4fdbce3c28

According to the above description From the demonstration, we can see that ChatTTS can achieve natural and smooth speech synthesis and support multiple speakers at the same time; it can also predict and control fine-grained prosodic features, including laughter, pauses, and inserted words; ChatTTS has surpassed Most open source TTS models.

Currently ChatTTS supports Chinese and English. The largest model was trained using more than 100,000 hours of Chinese and English data. The open source version in HuggingFace is a version that has been trained for 40,000 hours but has not yet been SFT.

It is worth mentioning that the audios displayed above are all produced based on speech synthesis technology and are intended to showcase technical achievements and are not intended to offend or infringe on the rights of others.

Once the project was released, netizens from all walks of life tried it out one after another and said that it was indeed difficult to tell whether the sound was real or fake.

Some people use GPT to generate text and let ChatTTS "read" it. The difference between the tone and intonation of real people is very small:

#The effect is so good, naturally I want to try it. How to use ChatTTS as your mouth substitute? You can refer to the following methods to operate.

Online experience address: https://huggingface.co/spaces/Dzkaka/ChatTTS

ChatTTS mainly has two core functions, the first is text-to-speech, and the second is real-time voice dialogue with a large language model. In addition to these functions, you can adjust the timbre of the digitally specified speaker in "Audio Seed", or roll the dice to randomly generate one. However, many testers said that if the same parameters are used every time, the tone generated is not necessarily fixed.

2Noise said that it currently supports sound cloning, but requires a larger amount of data.

After entering text in the text box, ChatTTS will automatically generate rhymes and pauses for you, and will also add some modal particles such as "then". If you add [laugh] and [uv_break] to the text when typing, you can manually control ChatTTS to produce some "laughter" when speaking.

However, ChatTTS cannot handle relatively long texts yet. Some netizens asked it to challenge the audiobook and found that the initial version could not be generated. Audio longer than 30 seconds requires manual repair. When encountering relatively long text, ChatTTS's word segmentation will also have problems.

The above is the detailed content of Hot ChatTTS broke through the open source voice ceiling and gained 9k stars in 3 days. For more information, please follow other related articles on the PHP Chinese website!