search
HomeTechnology peripheralsAIWith the help of Schrödinger Bridge, Zhu Jun's team at Tsinghua University develops a new speech synthesis system to address proliferation challenges

Recently, a speech synthesis system based on the Schrödinger Bridge [1] released by Professor Zhu Jun’s research group from the Department of Computer Science at Tsinghua University, relies on its “data-to-data” generation paradigm, defeating both in terms of sample quality and sampling speed. The "noise-to-data" paradigm of diffusion models.

With the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challenges

Paper link: https://arxiv.org/abs/2312.03491
Project website: https://bridge-tts.github.io/
Code implementation: https://github.com/thu-ml/Bridge-TTS

##Problem background

Since 2021, diffusion models have begun to become one of the core generation methods in the field of text-to-speech synthesis (TTS). First, methods such as Grad-TTS [2] proposed by Huawei's Noah's Ark Laboratory and DiffSinger [3] proposed by Zhejiang University have achieved high generation quality. Since then, many research works have effectively improved the sampling speed of diffusion models, such as through prior optimization [2, 3, 4], model distillation [5, 6], residual prediction [7] and other methods. However, as shown in this study, because the diffusion model is limited to the generation paradigm of "noise to data", its prior distribution always provides limited information for the generation target and cannot fully utilize the conditional information.
The latest research work in the field of speech synthesis, Bridge-TTS, relies on its generation framework based on Schrödinger bridge to realize the generation process of "data to data". For the first time, the priori speech synthesis Information is modified from noise to clean data, and is modified from distribution to deterministic representation.

The main architecture of this method is shown in the figure above. The input text is first extracted through the text encoder to extract the latent space representation of the generated target (mel-spectrogram, mel spectrum). Thereafter, unlike the diffusion model that incorporates this information into the noise distribution or uses it as conditional information, the Bridge-TTS method ‍ supports directly using it as prior information and supports random or deterministic sampling, High Quality , quicklygenerate targets.

Work results

In verifying the quality of speech synthesis On the standard data set LJ-Speech, the research team compared Bridge-TTS with 9 high-quality speech synthesis systems and accelerated sampling methods of diffusion models. As shown below, this method beats high-quality diffusion model-based TTS systems [2,3,7] in sample quality (1000 steps, 50 steps sampling) and in sampling speed, without any post-processing such as additional model distillation, it surpasses many acceleration methods, such as residual prediction, progressive distillation, and the latest consistency distillation [5, 6, 7].
With the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challenges
The following are examples of the generation effects of Bridge-TTS and the diffusion model-based method. For more generation sample comparisons, please visit the project website: https://bridge-tts.github. io/

  • 1000 step synthesis effect comparison

Enter text: "Printing, then, for our purpose, may be considered as the art of making books by means of movable types." With the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challenges
  • 4 Step Synthesis Effect Comparison

Input text: "The first books were printed in black letter, i.e. the letter which was a Gothic development of the ancient Roman character,」With the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challenges
  • ## 2 Comparison of step synthesis effects

Enter text: "The prison population fluctuated a great deal,"
With the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challengesThe following shows Bridge- TTS A case of deterministic synthesis (ODE sampling) in steps 2 and 4. In 4-step synthesis, this method significantly synthesizes more sample details than the diffusion model, and there is no problem of residual noise. In a 2-step synthesis, this method exhibits completely pure sampling trajectories, and refines more generated details at each step.
With the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challenges
In the frequency domain, more generated samples are shown below. In 1000 steps of synthesis, this method generates higher quality mel compared to the diffusion model. Spectrum, when the number of sampling steps drops to 50, the diffusion model has sacrificed some sampling details, while the method based on Schrödinger bridge still maintains high-quality generation effects. In 4-step and 2-step synthesis, this method does not require distillation, multi-stage training, and adversarial loss functions, and still achieves high-quality generation effects.

With the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challenges

##Comparison of the mel spectra of Bridge-TTS and the diffusion model-based method in 1000 steps of synthesis

With the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challenges

Comparison of Mel spectra between Bridge-TTS and diffusion model-based methods in 50-step synthesis

With the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challenges

Comparison of Mel spectra between Bridge-TTS and diffusion model-based methods in 4-step synthesis
With the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challenges
Comparison of Mel spectra between Bridge-TTS and diffusion model-based methods in 2-step synthesis

Once Bridge-TTS was released, it attracted enthusiastic attention on Twitter with its novel design and high-quality synthesis effects in speech synthesis. , received more than a hundred retweets and hundreds of likes, was selected into Huggingface’s Daily Paper on 12.7 and won the first place in the support rate on that day, and was also featured in LinkedIn, Weibo, Zhihu, Xiaohongshu and other domestic It was followed and reported on external platforms.

With the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challenges

Many foreign language websites also reported and discussed:

With the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challenges

Method introduction

##Schrodinger Bridge is a type of deep
generative model that has recently emerged following the diffusion model, has preliminary applications in image generation, image translation and other fields [8,9]. Unlike the diffusion model, which establishes a transformation process between data and Gaussian noise, the Schrödinger bridge supports transformation between any two boundary distributions. In the study of Bridge-TTS, the authors proposed a speech synthesis framework based on the Schrödinger bridge between paired data, which flexibly supports a variety of forward processes, prediction targets, and sampling processes. An overview of its method is shown in the figure below:

With the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challenges

  • Forward process: This study combines strong information priors and generation goals A fully solvable Schrödinger bridge is built between them, supporting flexible forward process selection, such as symmetric noise strategy:, constantWith the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challenges, and With the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challengesasymmetric noise strategy: , linear With the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challenges, and With the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challengesvariance-preserving (VP) noise strategies that correspond directly to diffusion models. This method found that in speech synthesis tasks, asymmetric noise strategies: linear (gmax) and VP processes have better generation effects than symmetric noise strategies. With the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challenges

With the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challenges

With the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challenges

  • Model training: This method maintains many advantages of the diffusion model training process, such as single stage, single model, and single loss function. And it compares various methods of model parameterization (Model parameterization), that is, the selection of network training targets, including noise prediction (Noise), generation target prediction (Data), and flow matching technology corresponding to the diffusion model [10,11] Velocity prediction (Velocity), etc. The article found that when the generation target, that is, the mel spectrum, is used as the network prediction target, relatively better generation results can be achieved.

With the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challengesWith the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challenges

  • Sampling process: Thanks to the fact that the Schrödinger bridge is completely solvable in this study By transforming the forward-backward SDE system corresponding to the Schrödinger bridge, the authors obtained Bridge SDE and Bridge ODE for inference. At the same time, due to the slow speed of direct simulation of Bridge SDE/ODE inference, in order to speed up sampling, this study used the exponential integrator commonly used in diffusion models [12,13], and gave the first-order SDE and ODE sampling forms of the Schrödinger bridge:

With the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challenges

#In 1-step sampling, the sampling forms of first-order SDE and ODE jointly degenerate into single-step prediction of the network. At the same time, they are closely related to posterior sampling/diffusion model DDIM sampling, and the article gives a detailed analysis in the appendix. The article also provides the second-order sampling SDE and ODE sampling algorithms of the Schrödinger bridge. The authors found that in speech synthesis, the generation quality is similar to a first-order sampling process.

In other tasks such as speech enhancement, speech separation, speech editing and other tasks where prior information is also strong, the authors expect that this research will also bring greater Value.

About the author

This study has three co-first authors: Chen Zehua, He Guande and Zheng Kaiwen both belong to Zhu Jun’s research group in the Department of Computer Science at Tsinghua University. The corresponding author of the article is Professor Zhu Jun. Tan Xu, chief research manager of Microsoft Research Asia, is a project collaborator.

With the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challenges

#                                                       Tan Xu, Chief Research Manager, Microsoft Research Asia

With the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challenges

Chen Zehua is a Shuimu Scholar postdoctoral fellow in the Department of Computer Science at Tsinghua University. His main research direction is probabilistic generative models and their applications in speech, sound effects, bioelectrical signal synthesis, etc. He has interned at many companies such as Microsoft, JD.com, and TikTok, and published many papers at important international conferences in the field of speech and machine learning, such as ICML/NeurIPS/ICASSP.

With the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challenges

He Guande is a third-year master's student at Tsinghua University. His main research direction is uncertainty estimation and generative models. He has previously participated in conferences such as ICLR and Published the paper as the first author.

With the help of Schrödinger Bridge, Zhu Juns team at Tsinghua University develops a new speech synthesis system to address proliferation challenges

Zheng Kaiwen is a second-year master's student at Tsinghua University. His main research direction is the theory and algorithm of deep generative models, and their applications in image, audio and 3D generation. He has previously published many papers at top conferences such as ICML/NeurIPS/CVPR, involving technologies such as flow matching and exponential integrators in diffusion models.

References:
[1] Zehua Chen, Guande He , Kaiwen Zheng, Xu Tan, and Jun Zhu. Schrodinger Bridges Beat Diffusion Models on Text-to-Speech Synthesis. arXiv preprint arXiv:2312.03491, 2023.
[2] Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail A. Kudinov. Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech. In ICML, 2021.
[3] Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, and Zhou Zhao. DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism. In AAAI, 2022.
[4] Sang-gil Lee, Heeseung Kim, Chaehun Shin, Xu Tan, Chang Liu, Qi Meng, Tao Qin, Wei Chen, Sungroh Yoon, and Tie-Yan Liu. PriorGrad : Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Prior. In ICLR, 2022.
##[5] Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu , Chenye Cui, and Yi Ren. ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech. In ACM Multimedia, 2022.
[6 ] Zhen Ye, Wei Xue, Xu Tan, Jie Chen, Qifeng Liu, and Yike Guo. CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Model. In ACM Multimedia, 2023.
[7] Zehua Chen, Yihan Wu, Yichong Leng, Jiawei Chen, Haohe Liu, Xu Tan, Yang Cui, Ke Wang, Lei He, Sheng Zhao, Jiang Bian, and Danilo P. Mandic. ResGrad: Residual Denoising Diffusion Probabilistic Models for Text to Speech. arXiv preprint arXiv:2212.14518, 2022.
##[8] Yuyang Shi, Valentin De Bortoli, Andrew Campbell , and Arnaud Doucet. Diffusion Schrödinger Bridge Matching. In NeurIPS 2023.
[9] Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos A . Theodorou, Weili Nie, and Anima Anandkumar. I2SB: Image-to-Image Schrödinger Bridge. In ICML, 2023.
##[10] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow Matching for Generative Modeling. In ICLR, 2023.
##[11] Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Improved Techniques for Maximum Likelihood Estimation for Diffusion ODEs. In ICML, 2023.
##[12] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. In NeurIPS, 2022.
[13] Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statistics. In NeurIPS, 2023.

The above is the detailed content of With the help of Schrödinger Bridge, Zhu Jun's team at Tsinghua University develops a new speech synthesis system to address proliferation challenges. For more information, please follow other related articles on the PHP Chinese website!

Statement
This article is reproduced at:机器之心. If there is any infringement, please contact admin@php.cn delete
The AI Skills Gap Is Slowing Down Supply ChainsThe AI Skills Gap Is Slowing Down Supply ChainsApr 26, 2025 am 11:13 AM

The term "AI-ready workforce" is frequently used, but what does it truly mean in the supply chain industry? According to Abe Eshkenazi, CEO of the Association for Supply Chain Management (ASCM), it signifies professionals capable of critic

How One Company Is Quietly Working To Transform AI ForeverHow One Company Is Quietly Working To Transform AI ForeverApr 26, 2025 am 11:12 AM

The decentralized AI revolution is quietly gaining momentum. This Friday in Austin, Texas, the Bittensor Endgame Summit marks a pivotal moment, transitioning decentralized AI (DeAI) from theory to practical application. Unlike the glitzy commercial

Nvidia Releases NeMo Microservices To Streamline AI Agent DevelopmentNvidia Releases NeMo Microservices To Streamline AI Agent DevelopmentApr 26, 2025 am 11:11 AM

Enterprise AI faces data integration challenges The application of enterprise AI faces a major challenge: building systems that can maintain accuracy and practicality by continuously learning business data. NeMo microservices solve this problem by creating what Nvidia describes as "data flywheel", allowing AI systems to remain relevant through continuous exposure to enterprise information and user interaction. This newly launched toolkit contains five key microservices: NeMo Customizer handles fine-tuning of large language models with higher training throughput. NeMo Evaluator provides simplified evaluation of AI models for custom benchmarks. NeMo Guardrails implements security controls to maintain compliance and appropriateness

AI Paints A New Picture For The Future Of Art And DesignAI Paints A New Picture For The Future Of Art And DesignApr 26, 2025 am 11:10 AM

AI: The Future of Art and Design Artificial intelligence (AI) is changing the field of art and design in unprecedented ways, and its impact is no longer limited to amateurs, but more profoundly affecting professionals. Artwork and design schemes generated by AI are rapidly replacing traditional material images and designers in many transactional design activities such as advertising, social media image generation and web design. However, professional artists and designers also find the practical value of AI. They use AI as an auxiliary tool to explore new aesthetic possibilities, blend different styles, and create novel visual effects. AI helps artists and designers automate repetitive tasks, propose different design elements and provide creative input. AI supports style transfer, which is to apply a style of image

How Zoom Is Revolutionizing Work With Agentic AI: From Meetings To MilestonesHow Zoom Is Revolutionizing Work With Agentic AI: From Meetings To MilestonesApr 26, 2025 am 11:09 AM

Zoom, initially known for its video conferencing platform, is leading a workplace revolution with its innovative use of agentic AI. A recent conversation with Zoom's CTO, XD Huang, revealed the company's ambitious vision. Defining Agentic AI Huang d

The Existential Threat To UniversitiesThe Existential Threat To UniversitiesApr 26, 2025 am 11:08 AM

Will AI revolutionize education? This question is prompting serious reflection among educators and stakeholders. The integration of AI into education presents both opportunities and challenges. As Matthew Lynch of The Tech Edvocate notes, universit

The Prototype: American Scientists Are Looking For Jobs AbroadThe Prototype: American Scientists Are Looking For Jobs AbroadApr 26, 2025 am 11:07 AM

The development of scientific research and technology in the United States may face challenges, perhaps due to budget cuts. According to Nature, the number of American scientists applying for overseas jobs increased by 32% from January to March 2025 compared with the same period in 2024. A previous poll showed that 75% of the researchers surveyed were considering searching for jobs in Europe and Canada. Hundreds of NIH and NSF grants have been terminated in the past few months, with NIH’s new grants down by about $2.3 billion this year, a drop of nearly one-third. The leaked budget proposal shows that the Trump administration is considering sharply cutting budgets for scientific institutions, with a possible reduction of up to 50%. The turmoil in the field of basic research has also affected one of the major advantages of the United States: attracting overseas talents. 35

All About Open AI's Latest GPT 4.1 Family - Analytics VidhyaAll About Open AI's Latest GPT 4.1 Family - Analytics VidhyaApr 26, 2025 am 10:19 AM

OpenAI unveils the powerful GPT-4.1 series: a family of three advanced language models designed for real-world applications. This significant leap forward offers faster response times, enhanced comprehension, and drastically reduced costs compared t

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

WebStorm Mac version

WebStorm Mac version

Useful JavaScript development tools

mPDF

mPDF

mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),

EditPlus Chinese cracked version

EditPlus Chinese cracked version

Small size, syntax highlighting, does not support code prompt function

DVWA

DVWA

Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software

SublimeText3 English version

SublimeText3 English version

Recommended: Win version, supports code prompts!