2024 ICASSP｜Innovative solution from ByteDance streaming audio team: solving packet loss compensation and general sound quality repair issues

In this year’s ICASSP 2024 Various Audio International Challenges, the ByteDance streaming audio team teamed up with the Audio Speech and Language Processing Research Laboratory of Northwestern Polytechnical University to perform Packet Loss Concealment (PLC) and sound quality restoration. (Speech Signal Improvement, SSI) In the two challenge tracks, it performed well on multiple indicators and achieved first and second place respectively, reaching the international leading level.

The Audio Challenge at the ICASSP Summit was jointly launched by the top international audio conference ICASSP and Microsoft, aiming to stimulate research on audio effects and sound quality improvement by various research institutions. Since the first session, it has attracted Amazon , Tencent, Alibaba, Baidu, Kuaishou, Chinese Academy of Sciences, Northwestern Polytechnical University and many other world-renowned companies and scientific research institutes participated. With the continuous development of technology in the field of streaming media, making the sound clear and authentic has become an inevitable trend in the development of the audio technology industry. Focusing on how to provide users with a better audio experience, multiple research teams have carried out end-to-end optimization of audio from collection to forwarding. This process includes how to deal with audio collection defects, algorithm processing defects, coding and decoding defects, and network transmission defects. Wait for integrated repair. In this challenge, the ByteDance streaming audio team participated in two challenge tracks, packet loss compensation and general sound quality repair, based on real business implementation scenarios.

ICASSP PLC Challenge aims to solve the problems of long-interval packet loss and full-band audio (48k Hz sampling rate) processing in network IP calls. The challenge has strict latency constraints while providing a demanding data set that reflects adverse network conditions. Subjective evaluation will be conducted using the P.804 multi-dimensional audio quality assessment method, while WER is also used to evaluate the intelligibility of the speech generated by the participating systems. The streaming audio technology team effectively reduced the complexity of the packet loss compensation model by optimizing the model structure. At the same time, through multi-discriminator adversarial training and multi-task learning, the packet loss compensation model can restore packet loss fragments with high quality and high intelligibility, and finally achieved the first place.

ICASSP SSI Challenge aims to solve five types of problems faced by speech signals in communication systems: frequency response distortion, discontinuity distortion, loudness distortion, noise and reverberation. This challenge uses subjective opinion scores and speech recognition rate under the ITU-TP.804 standard to comprehensively judge the rankings under the premise of strictly setting the model delay and causality. The streaming technology team uses a two-stage model structure to simplify the complex repair problem into multiple subtasks. In the first stage, it mainly repairs frequency response distortion, discontinuity distortion and loudness distortion, and performs preliminary noise reduction and de-reverberation; in the second stage This stage further removes artifacts generated in the first stage as well as residual noise. In the end, the team achieved second place on the real-time track.

Packet loss compensation system

In order to solve the problem of 48kHz full-band audio processing complexity, it is used in the packet loss compensation system A frequency domain model is developed, and the audio is divided into two sub-bands of 0-8kHz and 8-24kHz according to frequency for parallel processing. The main calculation amount is concentrated on the 0-8kHz frequency band that has a greater impact on the sense of hearing, achieving low-complexity and high-quality packet loss compensation. In order to deal with the problem of long-interval packet loss, a time-frequency dilated convolution module (TFDCM) is added after each layer of the codec. While maintaining a small size of the convolution kernel, it captures long-term through causal dilated convolution layer by layer in time and frequency dimensions. Time history information and frequency correlation.

In order to compensate for higher quality audio, frequency domain multi-resolution discriminator, time domain multi-period discriminator and MetricGAN are used in combination to conduct generative adversarial training, making the generated audio sound excellent. For long-interval packet loss and intelligibility issues, a multi-task learning framework is used. In addition to the usual speech signal similarity learning, fundamental frequency prediction and whisper-based semantic understanding loss functions are also introduced. The model can recover packet loss fragments longer than 100ms with high quality, and the recovered audio is highly intelligible. The word accuracy rate (WAcc) indicator leads all participating teams, and the overall evaluation score is tied for first place.

Packet loss compensation model structure diagram

Sound quality repair system

In order to repair audio that is affected by multiple distortions at the same time, a two-stage model architecture is used in the construction system, focusing on processing different distortions at different stages. The first-stage model uses mapping to directly predict the complex spectrum of the repaired audio, so that the model has the ability to generate audio missing components and eliminate interference signals at the same time. In order to improve the model's ability to capture information for a long time, the encoder The Time-Frequency Convlution Module (TFCM) is introduced into the decoder; due to the instability of the mapping method, artifacts may occur, so a two-stage model using masking (Mask) is introduced, and sub- The band-to-full-band modeling method performs fine-grained modeling of frequency bands to further eliminate artifacts and residual noise generated by the first-stage model.

In order to improve the naturalness of the generated audio components, a generative adversarial network framework is introduced, and multi-resolution discriminator and molecular band multi-resolution discriminator are used to assist the model training. At the same time, in order to make the multi-stage model converge more easily during training, the two-stage model is first pre-trained on the noise reduction and dereverberation tasks, and then the parameters of the trained one-stage model are frozen and compared with the pre-trained second-stage model. Stage models are cascaded for joint training, thereby accelerating model convergence.

Schematic diagram of sound quality repair model structure

Team introduction

Bytedance streaming audio team, dedicated to Provides high-quality, low-latency real-time audio and video communication capabilities across the global Internet, helping developers quickly build rich scene functions such as voice calls, video calls, interactive live broadcasts, retweet live broadcasts, etc. It currently covers interactive entertainment, education, conferences, Real-time audio and video interactive scenarios such as games, automobiles, finance, and IoT serve hundreds of millions of users.

The above is the detailed content of 2024 ICASSP｜Innovative solution from ByteDance streaming audio team: solving packet loss compensation and general sound quality repair issues. For more information, please follow other related articles on the PHP Chinese website!