Core principles of intelligent speech synthesis

Statistical parametric speech synthesis methods have attracted widespread attention in the field of speech synthesis due to their flexibility. In recent years, the application of deep neural network models in the field of machine learning research has achieved significant advantages compared with traditional methods. The application of modeling methods based on neural networks in statistical parametric speech synthesis has gradually deepened and has become one of the mainstream methods of speech synthesis.

Backend acoustic modeling for statistical parametric speech synthesis is the subject of this article.

The back-end framework of parameterized speech synthesis

As shown in the figure, the back-end framework of statistical parameter speech synthesis is described, mainly It includes two stages: training and synthesis.

In the training phase, the speech waveforms and corresponding text features in the sound library are used as input. Speech waveforms are extracted through a vocoder and combined with text features for acoustic modeling.

In the synthesis stage, according to the trained acoustic model, the text features to be synthesized are input and the corresponding acoustic features are predicted. The predicted acoustic features are then converted into speech waveforms using a vocoder. Vocoder and acoustic models are key components in statistical parametric speech synthesis systems.

The source filter model of speech generation is used to separate the short-term spectrum of speech into the fundamental frequency and spectral envelope during the speech waveform parameterization process. Usually, we obtain the excitation characteristics of speech by analyzing time domain waveforms or frequency domain harmonics, and then remove the periodicity of time and frequency from the amplitude spectrum obtained by the short-time Fourier transform of the speech waveform to obtain the spectrum package of speech. network. This method can help us better understand and process speech signals.

Due to the high dimensionality of the spectral envelope, modeling becomes difficult, so it is usually necessary to reduce the dimensionality of the spectral envelope. Reconstructing the speech waveform is the reverse process of recovering the original speech from the acoustic parameters of the speech. By given the fundamental frequency, spectral envelope and excitation characteristics of speech, combined with appropriate phase constraints, the STFT amplitude spectrum can be reconstructed.

Duration modeling is another module in statistical parametric speech synthesis. Duration modeling does not require a vocoder. The basic framework is similar to acoustic modeling. Statistical models are used to model the probability distribution of corresponding time lengths given text features.

After more than 20 years of development, the statistical parameter speech synthesis method based on HMM has become a mature speech synthesis method.

This section will introduce the hidden Markov model and its theoretical basis. Combined with certain phase constraints, the STFT amplitude spectrum is reconstructed. Duration modeling is another module in statistical parametric speech synthesis. Duration modeling does not require a vocoder. The basic framework is similar to acoustic modeling. Statistical models are used to model the probability distribution of corresponding time lengths given text features. After more than 20 years of development, the statistical parameter speech synthesis method based on HMM has become a mature speech synthesis method.

Hidden Markov model is a probabilistic model for sequence modeling, which consists of a set of hidden state variables and a set of observation variables. The HMM model has two assumptions.

The state variable obeys the first-order Markov chain; that is, the current state is only related to the previous state, as shown in formula (1).

The probability distribution of an observed variable at a certain moment is only related to the state at the current moment and has nothing to do with the state or observed variables at other moments, as shown in Eq. (2) shown.

Usually, in the HMM model

is cleverly formed The state transition matrix A of HMM, the probability density of the observed variables is:

It is worth noting that the output probability of HMM:

The core principle of acoustic modeling in the HMM-based statistical parametric speech synthesis method is to use the HMM model to perform probabilistic modeling of the acoustic feature sequence of speech in a given situation.

The configuration of the entire system includes the selection of speech acoustic features, the selection of modeling units and the configuration of the HMM model. Acoustic features in speech synthesis systems include excitation features and spectral features.

In the selection of spectral features, in order to reduce the difficulty of HMM modeling, low-dimensional spectral representation that removes the correlation between dimensions is generally used, such as Mel cepstrum and line spectrum pair features. Considering the short-term stationary characteristics of speech signals and the modeling ability of HM, HMMs in speech synthesis systems usually model phoneme-level units, such as vowel units in Chinese. Due to the timing characteristics of speech, the topology of HMM in audio modeling is often a one-way traversal state from left to right.

Statistical parameter speech synthesis system framework based on HMM

The figure describes the framework of the statistical parameter speech synthesis system based on HMM. It is divided into training stage and comprehensive stage. The training phase includes speech acoustic feature extraction and HMM model training. Since the HMM model uses phonemes as modeling units, three context-related phonemes are usually modeled to improve modeling accuracy.

In the first system training process, estimate the lower limit of the variance of the HMM model, then train the single-tone HMM model as the model initialization parameter, then train the context-related three-phoneme HMM model, and finally perform Mn pressure clustering based on Decision tree is carried out.

In the synthesis stage, the text is first analyzed, combined with the predicted time length, the context-related HMM model sequence is determined based on the decision tree, and then the continuous acoustic feature sequence and speech waveform are obtained through the maximum likelihood parameter generation algorithm Synthesized by a synthesizer. Statistical parametric speech synthesis systems based on HMM are too smooth; one reason is the limited modeling ability of HMM.

In recent years, as a branch of machine learning, deep learning has developed rapidly. Deep learning refers to the use of network models composed of multiple nonlinear transformations and multiple processing layers, namely neural networks. Due to the excellent modeling capabilities of DNN and inch, the acoustic modeling method based on DNN and RNN is applied to statistical parametric speech synthesis, and its effect is better than the acoustic modeling method based on HMM.

It has become the mainstream method of statistical parametric speech synthesis acoustic modeling. Speech synthesis systems based on DNN and RNN are similar in system framework.

Framework diagram of speech synthesis method based on neural network

As shown in the figure, the input features in the figure are extracted from the text Features; that is, using discrete or continuous numerical features to describe text.

The training of statistical parametric speech synthesis systems based on DNN and RNN usually adopts training criteria and uses BP algorithm and SGD algorithm to update model parameters so that the predicted acoustic parameters are as close as possible to the natural acoustic parameters. In the synthesis stage, text features are extracted from the synthesized text, then the corresponding acoustic parameters are predicted through DNN or RNN, and finally the speech waveform is synthesized through the vocoder.

Currently, modeling methods based on DNN and RNN are mainly applied to speech acoustic parameters, including fundamental frequency and spectrum parameters. Duration information still needs to be obtained through other systems. In addition, the input and output features of DNN and RNN models need to be aligned in time.

The above is the detailed content of Core principles of intelligent speech synthesis. For more information, please follow other related articles on the PHP Chinese website!