CN112802450B - Rhythm-controllable Chinese and English mixed speech synthesis method and system thereof - Google Patents
Rhythm-controllable Chinese and English mixed speech synthesis method and system thereof Download PDFInfo
- Publication number
- CN112802450B CN112802450B CN202110008079.2A CN202110008079A CN112802450B CN 112802450 B CN112802450 B CN 112802450B CN 202110008079 A CN202110008079 A CN 202110008079A CN 112802450 B CN112802450 B CN 112802450B
- Authority
- CN
- China
- Prior art keywords
- text
- duration
- pitch
- module
- energy
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001308 synthesis method Methods 0.000 title claims abstract description 15
- 230000015572 biosynthetic process Effects 0.000 claims abstract description 49
- 238000003786 synthesis reaction Methods 0.000 claims abstract description 49
- 238000001228 spectrum Methods 0.000 claims abstract description 48
- 230000033764 rhythmic process Effects 0.000 claims abstract description 22
- 238000012549 training Methods 0.000 claims abstract description 20
- 238000013528 artificial neural network Methods 0.000 claims abstract description 14
- 230000009191 jumping Effects 0.000 claims abstract description 8
- 238000000034 method Methods 0.000 claims description 24
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 9
- 230000009466 transformation Effects 0.000 claims description 8
- 239000013598 vector Substances 0.000 claims description 8
- 238000012545 processing Methods 0.000 claims description 7
- 238000007781 pre-processing Methods 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims description 2
- 230000005236 sound signal Effects 0.000 claims description 2
- 230000008859 change Effects 0.000 abstract description 7
- 239000011295 pitch Substances 0.000 description 58
- 238000002474 experimental method Methods 0.000 description 10
- 230000008569 process Effects 0.000 description 8
- 230000002457 bidirectional effect Effects 0.000 description 6
- 230000000694 effects Effects 0.000 description 4
- 238000013459 approach Methods 0.000 description 3
- 230000007547 defect Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000002372 labelling Methods 0.000 description 3
- 230000002194 synthesizing effect Effects 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 238000003066 decision tree Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000037433 frameshift Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 230000007847 structural defect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/08—Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
- G10L13/10—Prosody rules derived from text; Stress or intonation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/27—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique
- G10L25/30—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Biophysics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biomedical Technology (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Acoustics & Sound (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Machine Translation (AREA)
Abstract
The invention discloses a rhythm-controllable Chinese and English mixed voice synthesis method and system, and belongs to the field of voice synthesis. 1) And extracting phoneme pronunciation duration, energy, pitch and Mel frequency spectrum from Chinese and English texts and audios with prosody labels as a training set, and learning text coding expression aligned with the length of the Mel frequency spectrum. 2) And generating predicted energy and pitch through an energy and pitch prediction model, and realizing energy and pitch control. 3) The predicted energy, pitch and text coding representation are combined, the synthesized Mel spectrum is output by a decoder, and then the voice is synthesized by a vocoder. The invention uses the jumping neural network encoder to better control the rhythm pause information in the synthesized voice, and simultaneously uses the predicted time length, energy and pitch to finely control the rhythm pronunciation information of each frame in the synthesized voice, and generates the voice with richer rhythm change. And is completely finished by one model, and the language of the text does not need to be distinguished firstly.
Description
Technical Field
The invention belongs to the field of voice synthesis, relates to Chinese and English mixed voice synthesis, and particularly relates to a rhythm-controllable Chinese and English mixed voice synthesis method and system.
Background
With the development of deep learning in recent years, the speech synthesis technology has been greatly improved. Speech synthesis moves from the traditional parametric approach and the concatenation approach towards an end-to-end approach. They typically synthesize speech by first generating a mel-frequency spectrum from text features and then using the mel-frequency spectrum to synthesize the speech using a vocoder image. These end-to-end methods can be classified by structure into autoregressive models and non-autoregressive models. The autoregressive model is usually generated by autoregressive using an Encoder-Attention-Decoder (Encoder-Attention-Decoder) mechanism: to generate the current data point, all previous data points in the time series must be generated as model inputs, like Taoctron, taoctron 2, deep voice 3, clarinet, tasformer TTS. Although autoregressive models can produce satisfactory results, attention to Attention may be inadequate, leading to the phenomenon of duplication or word missing in the synthesized speech. The non-autoregressive model can parallelize the generation of mel spectrum from text features much faster than the autoregressive model, like ParaNet, fastspech, alignts, fastspech 2.
Many different voices can be synthesized for a given text and speaker because the synthesized voices are also affected by prosody. Prosody, in turn, can be controlled by fundamental frequency, energy, and duration. The prosody can be finely controlled in three aspects of energy, duration and pitch in an unsupervised manner. On the other hand, fine control of prosody by supervised means, like fastspech 2, has also been studied. However, the currently available autoregressive speech synthesis model has some disadvantages in practical production due to its complex network structure and autoregressive structural form:
(1) The model is complex, the requirement on computing resources is high, and the model cannot be used on hardware with low computing resources.
(2) The naturalness of long sentence synthesis decreases due to the autoregressive structural defect.
(3) In the aspect of voice control, time length, energy and pitch are mostly only adopted for control, and the voice rhythm is controlled incompletely;
(4) The Chinese and English mixed text cannot be synthesized;
therefore, how to realize the operation of the chinese-english hybrid system and the omnidirectional prosody adjustment on hardware with low computing resources and synthesize more natural voice is an unsolved problem in the field of computer intelligent voice synthesis.
Disclosure of Invention
The present invention is directed to solving the problems of the prior art, and on one hand, to overcome the disadvantage of the prior art that the prosody of the synthesized speech is not fully controlled, the present invention provides a method for controlling the prosody of the synthesized speech from four features, namely, text, duration, energy, and pitch labeled by the prosody. On the other hand, in order to overcome the defects that the Chinese-English mixed speech synthesis system needs to distinguish the languages of the text firstly and then synthesizes the text by using the models of the corresponding languages respectively, the invention provides a Chinese-English mixed model, and a method for directly synthesizing the text without distinguishing the languages. The invention reduces the requirement of the traditional complex speech model on computing resources by optimizing the speech synthesis model, overcomes the defect of an autoregressive network structure, realizes the synthesis of Chinese and English mixed texts and can comprehensively control the synthesized prosody.
In order to achieve the purpose, the invention adopts the following specific technical scheme:
a rhythm-controllable Chinese and English mixed speech synthesis method comprises the following steps:
1) Acquiring a Chinese and English mixed sample text with a rhythm label and a corresponding standard voice audio, converting the standard voice audio into a real Mel frequency spectrum, and extracting the real energy, the real pitch and the real duration of the standard voice audio; processing the Chinese and English mixed sample text with the prosodic tag to obtain a phoneme sequence with the prosodic tag;
2) Constructing a hybrid speech synthesis model, wherein the hybrid speech synthesis model comprises a jumping neural network encoder, a duration prediction module, a pitch prediction module, an energy prediction module and a decoder, and the jumping neural network encoder consists of an Embedding layer, a CBHG module and a jumping module;
3) Training the constructed mixed speech synthesis model by adopting a phoneme sequence with a rhythm label, which specifically comprises the following steps:
3.1 Processing the phoneme sequence with the prosodic tag by an Embedding embedded layer and a CBHG module in sequence to obtain text coding information, removing the prosodic tag from the text coding information by a hopping module, and obtaining the predicted duration of the text coding information by a duration prediction module;
3.2 For text coding information and predicted time length information without prosodic tags, time length is adjusted and then used as input of a pitch prediction module and an energy prediction module respectively to obtain predicted pitch and predicted energy; combining the predicted pitch, the predicted energy and the text coding information subjected to time length adjustment to be used as the input of a decoder to obtain a predicted Mel frequency spectrum;
3.2 Computing a duration loss from the predicted duration and the true duration, computing a pitch loss from the predicted pitch and the true pitch, computing an energy loss from the predicted energy and the true energy, and computing a mel-frequency spectrum loss from the predicted mel-frequency spectrum and the true mel-frequency spectrum; performing end-to-end training on the mixed speech synthesis model by combining various loss values;
4) The text to be synthesized is preprocessed and then used as the input of a trained mixed speech synthesis model to obtain a predicted Mel frequency spectrum, and then the predicted Mel frequency spectrum is synthesized by a vocoder to be output.
Further, the prosodic tags comprise prosodic words, prosodic phrases, intonation phrases, sentence ends and character boundaries. The prosodic tags are added by adopting a pre-trained prosodic phrase boundary prediction model, the text to be synthesized is input into the pre-trained prosodic phrase boundary prediction model, and the text to be synthesized with the prosodic tags is output.
Further, the decoder consists of a bi-directional LSTM and a linear affine transform.
Furthermore, the duration prediction module, the pitch prediction module and the energy prediction module are all composed of three one-dimensional convolution layers and a regularization layer, a bidirectional gating cycle unit GRU and a linear affine transformation.
The invention can utilize the jumping neural network encoder, the duration/energy/pitch prediction module and the neural network decoder to realize the prosody control in the synthesized voice. Compared with the prior art, the invention has the following beneficial effects:
(1) Compared with the traditional method for separately constructing various models, the method adopts a mode of directly converting text to acoustic characteristics, avoids the influence of single model prediction error on the effect of the whole model, thereby improving the fault-tolerant capability of the model and reducing the deployment cost of the model; and the CBHG module is adopted to effectively model the current and context information, extract the features of higher level and extract the context features of the sequence, and the method can learn more text pronunciation features which are difficult to define by people through data, thereby effectively improving the pronunciation effect of the voice.
(2) The text of the invention also comprises a preprocessing process before being input into the speech synthesis model, namely a process of adding prosody labels to the text in a prosodic phrase boundary prediction mode, thereby ensuring controllable text prosody and solving the defect of reduced naturalness of long sentence synthesis caused by unnatural prosody pause of synthesized speech in the traditional speech synthesis method; through a supervision mode, the text, the duration, the energy and the pitch with rhythm labeling are finely controlled, so that the rhythm of the synthesized voice is more comprehensively controlled, and the voice with richer and more natural rhythm change is synthesized;
(3) The invention simplifies the complexity of the training of the speech synthesis model by introducing the duration prediction module, and the traditional end-to-end speech synthesis model adopts the attention module to dynamically align the text and the audio, which needs a large amount of computing resource consumption and time consumption, but the invention avoids the alignment process of the text and the audio in the form of autoregressive attention, thereby reducing the requirement on computing resources, saving the computing cost of the model, realizing the aim of replacing the traditional Chinese and English two independent models by a Chinese-English mixed model, not only using the method of the invention for Chinese-English mixed speech synthesis, but also using the invention for other mixed languages.
Drawings
FIG. 1 is a general schematic diagram of a phonetic synthesis model for prosody controlled Chinese-English mixing used in the present invention;
FIG. 2 is a schematic diagram of a pitch/energy/duration prediction model used in the present invention;
FIG. 3 is the results of an experiment without text prosody control added;
FIG. 4 is the experimental results of the method of the present invention (text prosody tag added, normal energy, normal duration, normal pitch);
FIG. 5 is an experimental result obtained by increasing the amount of energy;
FIG. 6 is an experimental result obtained by increasing the size of a time period;
FIG. 7 is the experimental results for the case of normal pitches;
fig. 8 shows the experimental result obtained by setting the size of 1.5-fold pitch.
Detailed Description
The invention is further illustrated and described below with reference to the drawings and the detailed description.
Compared with a general mixed Chinese and English speech synthesis solution, the prosody pause information in the synthesized speech is better controlled by using the jumping neural network encoder, and meanwhile, the predicted duration, energy and pitch are used for finely controlling the prosody pronunciation information of each frame in the synthesized speech, so that the characteristics of the synthesized speech can be more accurately controlled, and the speech with richer prosody change is generated. On the other hand, the whole solution is completely finished by one model, and the language of the text does not need to be distinguished first, so that the complexity of the model is reduced.
As shown in fig. 1, the method for synthesizing Chinese and english mixed speech with controllable prosody of the present invention includes the following steps:
preprocessing an input Chinese and English text data sequence with rhythm labels to serve as the input of a hopping neural network encoder; the hopping neural network encoder consists of an Embedding embedded layer, a CBHG module and a hopping module;
step two, for the output of the hopping neural network encoder, combining the output of the CBHG module, and obtaining text encoding information with adjusted duration through the duration adjustment;
step three, the text coding information after the duration adjustment is respectively used as the input of a pitch prediction module and an energy prediction module to obtain a predicted pitch and predicted energy; combining the predicted pitch, the predicted energy and the text coding information subjected to time length adjustment to be used as the input of a decoder to obtain a predicted Mel frequency spectrum; the vocoder outputs the synthesized speech.
In one specific implementation of the invention, the hybrid speech synthesis model adopted by the invention consists of a hopping neural network encoder, a duration prediction module, a pitch prediction module, an energy prediction module and a decoder, wherein the hopping neural network encoder consists of an Embedding layer, a CBHG module and a hopping module; the transmission and processing process of the input text in the mixed speech synthesis model comprises the following steps:
1) Aiming at Chinese and English in the text, respectively converting the Chinese and English into corresponding pronunciation phonemes, and constructing a Chinese-English mixed phoneme dictionary; mapping Chinese and English phonemes with rhythm labels to serialized data by adopting a Chinese-English mixed phoneme dictionary to obtain a phoneme sequence w 1 ,w 2 ,…,w U Where U is the length of the text, w i Indicating phoneme information corresponding to the ith word in the text.
2) For serialized text data (phoneme sequence w) 1 ,w 2 ,…,w U ) Converted into phoneme vector sequence x through Embedding layer 1 ,x 2 ,…,x U And the prosodic tags comprise prosodic words (# 1), prosodic phrases (# 2), intonation phrases (# 3), sentence ends (# 4) and character boundaries (# S).
x 1 ,x 2 ,…,x U ＝Embedding(w 1 ,w 2 ,…,w U )；
x i Representing a phoneme vector corresponding to the ith word in the text, wherein Embedding (·) represents Embedding processing; for example, a text "she holds her shoes in the hands and deliberately steps on the bottom of a puddle with the feet shining. "convert to after labeling by rhythm label" she holds #1 shoe #1 in #1 hand # 3 bare #1 foot #2 and steps on #1 puddle #4 deliberately # 1. "
3) For the converted phoneme vector sequence x 1 ，x 2 ，…，x U The text coding information is input into a CBHG module, and the generated result is subjected to a duration prediction module and a skip module to generate predicted duration and text coding information which does not contain the prosodic tag position after skip coding; the CBHG module employed in this embodiment comprises a one-dimensional convolution filter bank, theseThe convolution kernel effectively models current as well as context information. Followed by a multi-level highway network to extract higher level features. And finally, a bidirectional gating cycle unit GRU cycle neural network RNN is used for extracting the context characteristics of the sequence.
Expressed by the formula:
t 1 ，t 2 ，…，t U ＝CBHG(x 1 ，x 2 ，…，x U )
wherein, t i Coding information of the ith word in the text;
4) Because the input serialized text data is added with prosody tags, but the tags do not have obvious pronunciation duration, skip coding is needed to remove the prosody tags to generate t' 1 ，t′ 2 ，…，t′ U′ (U '< U), where U' is the text length after removing prosodic tags.
t′ 1 ，t′ 2 ，…，t′ U′ ＝Skip_state(t 1 ，t 2 ，…，t U )
5) Text coding information t 'without rhythm label position after skip coding' 1 ，t′ 2 ，…，t′ U′ And length expansion is carried out by combining the duration prediction module, and the standard of the length expansion is as follows: in the training stage, the length of the real Mel frequency spectrum is required to be consistent; in the prediction stage, the prediction duration of each word is output according to the trained duration prediction module, and the length of each word is expanded according to the prediction duration; the text coding information t' after the time length adjustment is obtained after the expansion 1 ，t″ 2 ，…，t″ T And T is the frame number of the extracted real Mel spectrum.
The duration prediction model and the energy and pitch prediction model have the same network structure: the three one-dimensional convolution layers and the regularization layer are used for feature separation; a bidirectional GRU learns the relationship between the front and rear phoneme characteristics; finally, the duration/energy/pitch is predicted through a linear affine transformation.
t″ 1 ，t″ 2 ，…，t″ T ＝State_Expand(t′ 1 ，t′ 2 ，…，t′ U′ )
6) Predicting pitch and energy; and aiming at the text coding information and the predicted time length information without the prosodic tags, the text coding information and the predicted time length information are respectively used as the input of a pitch prediction module and an energy prediction module after time length adjustment to obtain predicted pitch and predicted energy for controlling the generated audio energy and pitch.
7) Encoding the predicted pitch and energy with the text information t ″) 1 ，t″ 2 ，…，t″ T Performing combined text encoding feature E 1 ，E 2 ，…，E T ；
For text coding information t ″ 1 ，t″ 2 ，…，t″ T Respectively obtaining the data through an energy and pitch prediction module:
e 1 ，e 2 ，…，e T ＝Energy_Predictor(t″ 1 ，t″ 2 ，…，t″ T )
p 1 ，p 2 ，…，p T ＝Pitch_Predictor(t″ 1 ，t″ 2 ，…，t″ T )
E 1 ，E 2 ，…，E T ＝((e 1 ，e 2 ，…，e T )+(p 1 ，p 2 ，…，p T ))*(t″ 1 ，t″ 2 ，…，t″ T )
wherein E is 1 ，E 2 ，…，E T For the combined text, encoding the information, e 1 ，e 2 ，…，e T To predict energy, p 1 ，p 2 ，…，p T To predict pitch, t ″) 1 ，t″ 2 ，…，t″ T And coding the information for the text after the time length is adjusted.
8) Encoding features E for text 1 ,E 2 ,…,E T Decoding to generate a predicted Mel frequency spectrum;
in an embodiment of the present invention, the decoder is specifically composed of a bi-directional LSTM and a linear affine transformation, and may be specifically expressed as:
encoding by BLSTM:
combining the bidirectional final hidden states to obtain h * Represents:
for the obtained h * Through linear affine transformation, a predicted mel spectrum can be generated:
M 1 ,M 2 ,…,M T ＝Linear(h * )
finally, the generated Mel frequency spectrum is synthesized into the voice with controllable rhythm by a common vocoder.
In one embodiment of the present invention, as shown in fig. 2, the duration prediction module, the pitch prediction module, and the energy prediction module are each composed of three one-dimensional convolution layers and regularization layers, a bidirectional gated loop unit GRU, and a linear affine transformation.
Adding prosodic tags to the text to be synthesized, wherein the addition of the prosodic tags is realized by adopting a pre-trained prosodic phrase boundary prediction model, inputting the text to be synthesized into the pre-trained prosodic phrase boundary prediction model, and outputting the text to be synthesized with the prosodic tags. The pre-trained prosodic phrase boundary prediction model adopts a decision tree or blstm-crf for predicting the boundary of a phrase and inserting prosodic tags, the text to be synthesized is input into the pre-trained prosodic phrase boundary prediction model, and the text to be synthesized with the prosodic tags is output.
Compared with the traditional method for separately constructing various models, the method adopts a mode of directly constructing from text to acoustic characteristics and an end-to-end training mode, calculates time loss according to the predicted time and the real time, calculates pitch loss according to the predicted pitch and the real pitch, calculates energy loss according to the predicted energy and the real energy, and calculates mel frequency spectrum loss according to the predicted mel frequency spectrum and the real mel frequency spectrum; performing end-to-end training on the mixed speech synthesis model by combining various loss values; the influence of single model prediction errors on the effect of the integral model is avoided, and therefore the fault tolerance of the model is improved.
The invention also discloses a rhythm-controllable Chinese and English mixed speech synthesis system, which comprises:
text pre-processing module (front end): the system is used for converting the text into a phoneme sequence with a prosodic tag, and when the mixed speech synthesis system is in a training mode, outputting a real Mel frequency spectrum, a real energy, a real pitch and a real duration according to a standard speech audio corresponding to the text; an encoder: the encoding device is used for encoding a phoneme sequence with a prosodic tag, and an Embedding embedded layer, a CBHG module and a jumping module are arranged in the encoder;
a duration prediction module: the system is used for predicting the duration of the text coding information output by the CBHG module and outputting the predicted duration;
an alignment module: aligning text coding information output by an encoder through a prediction duration, wherein the length of the text coding information is required to be consistent with that of a real Mel frequency spectrum in a training stage; in the prediction stage, the prediction duration of each word is output according to the trained duration prediction module, and the length of each word is expanded according to the prediction duration; the text coding information t after the time length adjustment is obtained after the expansion 1 ,t 2 ,…,t T And T is the frame number of the extracted real Mel spectrum.
A pitch prediction module: for reading the output sequence of the alignment module and predicting (or controlling) the pitch of the sequence;
an energy prediction module: for reading the output sequence of the alignment module and predicting (or controlling) the energy of the sequence;
a decoder: the system comprises a time length adjusting module, a pitch predicting module and a power module, wherein the time length adjusting module is used for combining text coding information, predicted pitch and predicted energy after time length adjustment and decoding combined codes to obtain a voice Mel frequency spectrum;
vocoder: when the mixed speech synthesis system is in a speech synthesis mode, the mixed speech synthesis system is started, automatically reads the speech Mel frequency spectrum output by the decoder, and converts the speech Mel frequency spectrum into a sound signal to play the speech.
The speech synthesis system needs to complete training before use, the training process needs to calculate time loss according to predicted time and real time, calculate pitch loss according to predicted pitch and real pitch, calculate energy loss according to predicted energy and real energy, and calculate mel-frequency spectrum loss according to predicted mel-frequency spectrum and real mel-frequency spectrum; and performing end-to-end training on the mixed speech synthesis model by combining various loss values.
Specifically, the text preprocessing module (front end) mainly functions to receive text data, normalize the text, analyze an XML tag, map a standard text to serialized data using a chinese-english mixed phoneme dictionary, and obtain a phoneme sequence w 1 ,w 2 ,…,w U Where U is the length of the text. The rhythm labeling process specifically comprises the following steps: the prosodic tags comprise prosodic words, prosodic phrases, intonation phrases, sentence ends and character boundaries, the prosodic tags are added by adopting a pre-trained prosodic phrase boundary prediction model, the text to be synthesized is input into the pre-trained prosodic phrase boundary prediction model, and the text to be synthesized with the prosodic tags is output. The training samples used in the training stage may be data with prosodic tags in the open source database.
Specifically, the main function of the encoder is to train and learn the text features of the phoneme sequence of the current sample, so that the phoneme sequence can be converted into a fixed dimension vector capable of representing the text features. Compared with the traditional parametric method speech synthesis algorithm, the function of the encoder is similar to the step of manually extracting the features in the parametric method, the encoder can learn representative feature vectors through data, a large amount of manpower is consumed in the process of manually extracting the features to carry out statistical criteria, and the labor cost is greatly increased. On the other hand, compared with the incomplete feature information possibly caused by manually extracting the features, sufficient feature information can be learned under the condition of comprehensive data coverage through the learned feature vectors.
Specifically, the duration prediction module and the alignment module are used for performing length expansion on coding information output by the coder, the introduction of the duration prediction module simplifies the complexity of the training of the speech synthesis model, and the traditional end-to-end speech synthesis model adopts the attention module to dynamically align texts and audios, which requires a large amount of computing resource consumption and time consumption.
In addition, the introduction of the duration prediction module, the pitch prediction module and the energy prediction module enables prosody to be adjusted in three aspects of duration, pitch and energy, and the specific adjustment mode can be realized by adding an adjustable parameter after the output value of each module and multiplying the output result by a coefficient.
Specifically, compared with the traditional decoder, the decoder is simple in structure and only consists of a bidirectional LSTM and a linear affine transformation, and the decoding speed is greatly improved.
The method is applied to the following embodiments to achieve the technical effects of the present invention, and specific steps in the embodiments are not described in detail.
Examples
The invention tests on a text data set containing 12500 audios and corresponding prosody labels, wherein 10000 Chinese, 2000 English and 500 Chinese-English are mixed. The invention preprocesses the data set as follows:
1) Extracting Chinese and English phoneme files and corresponding audio, and extracting pronunciation duration of the phoneme by using an open source tool Montreal-forced-aligner.
2) A mel-frequency spectrum is extracted for each audio, with a window size of 50 milliseconds, a frame shift of 12.5 milliseconds, and a dimension of 80 dimensions.
3) For each audio, the pitch of the audio is extracted using the World vocoder.
4) The energy of the mel spectrum is obtained by summing the mel spectrum extracted from the audio in dimension.
In order to objectively evaluate the performance of the algorithm of the invention, the invention performs the following comparative tests on the four aspects of text rhythm, energy, duration and pitch in the selected test set containing Chinese-English mixture:
1. fig. 4 and 3, experiment one: prosodic control is added to the text to compare it to text that is not added.
2. Fig. 5 and 4, experiment two: the magnitude of the altered energy is compared to the normal energy magnitude.
3. Fig. 6 and 4, experiment three: the length of the changed duration is compared with the normal duration.
4. Fig. 7 and 8, experiment four: the pitch change size is compared to the normal pitch size.
5. Control group: experiments performed using the method of the present invention.
Experiment one shows that: as shown in fig. 4, in the case of adding text prosody, the same text content, as seen from the abscissa, is longer in fig. 4 than in the case of synthesizing the voice of fig. 3. Because text prosody represents varying degrees of pause, the addition can make the synthesized speech more natural.
Experiment two comparisons show that: fig. 5 is based on fig. 4 with energy information enhanced, and it can be seen that the synthesized speech of fig. 5 is louder for the same input, and appears more intense in fig. 5 than in fig. 4 (due to the use of a black-and-white image, i.e., a larger area of off-white color).
Experiment three comparisons show that: fig. 6 is a graph of the duration of each phoneme increased on the basis of fig. 4, and it can be seen that under the same input, the synthesized speech of fig. 6 is longer and slower in speed from the abscissa, and meanwhile, the energy is also shared, and the result is that the speaking sound is lighter.
In order to more clearly and accurately verify the influence caused by pitch change, longer voices are selected for synthesis observation. Only the pitch parameter is changed in fig. 7 and 8, which show two curves, the upper curve representing energy and the lower curve representing pitch. Fig. 8 shows that the pitch is increased on the basis of fig. 7, compared with the two curves at the lower part, the curve with the increased pitch fluctuates more, the pitch is higher, generally speaking, the voice with the higher pitch is higher in energy, and slight change can be found compared with the two curves at the upper part. Listening to the synthesized speech at the same time verifies this change.
From the above experiments, the mixed speech synthesis system of the invention can control the text rhythm, energy, duration and pitch in the speech synthesis process, and is beneficial to the wide application of the speech synthesis system in industrial scenes.
Various technical features of the above embodiments can be arbitrarily combined, and for the sake of simplicity of description, all possible combinations of the technical features in the above embodiments are not described in detail, however, as long as there is no contradiction between the combinations of the technical features, the combinations should be considered as the scope of description in the present specification.
Claims (10)
1. A rhythm-controllable Chinese and English mixed speech synthesis method is characterized by comprising the following steps:
1) Acquiring a Chinese and English mixed sample text with a rhythm label and a corresponding standard voice audio, converting the standard voice audio into a real Mel frequency spectrum, and extracting the real energy, the real pitch and the real duration of the standard voice audio; processing the Chinese and English mixed sample text with the prosodic tag to obtain a phoneme sequence with the prosodic tag;
2) Constructing a hybrid speech synthesis model, wherein the hybrid speech synthesis model comprises a hopping neural network encoder, a duration prediction module, a pitch prediction module, an energy prediction module and a decoder, and the hopping neural network encoder consists of an Embedding layer, a CBHG module and a hopping module;
3) Training the constructed mixed speech synthesis model by adopting a phoneme sequence with a rhythm label, which specifically comprises the following steps:
3.1 Processing the phoneme sequence with the prosodic tag by an Embedding embedded layer and a CBHG module in sequence to obtain text coding information, removing the prosodic tag from the text coding information by a hopping module, and obtaining the predicted duration of the text coding information by a duration prediction module;
3.2 For text coding information and predicted time length information without prosodic tags, time length is adjusted and then used as input of a pitch prediction module and an energy prediction module respectively to obtain predicted pitch and predicted energy; combining the predicted pitch, the predicted energy and the text coding information subjected to time length adjustment to be used as the input of a decoder to obtain a predicted Mel frequency spectrum;
3.2 Computing a duration loss from the predicted duration and the true duration, computing a pitch loss from the predicted pitch and the true pitch, computing an energy loss from the predicted energy and the true energy, and computing a mel-frequency spectrum loss from the predicted mel-frequency spectrum and the true mel-frequency spectrum; performing end-to-end training on the mixed speech synthesis model by combining various loss values;
4) The text to be synthesized is preprocessed and then used as the input of a trained mixed speech synthesis model to obtain a predicted Mel frequency spectrum, and then the predicted Mel frequency spectrum is synthesized by a vocoder to be output.
2. The prosody-controlled Chinese-English mixed speech synthesis method according to claim 1, wherein the step 1) of processing the Chinese-English mixed sample text with the prosody tag to obtain the phoneme sequence with the prosody tag specifically comprises:
aiming at Chinese and English in the text, respectively converting the Chinese and English into corresponding pronunciation phonemes, and constructing a Chinese-English mixed phoneme dictionary; mapping Chinese and English phonemes with rhythm labels to serialized data by adopting a Chinese-English mixed phoneme dictionary to obtain a phoneme sequence w 1 ，w 2 ，…，w U Where U is the length of the text.
3. The prosody-controlled Chinese-English mixed speech synthesis method according to claim 1, wherein the step 3.1) is specifically:
3.1.2 For phoneme sequences w with prosodic tags 1 ，w 2 ，…，w U And is converted into a phoneme vector sequence x through an Embedding layer 1 ，x 2 ，…，x U Wherein U is the length of the text;
3.1.3 ) generating text coding information t using the converted phoneme vector sequence as input to the CBHG module 1 ，t 2 ，…，t U Respectively enabling the output result of the CBHG module to pass through a duration prediction module and a skip module to generate text coding information t 'without rhythm label positions after prediction duration and skip coding' 1 ，t′ 2 ，…，t′ U′ Wherein U '< U, wherein U' is the text length after removal of the prosodic tag.
4. The method as claimed in claim 1, wherein the prosodic tags include prosodic words, prosodic phrases, intonation phrases, sentence ends, and character boundaries.
5. The prosody-controlled Chinese and English mixed speech synthesis method according to claim 1, wherein the preprocessing in step 4) is specifically: adding prosodic tags to the text to be synthesized, wherein the addition of the prosodic tags is realized by adopting a pre-trained prosodic phrase boundary prediction model, inputting the text to be synthesized into the pre-trained prosodic phrase boundary prediction model, and outputting the text to be synthesized with the prosodic tags.
6. The prosody-controlled Chinese and English mixed speech synthesis method according to claim 1, wherein the duration adjustment in step 3.2) is specifically: text coding information t 'without rhythm label position after skip coding' 1 ，t′ 2 ，…，t′ U′ And length expansion is carried out by combining the duration prediction module, and the standard of the length expansion is as follows: in the training stage, the length of the real Mel frequency spectrum is required to be consistent; in the prediction stage, the prediction duration of each word is output according to the trained duration prediction module, and the length of each word is expanded according to the prediction duration; the text coding information t' after the time length adjustment is obtained after the expansion 1 ，t″ 2 ，…，t″ T T is the frame number of the extracted real Mel frequency spectrum, and U' is the text length after removing the prosodic tag.
7. The prosody-controlled Chinese-English mixed speech synthesis method according to claim 1, wherein the formula for combining the predicted pitch, the predicted energy and the text coding information with the adjusted duration in step 3.2) is as follows:
E 1 ，E 2 ，…，E T ＝((e 1 ，e 2 ，…，e T )+(p 1 ，p 2 ，…，p T ))*(t″ 1 ，t″ 2 ，…，t″ T )
wherein E is 1 ，E 2 ，…，E T For the combined text, encoding the information, e 1 ，e 2 ，…，e T To predict energy, p 1 ，p 2 ，…，p T For predicting pitch, t ″) 1 ，t″ 2 ，…，t″ T And T is the frame number of the extracted real Mel frequency spectrum, which is the text coding information after the time length is adjusted.
8. A prosody controlled chinese-english hybrid speech synthesis method according to claim 1, wherein said decoder comprises a bi-directional LSTM and a linear affine transformation.
9. The prosody-controlled Chinese and English mixed speech synthesis method of claim 1, wherein the duration prediction module, the pitch prediction module and the energy prediction module are each composed of three one-dimensional convolution layers and regularization layers, a two-way gating loop unit GRU and a linear affine transformation.
10. A speech synthesis system based on the prosody-controlled chinese-english hybrid speech synthesis method of claim 1, comprising:
a text preprocessing module: the system is used for converting the Chinese and English mixed text into a phoneme sequence with a prosodic tag, and outputting a real Mel frequency spectrum, a real energy, a real pitch and a real duration according to a standard voice audio corresponding to the text when the mixed voice synthesis system is in a training mode;
an encoder: the encoding device is used for encoding a phoneme sequence with a prosodic tag, and an Embedding layer, a CBHG module and a jumping module are arranged in the encoder;
a duration prediction module: the system is used for predicting the duration of the text coding information output by the CBHG module and outputting the predicted duration;
an alignment module: aligning text coding information output by an encoder through a prediction duration, wherein the length of the text coding information is required to be consistent with that of a real Mel frequency spectrum in a training stage; in the prediction stage, the prediction duration of each word is output according to the trained duration prediction module, and the length of each word is expanded according to the prediction duration; obtaining text coding information after the time length is adjusted after the expansion;
a pitch prediction module: for reading the output of the alignment module, generating a predicted pitch;
an energy prediction module: the device is used for reading the output of the alignment module and generating predicted energy;
a decoder: the system comprises a time length adjusting module, a prediction module and a prediction module, wherein the time length adjusting module is used for combining text coding information, prediction pitch and prediction energy after time length adjustment and decoding combined codes to obtain a prediction Mel frequency spectrum;
a vocoder: when the mixed speech synthesis system is in a speech synthesis mode, the system is started, and the predicted Mel frequency spectrum output by the decoder is automatically read and converted into a sound signal for speech playing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110008079.2A CN112802450B (en) | 2021-01-05 | 2021-01-05 | Rhythm-controllable Chinese and English mixed speech synthesis method and system thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110008079.2A CN112802450B (en) | 2021-01-05 | 2021-01-05 | Rhythm-controllable Chinese and English mixed speech synthesis method and system thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112802450A CN112802450A (en) | 2021-05-14 |
| CN112802450B true CN112802450B (en) | 2022-11-18 |
Family
ID=75808187
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110008079.2A Active CN112802450B (en) | 2021-01-05 | 2021-01-05 | Rhythm-controllable Chinese and English mixed speech synthesis method and system thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112802450B (en) |
Families Citing this family (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113362836B (en) * | 2021-06-02 | 2024-06-11 | 腾讯音乐娱乐科技（深圳）有限公司 | Vocoder training method, terminal and storage medium |
| CN113362804B (en) * | 2021-06-09 | 2024-03-19 | 平安科技（深圳）有限公司 | Method, device, terminal and storage medium for synthesizing voice |
| CN113096638B (en) * | 2021-06-09 | 2021-09-07 | 北京世纪好未来教育科技有限公司 | Speech synthesis model training method, speech synthesis method and device |
| CN113380221A (en) * | 2021-06-21 | 2021-09-10 | 携程科技(上海)有限公司 | Chinese and English mixed speech synthesis method and device, electronic equipment and storage medium |
| CN113421548B (en) * | 2021-06-30 | 2024-02-06 | 平安科技（深圳）有限公司 | Speech synthesis method, device, computer equipment and storage medium |
| CN113257221B (en) * | 2021-07-06 | 2021-09-17 | 成都启英泰伦科技有限公司 | Voice model training method based on front-end design and voice synthesis method |
| CN113327579A (en) * | 2021-08-03 | 2021-08-31 | 北京世纪好未来教育科技有限公司 | Speech synthesis method, speech synthesis device, storage medium and electronic equipment |
| CN113488021A (en) * | 2021-08-09 | 2021-10-08 | 杭州小影创新科技股份有限公司 | Method for improving naturalness of speech synthesis |
| CN115116428B (en) * | 2022-05-19 | 2024-03-15 | 腾讯科技（深圳）有限公司 | Prosodic boundary labeling method, device, equipment, medium and program product |
| CN116665643B (en) * | 2022-11-30 | 2024-03-26 | 荣耀终端有限公司 | Rhythm marking method and device and terminal equipment |
| CN116189654B (en) * | 2023-02-23 | 2024-06-18 | 京东科技信息技术有限公司 | Voice editing method and device, electronic equipment and storage medium |
| CN116092479B (en) * | 2023-04-07 | 2023-07-07 | 杭州东上智能科技有限公司 | Text prosody generation method and system based on comparison text-audio pair |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2595143A1 (en) * | 2011-11-17 | 2013-05-22 | Svox AG | Text to speech synthesis for texts with foreign language inclusions |
| CN110473515A (en) * | 2019-08-29 | 2019-11-19 | 郝洁 | A kind of end-to-end speech synthetic method based on WaveRNN |
| CN110534089A (en) * | 2019-07-10 | 2019-12-03 | 西安交通大学 | A kind of Chinese speech synthesis method based on phoneme and rhythm structure |
| WO2020019885A1 (en) * | 2018-07-25 | 2020-01-30 | 腾讯科技（深圳）有限公司 | Voice synthesis method, model training method, device and computer device |
| CN111292719A (en) * | 2020-02-07 | 2020-06-16 | 北京字节跳动网络技术有限公司 | Speech synthesis method, speech synthesis device, computer readable medium and electronic equipment |
| CN111816158A (en) * | 2019-09-17 | 2020-10-23 | 北京京东尚科信息技术有限公司 | Voice synthesis method and device and storage medium |
| CN111883102A (en) * | 2020-07-14 | 2020-11-03 | 中国科学技术大学 | Sequence-to-sequence speech synthesis method and system for double-layer autoregressive decoding |
| CN112151005A (en) * | 2020-09-28 | 2020-12-29 | 四川长虹电器股份有限公司 | Chinese and English mixed speech synthesis method and device |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20200015418A (en) * | 2018-08-02 | 2020-02-12 | 네오사피엔스 주식회사 | Method and computer readable storage medium for performing text-to-speech synthesis using machine learning based on sequential prosody feature |
| KR102579843B1 (en) * | 2019-05-23 | 2023-09-18 | 구글 엘엘씨 | Variational embedding capacity in expressive end-to-end speech synthesis. |
-
2021
- 2021-01-05 CN CN202110008079.2A patent/CN112802450B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2595143A1 (en) * | 2011-11-17 | 2013-05-22 | Svox AG | Text to speech synthesis for texts with foreign language inclusions |
| WO2020019885A1 (en) * | 2018-07-25 | 2020-01-30 | 腾讯科技（深圳）有限公司 | Voice synthesis method, model training method, device and computer device |
| CN110534089A (en) * | 2019-07-10 | 2019-12-03 | 西安交通大学 | A kind of Chinese speech synthesis method based on phoneme and rhythm structure |
| CN110473515A (en) * | 2019-08-29 | 2019-11-19 | 郝洁 | A kind of end-to-end speech synthetic method based on WaveRNN |
| CN111816158A (en) * | 2019-09-17 | 2020-10-23 | 北京京东尚科信息技术有限公司 | Voice synthesis method and device and storage medium |
| CN111292719A (en) * | 2020-02-07 | 2020-06-16 | 北京字节跳动网络技术有限公司 | Speech synthesis method, speech synthesis device, computer readable medium and electronic equipment |
| CN111883102A (en) * | 2020-07-14 | 2020-11-03 | 中国科学技术大学 | Sequence-to-sequence speech synthesis method and system for double-layer autoregressive decoding |
| CN112151005A (en) * | 2020-09-28 | 2020-12-29 | 四川长虹电器股份有限公司 | Chinese and English mixed speech synthesis method and device |
Non-Patent Citations (2)
| Title |
|---|
| EMPHASIS: An Emotional Phoneme-based Acoustic Model for Speech Synthesis System;Hao Li等;《arXiv》;20180625;全文 * |
| 基于深度学习的上海话语音合成的研究;王辉;《中国优秀硕士学位论文全文数据库信息科技辑》;20200615;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112802450A (en) | 2021-05-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112802450B (en) | Rhythm-controllable Chinese and English mixed speech synthesis method and system thereof | |
| CN112863483B (en) | Voice synthesizer supporting multi-speaker style and language switching and controllable rhythm | |
| CN112802448B (en) | Speech synthesis method and system for generating new tone | |
| Blaauw et al. | A neural parametric singing synthesizer | |
| CN111739508B (en) | End-to-end speech synthesis method and system based on DNN-HMM bimodal alignment network | |
| WO2020118521A1 (en) | Multi-speaker neural text-to-speech synthesis | |
| CN108899009B (en) | Chinese speech synthesis system based on phoneme | |
| CN113470662A (en) | Generating and using text-to-speech data for keyword spotting systems and speaker adaptation in speech recognition systems | |
| US20230169953A1 (en) | Phrase-based end-to-end text-to-speech (tts) synthesis | |
| KR102272554B1 (en) | Method and system of text to multiple speech | |
| Terashima et al. | Cross-speaker emotion transfer for low-resource text-to-speech using non-parallel voice conversion with pitch-shift data augmentation | |
| CN115620699A (en) | Speech synthesis method, speech synthesis system, speech synthesis apparatus, and storage medium | |
| KR20190135853A (en) | Method and system of text to multiple speech | |
| Choi et al. | A melody-unsupervision model for singing voice synthesis | |
| Li et al. | End-to-end mongolian text-to-speech system | |
| Kang et al. | Connectionist temporal classification loss for vector quantized variational autoencoder in zero-shot voice conversion | |
| CN115424604B (en) | Training method of voice synthesis model based on countermeasure generation network | |
| Nazir et al. | Deep learning end to end speech synthesis: A review | |
| WO2010104040A1 (en) | Voice synthesis apparatus based on single-model voice recognition synthesis, voice synthesis method and voice synthesis program | |
| Krug et al. | Articulatory synthesis for data augmentation in phoneme recognition | |
| CN116994553A (en) | Training method of speech synthesis model, speech synthesis method, device and equipment | |
| Chandra et al. | Towards the development of accent conversion model for (l1) bengali speaker using cycle consistent adversarial network (cyclegan) | |
| Ngoc et al. | Adapt-Tts: High-Quality Zero-Shot Multi-Speaker Text-to-Speech Adaptive-Based for Vietnamese | |
| Pitrelli et al. | Expressive speech synthesis using American English ToBI: questions and contrastive emphasis | |
| CN116403562B (en) | Speech synthesis method and system based on semantic information automatic prediction pause |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |