CN111191787B - Training method and device of neural network for extracting speaker embedded features - Google Patents
Training method and device of neural network for extracting speaker embedded features Download PDFInfo
- Publication number
- CN111191787B CN111191787B CN201911391244.6A CN201911391244A CN111191787B CN 111191787 B CN111191787 B CN 111191787B CN 201911391244 A CN201911391244 A CN 201911391244A CN 111191787 B CN111191787 B CN 111191787B
- Authority
- CN
- China
- Prior art keywords
- level
- segment
- layer
- speaker
- neural network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/60—Information retrieval; Database structures therefor; File system structures therefor of audio data
- G06F16/68—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/211—Selection of the most significant subset of features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Library & Information Science (AREA)
- Multimedia (AREA)
- Databases & Information Systems (AREA)
- Stereophonic System (AREA)
Abstract
The invention discloses a training method and a device of a neural network for extracting speaker embedded features, wherein the neural network comprises a plurality of frame level layers, a statistical pooling layer and a plurality of segment level layers, and the method comprises the following steps: receiving and processing an input audio clip via a plurality of frame-level layers; aggregating, via a statistical pooling layer, frame-level spectral features into segment-level spectral features; splitting a first multilayer linear layer on the basis of the statistical pooling layer to calculate a first channel loss of the segment-level spectrum characteristics; merging the segment-level spectral features into utterance-level spectral features via a plurality of segment-level layers, and calculating a speaker loss for the utterance-level spectral features; re-splitting a second multi-layered linear layer on the basis of the plurality of segment-level layers for calculating a second channel loss of the speech-level spectral feature; and training the neural network by controlling a sum of the first channel loss, the second channel loss, and the speaker loss. The neural network trained by the scheme of the application can extract the channel-independent speaker embedding characteristics.
Description
Technical Field
The invention belongs to the technical field of neural network training, and particularly relates to a training method and a training device for a neural network for extracting speaker embedded features.
Background
The purpose of Speaker Verification (SV) is to verify the identity of a user's requirements from his voice segment. Recently, Deep Neural Network (DNN) based speaker embedded learning has become the main method in this field. Researchers have studied different architectures, different loss functions, and different model compensation methods, which greatly improve the performance of SV systems.
Although deep learning techniques have enjoyed great success in the field of SV research, it is still very difficult to construct SV systems for practical use. It is well known that speaker verification is more vulnerable than speech recognition in terms of system robustness. To improve the robustness of SV systems, two sources of variability need to be addressed: voice content and channel variability. For text independent speaker verification, which requires that two utterances from the same speaker with different phonetic content be grouped together, it is important to cope with phoneme variations in the speaker modeling process. For text dependent and text independent speaker verification tasks in the real world using different devices and recording environments, system performance will drop dramatically due to this channel mismatch.
In the related art, for the characteristics or noise of the channel different from the speaker, front-end processing is generally used to remove the characteristics or noise of the channel different from the speaker, or countertraining is used to remove the characteristics or noise.
The purpose of these similar techniques is to eliminate the differences or noise that different channels cause to the characteristics of the same speaker, but in different ways.
The inventor finds in the process of implementing the present application that although the speaker-independent characteristics caused by the channel difference need to be eliminated, information about the channel can be utilized, and the utilization of the information can help the neural network to extract better acoustic features. None of the above-mentioned related art makes good use of the information of the channel.
Disclosure of Invention
The embodiment of the invention provides a training method and a training device for a neural network for extracting speaker embedded features, which are used for solving at least one of the technical problems.
In a first aspect, an embodiment of the present invention provides a training method for a neural network for extracting speaker embedded features, where the neural network includes a plurality of frame-level layers, a statistical pooling layer, and a plurality of segment-level layers, and the method includes: receiving and processing an input audio segment via the plurality of frame-level layers, wherein the plurality of frame-level layers are used to extract frame-level spectral features; aggregating, via the statistical pooling layer, the frame-level spectral features into segment-level spectral features; splitting a first multi-layer linear layer on the basis of the statistical pooling layer to calculate a first channel loss of the segment-level spectral feature; merging the segment-level spectral features into utterance-level spectral features via the plurality of segment-level layers and calculating a speaker loss for the utterance-level spectral features; re-splitting a second multi-layered linear layer on the basis of the plurality of segment-level layers for computing a second channel loss for the speech-level spectral feature; and training the neural network by controlling a sum of the first channel loss, the second channel loss, and the speaker loss.
In a second aspect, an embodiment of the present invention provides a training apparatus for a neural network for extracting speaker-embedded features, wherein the neural network includes a plurality of frame-level layers, a statistical pooling layer and a plurality of segment-level layers, and the apparatus includes: a receive processing module configured to receive and process an input audio segment via the plurality of frame-level layers, wherein the plurality of frame-level layers are used to extract frame-level spectral features; an aggregation module configured to aggregate the frame-level spectral features into segment-level spectral features via the statistics pooling layer; a first branching module configured to further split a first multi-layered linear layer based on the statistical pooling layer for calculating a first channel loss of the segment-level spectral feature; a merging module configured to merge the segment-level spectral features into utterance-level spectral features via the plurality of segment-level layers and to compute a speaker loss for the utterance-level spectral features; a second branching module configured to re-split a second multi-layered linear layer on the basis of the plurality of segment-level layers for calculating a second channel loss of the speech-level spectral feature; and a training module configured to train the neural network by controlling a sum of the first channel loss, the second channel loss, and the speaker loss.
In a third aspect, an electronic device is provided, comprising: the training method comprises at least one processor and a memory which is in communication connection with the at least one processor, wherein the memory stores instructions which can be executed by the at least one processor, and the instructions are executed by the at least one processor so as to enable the at least one processor to execute the steps of the training method for extracting the neural network of the embedded characteristic of the speaker according to any embodiment of the invention.
In a fourth aspect, the present invention further provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to execute the steps of the training method for extracting a neural network of speaker-embedded features according to any one of the embodiments of the present invention.
The method and the device provided by the application can help the neural network to extract better acoustic characteristics and eliminate the influence of the channel characteristics on the embedding characteristics of the speaker, and have better effect than the previous method for directly eliminating the channel characteristics.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the description below are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
FIG. 1 is a flowchart of a training method for a neural network for extracting speaker-embedded features according to an embodiment of the present invention;
FIG. 2 is a flowchart of a training method for a neural network for extracting speaker-embedded features according to an embodiment of the present invention;
FIG. 3 is a diagram of a network architecture of a neural network model of a neural network-based text classification method according to an embodiment of the present invention;
FIG. 4 is a diagram illustrating an embodiment of a training method for a neural network for extracting speaker-embedded features according to the present invention;
FIG. 5 is a block diagram of a training apparatus for neural network extraction of speaker-embedded features according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, which shows a flowchart of an embodiment of a training method for a neural network for extracting speaker-embedded features according to the present application, the training method for a neural network for extracting speaker-embedded features of the present embodiment may be applied to training a neural network for extracting speaker-embedded features, where the neural network may include a plurality of frame-level layers, a statistical pooling layer, and a plurality of segment-level layers.
As shown in fig. 1, in step 101, an input audio clip is received and processed via the plurality of frame level layers;
aggregating, in step 102, the frame-level spectral features into segment-level spectral features via the statistical pooling layer;
in step 103, splitting a first multi-layered linear layer based on the statistical pooling layer for calculating a first channel loss of the segment-level spectral feature;
in step 104, merging the segment-level spectral features into utterance-level spectral features via the plurality of segment-level layers, and calculating speaker loss for the utterance-level spectral features;
in step 105, splitting a second multi-layered linear layer on the basis of the plurality of segment-level layers for calculating a second channel loss of the speech-level spectral feature;
in step 106, the neural network is trained by controlling a sum of the first channel loss, the second channel loss, and the speaker loss.
In this embodiment, for step 101, the training device of the neural network for extracting speaker-embedded features receives and processes the input audio segment via a plurality of frame-level layers in the neural network, wherein the plurality of frame-level layers are used for extracting frame-level spectral features. Thereafter, for step 102, the frame-level spectral features are aggregated into segment-level spectral features via the statistical pooling layer. For step 103, a first multi-layered linear layer is split based on the re-statistical pooling layer to calculate a first channel loss of the segment-level spectral feature.
Then, for step 104, segment-level spectral features are merged into utterance-level spectral features via a plurality of segment-level layers, and speaker loss for the utterance-level spectral features is calculated at that layer. Thereafter, for step 105, a second multi-layered linear layer is split on the basis of the subsequent multi-layered layers for calculating a second channel loss of the speech-level spectral feature. Finally, for step 106, the neural network is trained by controlling the values of the first channel loss, the second channel loss, and the speaker loss.
The method of the embodiment calculates different losses by splitting branches at different layers respectively in the process of processing the audio, and then can train out the channel-independent speaker embedding characteristics by controlling the losses.
In some optional embodiments, the method further comprises: inserting a gradient inversion layer before the first multi-layer linear layer for counter training; and/or inserting a gradient inversion layer before the second multi-layer linear layer for counter training. For example, a gradient inversion layer may be inserted just before the second multi-layered linear layer to help the network eliminate the information of the channel during the competitive training, thereby extracting the channel-independent speaker-embedded features. The gradient inversion layer may be inserted only before the first multilayer linear layer, or the gradient inversion layer may be inserted before both the first multilayer linear layer and the second multilayer linear layer.
In fact, the inventors conducted separate experiments on the branch corresponding to speaker loss plus channel loss 1 (corresponding to the second channel loss), when the branch corresponding to channel loss 2 (corresponding to the first channel loss) is removed, or the branch corresponding to speaker loss plus channel loss 2.
In these two separate experiments, it was found that it would be better to insert a gradient inversion layer at channel loss 1 and to add no gradient inversion at channel loss 2.
Therefore, we believe that the speaker penalty plus the gradient flipping layer inserted channel penalty 1 and the non-gradient flipping channel penalty 2 are better (when the three penalties coexist), and the same is true.
Since it has been verified that the channel loss 2 without gradient inversion is better in the separate experiment of the speaker loss plus the channel loss 2, the combination of the speaker loss plus the channel loss 1 of the gradient inversion layer plus the channel loss 2 of the gradient inversion layer is not tested when the three are combined. We speculate that the results are not the best, but not too bad.
Therefore, in the shallow part of the network, the scheme of the embodiment can help the network to learn the information of the channel, so that better acoustic features can be extracted, and in the embedding feature level of the speaker, namely the deep layer of the neural network, the scheme of the embodiment can help the neural network to eliminate the features of the channel, so that the embedding features of the speaker irrelevant to the channel are extracted.
In some optional embodiments, the first channel penalty and the second channel penalty comprise using cross entropy calculations.
In other alternative embodiments, the speaker loss comprises using an additional angular margin loss calculation. This additional angular margin loss imposes a more stringent constraint that forces the similarity of the correct class to be larger than the similarity of the incorrect class by a margin m.
Further optionally, the plurality of frame-level layers include a time-delay neural network feature extractor, and the plurality of segment-level layers include a linear embedding layer. Therefore, the TDNN (Time Delay Neural Network) feature extractor can be used for extracting frame-level features better. The speaker embedding characteristics can be better extracted by utilizing the linear embedding layer.
Further optionally, the neural network comprises a deep neural network, and the deep neural network is used as a main method for embedding and learning of the speaker, and has the capability of better extracting embedding characteristics of the speaker.
The inventor finds that the drawbacks of the prior art are caused by the implementation means of these techniques, which aim to eliminate the channel difference and do not realize the utilization of the channel information.
Those skilled in the art are faced with the deficiencies of the prior art because the characteristics of the channel are detrimental to the final speaker-embedded feature, and it is common practice to eliminate the effects of the channel characteristics.
The characteristics of the channel are detrimental to the final speaker-embedded feature, and it is the most straightforward idea to eliminate this characteristic. The scheme of the present application using channel information is not easily conceivable.
In the scheme of the embodiment of the application, when the neural network is used for extracting the speaker embedded features, the neural network can be helped to learn the channel information at the bottom layer of the network, so that the network can learn better acoustic features. Then, at the speaker embedding feature level, the network is helped to eliminate the characteristics related to the channel, so that the extracted speaker features are independent of the channel.
FIG. 2 is a flow chart of a neural network for extracting channel independent speaker embedding features.
As shown in fig. 2, a header for classifying channels is first introduced in the statistical pooling layer, i.e. a shallow part of the neural network, to generate channel loss 2, which helps the network to learn the information of the channels, thereby helping the network to extract better acoustic features.
The speaker loses information that helps the network learn about the speaker.
That is, in the shallow part of the neural network, our design will help the network to learn the information of the channel, so as to extract better acoustic features, and in the speaker embedded feature level (deep layer of the neural network), our design will help the neural network to eliminate the characteristics of the channel, so as to extract the speaker embedded features irrelevant to the channel.
The embodiment of the application can directly achieve the following effects: the method can help the neural network to extract better acoustic characteristics and eliminate the influence of the channel characteristics on the embedding characteristics of the speaker, and has better effect than the previous method for directly eliminating the channel characteristics.
The embodiment of the application can realize the following deeper effects: from this experiment we can conclude that even though some acoustic properties are not useful in the final task, we can still use this existing information to help the network learn better acoustic features.
The following description is given to a specific example describing some problems encountered by the inventor in implementing the present invention and a final solution so as to enable those skilled in the art to better understand the solution of the present application.
Recently, much work has been done to mitigate the interference properties in certain tasks. Countermeasure training has been used in the speech domain to suppress speaker information in speech recognition, for example, domain countermeasure training for speaker anti-spoofing. Or use GAN and GRL strategies and perform countertraining to cope with channel variations in SV missions.
All of the above prior art aims at eliminating interference information in the main task, while antagonistic training is the most common method. However, it should be appreciated that there are two approaches that can take advantage of existing interference information, namely multitask learning and countervailing learning. Our previous work shows the possibility of combining two approaches to better exploit speech information in text-independent SV tasks: speech information is encouraged in early frame level layers, while such information is suppressed in later speaker embedding layers.
In the present application example, we follow similar ideas in previous work. We assume that even if we wish to obtain channel independent embedding, the available channel information can be used for better generic acoustic feature learning in the shallow model layer and then suppressed in later speaker embedding. As our experiments demonstrate, it is beneficial to apply multitasking learning and antagonistic training to speaker embedding before embedding the extraction layer. When we combine multitask learning with confrontational training, two training strategies were devised, including a joint mode and a progressive mode. Experiments were performed on TD-SV data sets based on the wake words. The best systems achieve a relative boost of 10.77% and 9.37% when using recording equipment and environment as channel information, respectively.
Related work
Deep Neural Networks (DNNs) are known for their powerful modeling capabilities and flexibility, and DNN-based speaker embedding has become the dominant speaker identification method. x-vector is a typical method used by many researchers. In the x-vector framework, a time-delay neural network (TDNN) is trained to distinguish different speakers in training data. The frame-level spectral features will first go through several frame-level layers, followed by a statistics pooling layer that aggregates the frame-level representations into a single segment-level representation. One or more embedding layers may be incorporated into the utterance level layer to extract speaker embedding. In our experiments, an x-vector extractor was used as the baseline and stem for our proposed framework.
Multitasking and antagonistic training
Our previous work has successfully combined multitasking and counter-training to better use speech information in text-independent speaker verification tasks. The whole system is shown in fig. 3.
FIG. 3 illustrates a structure that combines multitasking and antagonistic training to better utilize phonetic knowledge in a text-independent speaker verification task.
The main idea is to integrate speech information at the shallow level of the model, which facilitates generic feature learning and suppresses phoneme variations in the final speaker-embedded layer. As shown in fig. 3, in addition to the main speaker branch in the original x-vector framework, our proposed architecture also includes a frame-level multitask phoneme branch and a segment-level antagonistic phoneme branch. By combining the three branch supervisory signals, we observe a significant improvement in performance over the text independent speaker verification task.
Description of the model
Inspired by the success of previous work, we wanted to employ a similar strategy to learn channel independent speaker embedding, whereas channel implies recording equipment and environment in the experiment. With this new architecture, we wish to enhance the learning of channel variability for different training utterances on the shallow layer of the neural network, which are then suppressed in the latter embedding layer, ultimately resulting in better channel independent speaker embedding.
Model architecture
Unlike speech information, channel information is at the segment level, so unlike fig. 3, we will explore two locations for multitask and adversarial learning at the segment level.
Fig. 4 shows the proposed structure of applying channel level multitasking and antagonistic training at different positions of the model. For the first type, we now do not perform the multitasking/countermeasure training on the output of the last frame level layer, but rather split the branches after the merge layer. The second type is the same as in fig. 3, performed directly on the embedding layer. When antagonistic training is employed, a Gradient Reverse Layer (GRL) will be inserted into the normal multitasking branch to flip the sign of the computed gradient.
Loss function
For having speaker tag ysAnd channel label ycInput segment x of (1), total loss of model optimization by speaker loss LsAnd channel loss Lc1,Lc2Make up of
Lc1And Lc2Indicating channel loss with branches inserted into the x-vector embedding and statistics layers, respectively. Cross entropy will be used for channel classification branching I ∈ (1, 2), l denotes the number of channel classes.
For the speaker classification module, the key component of the model, we use the recently proposed additional angular margin loss as our main speaker loss. The additional angular margin loss imposes a more stringent constraint that forces the similarity of the correct class to be larger than the similarity of the incorrect class by a margin m.
Wherein,Is the normalized second linear layer output of the x-vector architecture.Representing weightsNormalized j column (j). n represents the number of speakers. The additional angular margin loss also increases the scale parameter s, which helps the model converge faster. For all experiments, we chose m = 0.2 and s = 30.
Training strategy
Assuming that channel information facilitates generic feature learning at the shallow layers of the model, but this information is not required for final speaker embedding, we have investigated two different training strategies to combine multitask learning and countertraining and obtain final channel independent speaker embedding.
Joint multi-task confrontation training
In this strategy, the entire architecture and all parameters are optimized simultaneously using multitask learning and antagonistic training, and model training is performed using three loss functions. Multitask learning will be applied to the statistics summarization layer and countermeasure training will be applied to the embedding layer.
Progressive multi-task confrontation training
In this case, we divide the model optimization into two phases and in the first phase first apply multitask learning over several training cycles. The multitask learning branch is then discarded, while the antagonistic training branch is in the second phase.
Experimental setup
Data set
The embodiment of the application uses a data set based on the awakening words. The average duration of all segments is about 1.0 second. Each person is required to repeatedly wake up words using a particular device in different environments. The data set is labeled with a device. But the environmental label is not well labeled on all devices. We selected 1.6M speech from 2k different speakers as the training set. To produce more training data, utterances recorded in quiet environments are enhanced using the noise of the MUSAN data set to ultimately obtain a 5.2M utterance training set.
For experiments using channel information, we consider here different recording devices and environments as available channel information. The environment represents a scene where recordings are collected, such as quiet, office and automobile, etc. Furthermore, we also consider the enhancement noise as a different environmental type. Experiments will be performed with respect to the device or environment, respectively, as channel information.
Data preparation using device type as channel information
For experiments that consider the device type as channel variability, all of the above training data was used for training, for a total of 5 device types. The other 20543 utterances from 94 speakers were not included in the training set for enrollment and testing. For each speaker, we selected 4 clean voices as enrollment data and used the remaining voices of that speaker to generate the target trials. In addition, for each enrolled speaker, all utterances of other speakers are used to generate non-targeted test samples. Finally, we obtained 20167 target and 636798 non-target tests.
Table 1 shows a basic x-vector extractor configuration
Data preparation using recording environment category as channel information
Since the environmental label is not consistent across all devices, we only performed this experiment on the data set on certain specific devices. We selected the data recorded by both devices from all training data for experimentation (hereinafter they will be referred to as Device1 (Device 1) and Device2 (Device 2) datasets, respectively). The number of environment types in both Device1 and Device2 datasets is 6. The Device1 data set consisted of 352 speakers and 594583 speech, while the Device2 data set consisted of 512 speakers and 841450 speech.
The present embodiment uses the same strategy as before to generate test trials. Finally, the Device1 test set contained 35 speakers, 8732 target trials and 324888 non-target trials. The Device2 test set contained 29 speakers, 5555 target trials and 158788 non-target trials. The results on both devices will be reported separately.
System configuration
The basic speaker-embedding extractor is an x-vector system with fewer parameters than the original parameters, a more detailed configuration is provided in Table 1. All architectures are implemented using a pytorech. The 40-dimensional Fbank features were extracted using the Kaldi toolkit and silence frames were deleted using an energy-based voice activity detector. The extracted embeddings are first length normalized and then the score is calculated using the biccovariance PLDA.
Baseline system
Our baseline system is a normal x-vector, whose structure is shown in Table 1, and only speaker classification loss will be used as the optimization goal. The margin of the additional angular margin loss m will increase linearly from 0.0 to 0.2 along the training iteration. We use the SGD optimizer to optimize our network and set the momentum and learning rate to 0.9 and 0.0001, respectively.
The proposed system
The channel classification block consists of three linear layers with a batchnorm layer inserted. The three linear layers have the size of (inputdiim) x (channel class number). When a multitasking or antagonistic responsible person is added to our reference network, the speaker and channel classification tasks will be jointly trained from scratch. Results and discussion
Using environment tags as channel information
Exploring environmental information at different locations of a model
The logical position of the multitasking and counter-branching is explored first. As shown in table 2, the performance of SV tasks can be improved by performing the training of confrontation at the embedding layer and furthermore, the result of multi-task training at the statistics pool layer is better than that obtained by the confrontation training, which verifies our previous assumption that channel information may contribute to the general feature learning of the shallower model layer.
Table 2 shows a comparison of the results of the multitasking or antagonistic training performed at different locations of the model using the context information, STAMT and STA-ADV indicating the multitasking or antagonistic training performed at the statistics summary level, and EMB-MT and EMB-ADV indicating that the correlation learning was performed at the embedding level.
TABLE 2 comparison of multitask or antagonistic training results at different locations of the model using environmental information
Joint and progressive multi-task confrontation training using environmental information
The previous experimental results of the embodiment of the application show that the environmental information is encouraged in the shallow layer (namely, the statistical pool layer) of the model, and the environmental information is inhibited in the embedded layer of the latter, so that the performance of the model can be improved. We then combine multitask learning and antagonistic training in the single architecture presented herein. Two training strategies were performed and compared, with the results shown in table 3. It can be seen that the proposed new architecture can be further improved and is always better under all conditions. For both training strategies, the progressive mode seems to be better than the joint mode. The best system may achieve a relative improvement of 10.77% on average compared to baseline.
Table 3 shows a comparison of two training strategies for the proposed architecture usage environment information, join representing the JOINT multitask confrontation training mode and progress representing the PROGRESSIVE multitask confrontation training mode of the proposed architecture.
TABLE 3 comparison of two training strategies for the proposed architecture usage environment information
Using device tags as channel information
In the present embodiment, we describe the results obtained when the device tag is used as channel information. In this embodiment, similar experiments as in the previous embodiments will be performed, and the device tag for each utterance will be used as a channel tag instead of an environment tag.
Table 4. comparison of different systems using device information.
The results of the multitask training and the countermeasure training using the device information are shown in table 4. As can be seen from the middle part of the table, it is preferable to insert multitask training and countermeasure training into the statistics pool and embedding layer, respectively, i.e. the conclusions in the previous embodiment. Furthermore, as with the results in the previous embodiments, the architecture integrates multitask learning and opponent training, which can be further improved. For both training strategies, the progressive mode was still slightly better, with a relative improvement of 9.37% compared to baseline.
From the above results, consistent observations can be obtained in both environment-based and device-based channel-independent training. It demonstrates the effectiveness of our proposed new framework and can achieve better channel independent speaker embedding through the new approach.
In the embodiment of the application, a framework combining multitasking and antagonistic training at different positions of a model is provided so as to better utilize channel information. Two independent experiments were performed to validate the proposed model, considering different devices or recording environments as channel labels. Consistent performance improvements were observed under both experimental conditions. The results indicate that enhancing channel information in the shallower layers of the model facilitates generic feature learning, while suppressing such information in the higher layers facilitates learning better channel independent speaker embedding. Two training strategies were designed to optimize the entire model and the new framework could achieve better performance. The progressive learning mode is slightly better than the joint learning mode. The optimal system achieved a relative improvement in EER of about 10.0% for both environmental and equipment levels compared to baseline.
Referring to fig. 5, a block diagram of a training apparatus for a neural network for extracting speaker-embedded features is shown, wherein the neural network includes a plurality of frame-level layers, a statistical pooling layer and a plurality of segment-level layers.
As shown in fig. 5, the training apparatus 500 for extracting a neural network of speaker-embedded features includes a receiving processing module 510, an aggregation module 520, a first branch module 530, a merging module 540, a second branch module 550, and a training module 560.
Wherein the receiving processing module 510 is configured to receive and process the input audio segment via the plurality of frame-level layers, wherein the plurality of frame-level layers are used for extracting frame-level spectral features; an aggregation module 520 configured to aggregate the frame-level spectral features into segment-level spectral features via the statistics pooling layer; a first splitting module 530 configured to split a first multi-layered linear layer on the basis of the statistical pooling layer for calculating a first channel loss of the segment-level spectral feature; a merging module 540 configured to merge the segment-level spectral features into utterance-level spectral features via the plurality of segment-level layers and calculate a speaker loss for the utterance-level spectral features; a second branching module 550 configured to further split a second multi-layered linear layer on the basis of the plurality of segment-level layers for calculating a second channel loss of the speech-level spectral feature; and a training module 560 configured to train the neural network by controlling a sum of the first channel loss, the second channel loss, and the speaker loss.
In some optional embodiments, the apparatus further comprises: inserting a gradient inversion layer before the first multi-layer linear layer for counter training; and/or a gradient inversion layer is inserted before the second multi-layer linear layer for counter training.
It should be understood that the modules recited in fig. 5 correspond to various steps in the method described with reference to fig. 1. Thus, the operations and features described above for the method and the corresponding technical effects are also applicable to the modules in fig. 5, and are not described again here.
It should be noted that the modules in the embodiments of the present application are not intended to limit the solution of the present application, for example, the receiving and processing module may be described as a module that receives and processes the input audio clip via the plurality of frame-level layers. In addition, the related functional modules may also be implemented by a hardware processor, for example, the receiving module may also be implemented by a processor, which is not described herein again.
In still other embodiments, an embodiment of the present invention further provides a non-transitory computer storage medium, where computer-executable instructions are stored, where the computer-executable instructions may perform a training method for a neural network for extracting speaker-embedded features in any of the above method embodiments, where the neural network includes a plurality of frame-level layers, a statistical pooling layer, and a plurality of segment-level layers;
as one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions configured to:
receiving and processing an input audio segment via the plurality of frame-level layers, wherein the plurality of frame-level layers are used to extract frame-level spectral features;
aggregating, via the statistical pooling layer, the frame-level spectral features into segment-level spectral features;
splitting a first multilayer linear layer on the basis of the statistical pooling layer for calculating a first channel loss of the segment-level spectral feature;
merging, via the plurality of segment-level layers, the segment-level spectral features into utterance-level spectral features and calculating a speaker loss for the utterance-level spectral features;
re-splitting a second multi-layered linear layer on the basis of the plurality of segment-level layers for computing a second channel loss for the speech-level spectral feature;
training the neural network by controlling a sum of the first channel loss, the second channel loss, and the speaker loss.
The non-volatile computer readable storage medium may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the stored data area may store data created from use of a training device that extracts a neural network of speaker-embedded features, and the like. Further, the non-volatile computer-readable storage medium may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, the non-transitory computer readable storage medium optionally includes memory located remotely from the processor, which may be connected over a network to a training device of a neural network that extracts the speaker embedded features. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
Embodiments of the present invention further provide a computer program product, the computer program product includes a computer program stored on a non-volatile computer-readable storage medium, the computer program includes program instructions, when the program instructions are executed by a computer, the computer executes any one of the above training methods for a neural network for extracting speaker-embedded features.
Fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 6, the electronic device includes: one or more processors 610 and a memory 620, with one processor 610 being an example in fig. 6. The device for the training method of the neural network for extracting the speaker embedded features may further include: an input device 630 and an output device 640. The processor 610, memory 620, input device 630, and output device 640 may be connected by a bus or other means, such as by bus in fig. 6. The memory 620 is a non-volatile computer-readable storage medium as described above. The processor 610 executes various functional applications of the server and data processing by running nonvolatile software programs, instructions and modules stored in the memory 620, so as to implement the training method of the neural network for extracting the speaker embedded features of the above method embodiments. The input device 630 may receive input numeric or character information and generate key signal inputs related to user settings and functional controls of the training device of the neural network that extracts speaker-embedded features. The output device 640 may include a display device such as a display screen.
The product can execute the method provided by the embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method. For technical details that are not described in detail in this embodiment, reference may be made to the method provided in the embodiment of the present invention.
As an embodiment, the electronic device is applied in a training apparatus of a neural network for extracting speaker embedded features, wherein the neural network includes a plurality of frame-level layers, a statistical pooling layer and a plurality of segment-level layers, and includes:
at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to:
receiving and processing an input audio segment via the plurality of frame-level layers, wherein the plurality of frame-level layers are used to extract frame-level spectral features;
aggregating, via the statistical pooling layer, the frame-level spectral features into segment-level spectral features;
splitting a first multi-layer linear layer on the basis of the statistical pooling layer to calculate a first channel loss of the segment-level spectral feature;
merging, via the plurality of segment-level layers, the segment-level spectral features into utterance-level spectral features and calculating a speaker loss for the utterance-level spectral features;
re-splitting a second multi-layered linear layer on the basis of the plurality of segment-level layers for computing a second channel loss for the speech-level spectral feature;
training the neural network by controlling a sum of the first channel loss, the second channel loss, and the speaker loss.
The electronic device of the embodiments of the present application exists in various forms, including but not limited to:
(1) a mobile communication device: such devices are characterized by mobile communications capabilities and are primarily targeted at providing voice and data communications. Such terminals include smart phones (e.g., iphones), multimedia phones, functional phones, and low-end phones, among others.
(2) Ultra mobile personal computer device: the equipment belongs to the category of personal computers, has calculation and processing functions and generally has mobile internet access characteristics. Such terminals include: PDA, MID, and UMPC devices, etc., such as ipads.
(3) A portable entertainment device: such devices may display and play multimedia content. Such devices include audio and video players (e.g., ipods), handheld game consoles, electronic books, and smart toys and portable car navigation devices.
(4) The server is similar to a general computer architecture, but has higher requirements on processing capacity, stability, reliability, safety, expandability, manageability and the like because of the need of providing high-reliability service.
(5) And other electronic devices with data interaction functions.
The above-described embodiments of the apparatus are merely illustrative, and units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A training method for a neural network to extract speaker-embedded features, wherein the neural network comprises a plurality of frame-level layers, a statistical pooling layer, and a plurality of segment-level layers, the method comprising:
receiving and processing an input audio segment via the plurality of frame-level layers, wherein the plurality of frame-level layers are used to extract frame-level spectral features;
aggregating, via the statistical pooling layer, the frame-level spectral features into segment-level spectral features;
splitting a first multi-layer linear layer on the basis of the statistical pooling layer to calculate a first channel loss of the segment-level spectral feature;
merging, via the plurality of segment-level layers, the segment-level spectral features into utterance-level spectral features and calculating a speaker loss for the utterance-level spectral features;
re-splitting a second multi-layered linear layer on the basis of the plurality of segment-level layers for computing a second channel loss for the speech-level spectral feature;
training the neural network by controlling a sum of the first channel loss, the second channel loss, and the speaker loss.
2. The method of claim 1, wherein the method further comprises: inserting a gradient inversion layer before the first multi-layer linear layer for counter training; and/or
A gradient inversion layer is inserted before the second multi-layer linear layer for counter training.
3. The method of claim 1, wherein the first channel penalty and the second channel penalty comprise using cross entropy calculations.
4. The method of claim 1, wherein the speaker loss comprises using an additional angular margin loss calculation.
5. The method of any of claims 1-4, wherein the plurality of frame level layers comprise a time-lapse neural network feature extractor and the plurality of segment level layers comprise a linear embedding layer.
6. The method of claim 5, wherein the neural network comprises a deep neural network.
7. A training apparatus for a neural network for extracting speaker-embedded features, wherein the neural network includes a plurality of frame-level layers, a statistical pooling layer, and a plurality of segment-level layers, the apparatus comprising:
a receive processing module configured to receive and process an input audio segment via the plurality of frame-level layers, wherein the plurality of frame-level layers are used to extract frame-level spectral features;
an aggregation module configured to aggregate the frame-level spectral features into segment-level spectral features via the statistical pooling layer;
a first branching module configured to further split a first multi-layered linear layer based on the statistical pooling layer for calculating a first channel loss of the segment-level spectral feature;
a merging module configured to merge the segment-level spectral features into utterance-level spectral features via the plurality of segment-level layers and to compute a speaker loss for the utterance-level spectral features;
a second branching module configured to re-split a second multi-layered linear layer on the basis of the plurality of segment-level layers for calculating a second channel loss of the speech-level spectral feature;
a training module configured to train the neural network by controlling a sum of the first channel loss, the second channel loss, and the speaker loss.
8. The apparatus of claim 7, further comprising:
inserting a gradient inversion layer before the first multi-layer linear layer for counter training; and/or
A gradient inversion layer is inserted before the second multi-layer linear layer for counter training.
9. An electronic device, comprising: at least one processor, and a memory communicatively coupled to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method of any one of claims 1 to 6.
10. A storage medium on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 6.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911391244.6A CN111191787B (en) | 2019-12-30 | 2019-12-30 | Training method and device of neural network for extracting speaker embedded features |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911391244.6A CN111191787B (en) | 2019-12-30 | 2019-12-30 | Training method and device of neural network for extracting speaker embedded features |
Publications (2)
Publication Number | Publication Date |
---|---|
CN111191787A CN111191787A (en) | 2020-05-22 |
CN111191787B true CN111191787B (en) | 2022-07-15 |
Family
ID=70707899
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201911391244.6A Active CN111191787B (en) | 2019-12-30 | 2019-12-30 | Training method and device of neural network for extracting speaker embedded features |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN111191787B (en) |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111816167B (en) * | 2020-07-22 | 2022-08-26 | 思必驰科技股份有限公司 | Speaker embedding learning method, speaker identification method and system |
CN111863003B (en) * | 2020-07-24 | 2022-04-15 | 思必驰科技股份有限公司 | Voice data enhancement method and device |
CN113129900A (en) * | 2021-04-29 | 2021-07-16 | 科大讯飞股份有限公司 | Voiceprint extraction model construction method, voiceprint identification method and related equipment |
Family Cites Families (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108417217B (en) * | 2018-01-11 | 2021-07-13 | 思必驰科技股份有限公司 | Speaker recognition network model training method, speaker recognition method and system |
CN108766440B (en) * | 2018-05-28 | 2020-01-14 | 平安科技(深圳)有限公司 | Speaker separation model training method, two-speaker separation method and related equipment |
CN110222841A (en) * | 2019-06-17 | 2019-09-10 | 苏州思必驰信息科技有限公司 | Neural network training method and device based on spacing loss function |
-
2019
- 2019-12-30 CN CN201911391244.6A patent/CN111191787B/en active Active
Also Published As
Publication number | Publication date |
---|---|
CN111191787A (en) | 2020-05-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN111191787B (en) | Training method and device of neural network for extracting speaker embedded features | |
CN109637546B (en) | Knowledge distillation method and apparatus | |
Barker et al. | The third ‘CHiME’speech separation and recognition challenge: Dataset, task and baselines | |
CN110136749B (en) | Method and device for detecting end-to-end voice endpoint related to speaker | |
Kwon et al. | The ins and outs of speaker recognition: lessons from VoxSRC 2020 | |
CN108417201B (en) | Single-channel multi-speaker identity recognition method and system | |
CN108109613B (en) | Audio training and recognition method for intelligent dialogue voice platform and electronic equipment | |
CN108417224B (en) | Training and recognition method and system of bidirectional neural network model | |
CN110706692B (en) | Training method and system of child voice recognition model | |
CN108766445A (en) | Method for recognizing sound-groove and system | |
CN111835784B (en) | Data generalization method and system for replay attack detection system | |
CN104036774A (en) | Method and system for recognizing Tibetan dialects | |
WO2017166651A1 (en) | Voice recognition model training method, speaker type recognition method and device | |
CN111862942B (en) | Method and system for training mixed speech recognition model of Mandarin and Sichuan | |
CN113362829B (en) | Speaker verification method, electronic device and storage medium | |
CN110349569B (en) | Method and device for training and identifying customized product language model | |
CN110753263A (en) | Video dubbing method, device, terminal and storage medium | |
CN106782604A (en) | The processing method of speech message, device and its electronic equipment | |
CN111863003B (en) | Voice data enhancement method and device | |
Cáceres et al. | The Biometric Vox system for the ASVspoof 2021 challenge | |
CN110232928B (en) | Text-independent speaker verification method and device | |
CN111081260A (en) | Method and system for identifying voiceprint of awakening word | |
CN110232927B (en) | Speaker verification anti-spoofing method and device | |
CN110600029A (en) | User-defined awakening method and device for intelligent voice equipment | |
CN106531184A (en) | Intelligent hardware piano music finishing training guidance method based on melody characteristic matching |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
CB02 | Change of applicant information | ||
CB02 | Change of applicant information |
Address after: 215123 building 14, Tengfei Innovation Park, 388 Xinping street, Suzhou Industrial Park, Suzhou City, Jiangsu Province Applicant after: Sipic Technology Co.,Ltd. Address before: 215123 building 14, Tengfei Innovation Park, 388 Xinping street, Suzhou Industrial Park, Suzhou City, Jiangsu Province Applicant before: AI SPEECH Co.,Ltd. |
|
GR01 | Patent grant | ||
GR01 | Patent grant |