WO2023134550A1 - 特征编码模型生成方法、音频确定方法以及相关装置 - Google Patents
特征编码模型生成方法、音频确定方法以及相关装置 Download PDFInfo
- Publication number
- WO2023134550A1 WO2023134550A1 PCT/CN2023/070800 CN2023070800W WO2023134550A1 WO 2023134550 A1 WO2023134550 A1 WO 2023134550A1 CN 2023070800 W CN2023070800 W CN 2023070800W WO 2023134550 A1 WO2023134550 A1 WO 2023134550A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- sample
- audio
- encoding
- feature
- audios
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/60—Information retrieval; Database structures therefor; File system structures therefor of audio data
- G06F16/68—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/683—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/60—Information retrieval; Database structures therefor; File system structures therefor of audio data
- G06F16/65—Clustering; Classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H1/00—Details of electrophonic musical instruments
- G10H1/0008—Associated control or indicating means
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
- G10H2210/031—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal
Definitions
- the present disclosure relates to the technical field of artificial intelligence, and in particular, to a method for generating a feature coding model, a method for determining audio, and related devices.
- the present disclosure provides a method for generating a feature encoding model, including:
- a target loss value of a target loss function based on a plurality of encoding vectors, a plurality of class prediction values of the sample audio, and a plurality of class labels of the sample audio, and update based on the target loss value parameters of the feature coding model to reduce the difference between the coding vectors of the sample audio belonging to the same class, increase the difference between the coding vectors of the sample audio belonging to different classes, and reducing the difference between the category prediction value and the label category of the plurality of sample audios to obtain the trained feature coding model.
- an audio determination method including:
- the second feature vectors of the multiple candidate audios are predetermined through the trained feature coding model
- the feature coding model is obtained according to the feature coding model generation method described in the first aspect.
- the present disclosure provides a training device for a feature encoding model, including:
- the first obtaining module is configured to obtain a plurality of audio samples marked with class labels
- a first extraction module configured to extract audio features of a plurality of said sample audios
- An encoding classification module configured to encode the audio features of the plurality of sample audios through the feature encoding model, obtain a plurality of encoding vectors of the plurality of sample audios, and pair the performing classification processing on a plurality of the sample audios to obtain category prediction values of the plurality of sample audios;
- the first determination module is configured to determine a target loss value of a target loss function according to a plurality of encoding vectors, a plurality of class prediction values of the sample audio, and a plurality of class labels of the sample audio, and update the parameters of the feature encoding model based on the target loss value, so as to reduce the difference between the encoding vectors of the sample audio belonging to the same category, and increase the encoding vectors of the sample audio belonging to different categories.
- the difference between encoding vectors, and reducing the difference between the category prediction value and the label category of a plurality of the sample audios are used to obtain the trained feature encoding model.
- an audio determination device including:
- the second obtaining module is configured to obtain the audio to be queried
- the second extraction module is configured to extract the audio features of the audio to be queried
- a processing module configured to process the audio to be queried according to the trained feature coding model to obtain a first feature vector of the audio to be queried;
- the second determining module is configured to determine from the reference feature library the audio to be queried based on the similarity between the first feature vector and the second feature vectors of multiple candidate audios in the reference feature library
- the target candidate audio belonging to the same audio; the second feature vectors of a plurality of the candidate audios are predetermined by the trained feature coding model;
- the feature coding model is obtained according to the feature coding model generation method described in the first aspect.
- the present disclosure provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processing device, the steps of the methods described in the first aspect and the second aspect are implemented.
- an electronic device including:
- At least one processing device configured to execute the at least one computer program in the storage device to implement the steps of the methods of the first aspect and the second aspect.
- the target loss value of the target loss function used for training the feature coding model can reduce the difference between the coding vectors of sample audio belonging to the same category, and increase the difference between the coding vectors of sample audio belonging to different categories. Inter-class differences, and reducing the difference between the category prediction value and label category of multiple sample audio, so that the feature encoding model pays attention to the inter-class distance while also paying attention to the intra-class distance, which improves the robustness of the trained feature encoding model and The trained feature encoding model can identify the feature vector of the audio output, thereby improving the accuracy of the search results of cover songs.
- Fig. 1 is a flowchart showing a method for generating a feature encoding model according to an exemplary embodiment of the present disclosure.
- Fig. 2 is a flow chart of determining a target loss value of a target loss function according to an exemplary embodiment of the present disclosure.
- Fig. 3 is a structural diagram of a feature encoding model according to an exemplary embodiment of the present disclosure.
- Fig. 4 is a flowchart showing an audio determination method according to an exemplary embodiment of the present disclosure.
- Fig. 5 is a block diagram showing a device for generating a feature encoding model according to an exemplary embodiment of the present disclosure.
- Fig. 6 is a block diagram of an audio determination device according to an exemplary embodiment of the present disclosure.
- Fig. 7 is a schematic structural diagram of an electronic device according to an exemplary embodiment of the present disclosure.
- the term “comprise” and its variations are open-ended, ie “including but not limited to”.
- the term “based on” is “based at least in part on”.
- the term “one embodiment” means “at least one embodiment”; the term “another embodiment” means “at least one further embodiment”; the term “some embodiments” means “at least some embodiments.” Relevant definitions of other terms will be given in the description below.
- the cover retrieval task may refer to retrieving a target audio belonging to the same audio from a music library for a given audio.
- the cover song retrieval task is regarded as a classification task, and the cover song model is trained according to the classification loss function, and then the feature vector of a given audio is obtained according to the cover song model, and the cover song retrieval task is completed based on the feature vector, wherein the cover song model includes Simple models for convolutional, pooling, and linear layers.
- the classification loss function emphasizes the distance between classes, and does not pay attention to the distance within the class, resulting in a large distance within the class, which makes the cover model unable to accurately classify the audio belonging to the same category, and then uses
- the feature vector output by the cover model cannot effectively distinguish the audio, which reduces the discriminability of the feature vector, thereby reducing the accuracy of the cover search results, and the robustness of the cover model trained in this way is poor; in addition,
- the structure of the cover song model is simple, so that the feature vector obtained by the cover song model cannot effectively represent the cover feature of the corresponding audio, which further reduces the accuracy of the cover song retrieval result.
- the embodiment of the present disclosure discloses a method for generating a feature coding model.
- the target loss value of the target loss function can reduce the difference between the coding vectors of sample audio belonging to the same category, and increase the difference between sample audio belonging to different categories.
- the difference between the encoding vectors and the difference between the category prediction value and the label category of multiple sample audios are reduced, so that the model pays attention to the inter-class distance while also paying attention to the intra-class distance, which improves the performance of the trained feature encoding model on the audio output.
- Fig. 1 is a flowchart showing a method for generating a feature encoding model according to an exemplary embodiment of the present disclosure. As shown in Figure 1, the method includes:
- Step 110 acquiring a plurality of audio samples marked with class labels.
- the sample audio may be data input into the feature encoding model for training the feature encoding model.
- Sample audio may include music data, such as songs.
- the label can be used to represent certain real information of the sample audio, and the category label can be used to represent the category of the sample audio.
- the sample audios belonging to the same audio among the multiple sample audios may be marked with the same category label.
- multiple sample audios may include different versions of each of the multiple songs, and the songs corresponding to different versions of the same song may be labeled with the same category label. It can be understood that, among multiple sample audios, sample audios that belong to the same audio and sample audios that do not belong to the same audio can be distinguished through the category label.
- category labels can be marked on multiple sample audios by manual labeling.
- a plurality of sample audios may be acquired through a storage device or by calling a related interface.
- Step 120 extract audio features of a plurality of audio samples.
- the audio features may include at least one of the following: spectral features, Mel spectral features, spectrogram features, and Constant-Q Transform (Constant-Q Transform, CQT) features.
- spectral features, mel spectral features, and spectrogram features of multiple sample audios may be extracted according to Fourier transform, and constant-Q transform features of multiple sample audios may be extracted according to constant-Q filters.
- corresponding audio features may be extracted according to corresponding audio processing libraries.
- the constant-Q transformation feature can reflect the pitch of the corresponding pitch position of the sample audio at each time unit, and the constant-Q transformation feature thus obtained is a two-dimensional pitch-time matrix, in which each The elements represent the pitch corresponding to the time and the corresponding pitch position.
- the time unit can be specifically set according to actual conditions, for example, 0.22s.
- the pitch positions can be specifically set according to actual conditions, for example, each octave has 12 pitch positions. It can be understood that the time unit and the pitch position can also be other values, for example, the time unit is 0.1, and the pitch position is 6 pitches per octave, which is not limited in this disclosure.
- the constant-Q transform feature contains time and pitch information
- the constant-Q transform feature can indirectly reflect the melody information of the sample audio. Since the adaptation (or cover) of music usually keeps the melody of the music unchanged as a whole, the melody information can better reflect whether the audios belong to the same audio, so that the trained feature encoding model can encode the audio output. It can effectively characterize the cover feature of the audio, and improve the accuracy of the cover search results. And in the music data, the sound is distributed exponentially, and the characteristics obtained by the Fourier transform are linearly distributed. The frequency points of the two cannot be one-to-one correspondence, which will cause errors in the estimated values of certain scale frequencies. Constant Q transform The feature has an exponential distribution law, which is consistent with the sound distribution of the music data, and is more suitable for search for cover songs, thereby improving the accuracy of search results for cover songs.
- Step 130 Encode the audio features of the multiple sample audios through the feature coding model to obtain multiple encoding vectors of the multiple sample audios, and classify the multiple sample audios according to the multiple encoding vectors to obtain the multiple sample audios category predictions.
- Step 140 determine the target loss value of the target loss function according to multiple encoding vectors, category prediction values of multiple sample audios, and category labels of multiple sample audios, and update the parameters of the feature coding model based on the target loss value to reduce The difference between the encoding vectors of sample audio belonging to the same category, increasing the difference between the encoding vectors of sample audio belonging to different categories, and reducing the difference between the category prediction value and label category of multiple sample audio, get the trained Feature Encoding Model.
- the parameters of the feature encoding model may be updated based on the target loss value until the target loss value satisfies a preset condition. For example, the target loss value converges, or the target loss value is smaller than a preset value. When the target loss value satisfies the preset condition, the feature encoding model training is completed, and a trained feature encoding model is obtained. For specific details about determining the target loss value of the target loss function, refer to FIG. 2 and its related descriptions, which will not be repeated here.
- the difference between the encoding vectors of the sample audio of the same category and the difference between the encoding vectors of the sample audio of different categories can be represented by the distance between the respective corresponding encoding vectors. Understandably, the smaller the distance, the smaller the difference. In some embodiments, the distance may include, but is not limited to, cosine distance, Euclidean distance, Manhattan distance, Mahalanobis distance, Minkowski distance, and the like.
- the difference between the coding vectors of the sample audio of the same class can represent the intra-class distance
- the difference between the coding vectors of the sample audio of different classes and the difference of the category prediction value and the label category of multiple sample audio Can represent the distance between classes.
- the loss value of the target loss function is related to both the inter-class distance and the intra-class distance.
- both the inter-class distance and the intra-class distance are paid attention to, which improves the robustness of the trained feature encoding model. and the discriminability of the feature vector (i.e., the encoding vector) output by the feature encoding model.
- the feature coding model can output The more similar the encoding vectors of , the more dissimilar the encoding vectors are for the sample audio outputs of different classes. It can be seen that the encoding vector output by the trained feature encoding model can effectively distinguish different audios, and further improve the discriminability of the feature vector output by the feature encoding model. It can improve the accuracy of the search result of the cover song.
- Fig. 2 is a flow chart of determining a target loss value of a target loss function according to an exemplary embodiment of the present disclosure. As shown in Figure 2, the method includes:
- a preset sample set is determined according to a plurality of audio samples, and a plurality of training sample groups are constructed according to the preset sample set, and each training sample group includes anchor samples, positive samples and negative samples.
- the preset sample set may be a sample set composed of some or all sample audios among the plurality of sample audios.
- the preset sample set may be composed of a preset number of randomly selected audio samples.
- the preset sample set may be composed of P*K sample audios selected from a plurality of sample audios, where P represents the number of categories, and the number of categories may refer to the number of different category labels included in the sample audios in the preset sample set Quantity, K represents the number of audio samples corresponding to each category in P categories, both P and K are positive integers greater than 1.
- the anchor sample is any sample audio in the preset sample set
- the positive sample is the sample audio that belongs to the same category as the anchor sample in the preset sample set
- the negative sample is that the preset sample set does not belong to the same category as the anchor sample
- the sample audio for .
- P*K training sample groups can be constructed through the preset sample set.
- Step 220 according to the encoding vector corresponding to the sample included in each training sample group, determine the first loss value of the first loss function, and, according to the category prediction value of multiple sample audios and the difference between the category labels of multiple sample audios, A second loss value for the second loss function is determined.
- the encoding vectors corresponding to the samples included in each training sample group may refer to the encoding vectors corresponding to the anchor samples, positive samples, and negative samples included in each training sample group.
- the first loss function is used to reflect the difference between the encoding vectors of the anchor samples and the encoding vectors of the positive samples and the encoding vectors of the negative samples.
- the difference between encoding vectors can be characterized by distance, so, in some embodiments, the distance between the encoding vector of the anchor sample and the encoding vector of the positive sample, and the distance between the encoding vector of the anchor sample and the negative The distance between the encoded vectors of the samples constructs the first loss function.
- the first loss function may be a triplet loss function
- the loss value of the triplet loss function ie, the first loss value of the first loss function
- loss tri represents the loss value of the triplet loss function
- represents an anchor sample represents a positive sample
- ⁇ indicates the threshold, which can be set according to the actual situation
- [] + indicates that when the value in “[]” is greater than 0, take this value as the loss value, and when it is less than 0, the loss value is 0 .
- the second loss function may be a classification loss function, for example, a cross-entropy loss function, and correspondingly, the second loss value of the second loss function may be a loss value of the cross-entropy loss function.
- the cross-entropy loss function please refer to relevant knowledge in the field, and will not repeat it here.
- Step 230 Determine a target loss value of the target loss function based on the first loss value of the first loss function and the second loss value of the second loss function.
- the target loss value may be determined according to a weighted summation result of the first loss value and the second loss value.
- the target loss function for training the feature encoding model is constructed by using the triplet loss function and the classification loss function, that is, using multiple loss functions to train the feature encoding model, so that the intra-class distance is well controlled , the boundaries of different categories are more obvious, thereby improving the discriminability of the feature vector of the feature encoding model for the audio output.
- the feature encoding model is trained in an end-to-end manner, which improves the convenience of model training.
- Fig. 3 is a structural diagram of a feature encoding model according to an exemplary embodiment of the present disclosure.
- the feature encoding model may include an encoding network 310 .
- the audio features of multiple sample audios are encoded by a feature coding model to obtain multiple encoding vectors of multiple sample audios, including: encoding the audio features of multiple sample audios according to the encoding network 310 to obtain Multiple encoding vectors for multiple samples of audio.
- encoding network 310 may comprise a residual network or a convolutional network.
- the residual network or the convolutional network can be specifically determined according to the actual situation.
- the residual network can include ResNet50 or ResNet50-IBN, and the convolutional network can include VGG16, etc.
- the residual network may include at least one of an IN (Instance Normalization, IN) layer and a BN (Batch Normailzation, BN) layer.
- ResNet50-IBN may include an IN layer and a BN layer.
- the IN layer Through the IN layer, the feature encoding network can learn the style-invariant features of music, and make better use of the diverse styles of music corresponding to multiple sample audios.
- the BN layer makes it easier to extract the content information of the sample audio, such as pitch, rhythm, Timbre, Volume, Genre, etc. It is easier to extract the information in the audio feature through the IN layer and the BN layer in the ResNet50-IBN network, so that the encoding vector output by the encoding network 310 can effectively represent the cover feature of the corresponding sample audio.
- the encoding network 310 may further include a GeM (Generalized mean, GeM) pooling layer. Encode the audio features of multiple sample audios according to the encoding network 310 to obtain multiple encoding vectors of multiple sample audios, including: encoding the audio features of multiple sample audios according to the residual network or convolutional network to obtain multiple Multiple initial encoding vectors of sample audio; multiple initial encoding vectors are processed according to the GeM pooling layer to obtain multiple encoding vectors of multiple sample audios.
- the GeM pooling layer can reduce the loss of features encoded from the residual network or convolutional network. For example, the GeM pooling layer can reduce the loss of features encoded from the ResNet50-IBN network, thereby improving the sample The validity of the cover feature represented by the encoding vector of the feature.
- the encoding vector output by the encoding network 310 of the trained feature encoding model can be used as the feature vector of the audio output by the feature encoding model.
- the encoding vector output by the residual network or the convolutional network in the encoding network 310 can be used as the feature vector of the audio output by the trained feature encoding model, or the GeM pooling layer in the encoding network 310 can be output The encoding vector of is used as the feature vector of the audio output by the trained feature encoding model.
- the feature encoding model includes a BN layer 320 and a classification layer 330
- the method for generating the feature encoding model further includes: processing multiple encoding vectors according to the BN layer 320 to obtain a plurality of regularized encoding vectors; Perform classification processing on multiple sample audios according to multiple encoding vectors to obtain category prediction values of multiple sample audios, including: performing classification processing on the regularized multiple encoding vectors according to the classification layer 330 to obtain categories of multiple sample audios Prediction value; wherein, the coding vector output by the BN layer 320 of the trained feature coding model can be used as the feature vector of the audio output by the feature coding model.
- the BN layer 320 may be disposed between the encoding network 310 or the GeM pooling layer, and the classification layer 330, and the BN layer 320 and the classification layer 330 constitute a BNNeck.
- the encoded vectors output by the encoding network 310 or the GeM pooling layer can be used to calculate the first loss value, and the multiple encoded vectors are processed by the BN layer 320 to obtain regularized encoded vectors. Regularization balances the encoded vectors The features of each dimension, and thus the second loss value calculated from the category prediction value obtained by classifying multiple encoded vectors after regularization is easier to converge.
- BNNeck reduces the restriction of the encoding vector of the second loss value before the BN layer (that is, the encoding vector output by the encoding network or the GeM pooling layer), and the less constraints of the second loss value make the first loss value easier to converge at the same time, Furthermore, BNNeck can improve the training efficiency of the feature encoding model. In addition, BNNeck can better maintain the boundary between classes, so that the feature encoding model and the feature vector of the feature encoding model for audio output can significantly enhance the discriminability and robustness.
- Fig. 4 is a flowchart showing an audio determination method according to an exemplary embodiment of the present disclosure. As shown in Figure 4, the method includes:
- Step 410 acquire the audio to be queried.
- Step 420 extract audio features of the audio to be queried.
- the audio to be queried may be an audio whose cover version needs to be queried. For example, you need to query for songs whose cover songs are sung.
- steps 410 and 420 are similar to the above steps 110 and 120, for details, please refer to the above steps 110 and 120, which will not be repeated here.
- Step 430 Process the audio features of the audio to be queried according to the trained feature coding model to obtain a first feature vector of the audio to be queried.
- the first feature vector of the audio to be queried may be the encoded network (for example, residual network, convolutional network or GeM pooling layer) or BN after the trained feature coding model processes the audio to be queried.
- the encoding vector for the output of the layer may be the encoded network (for example, residual network, convolutional network or GeM pooling layer) or BN after the trained feature coding model processes the audio to be queried.
- the encoding vector for the output of the layer.
- Step 440 based on the similarity between the first feature vector and the second feature vectors of multiple candidate audios in the reference feature library, determine from the reference feature library the target candidate audio that belongs to the same audio as the audio to be queried; multiple candidate audio
- the second feature vector of the audio is predetermined by the trained feature encoding model.
- the feature coding model is obtained according to the feature coding model generation method described in steps 110-140 above.
- belonging to the same audio may mean that the audio to be queried and the target candidate audio are different interpretations of the same audio, for example, the audio to be queried and the target candidate audio are different cover versions of the same song.
- candidate audios whose similarity is greater than a preset threshold may be determined as target candidate audios.
- the preset threshold can be specifically set according to actual conditions, for example, 0.95 or 0.98.
- the feature vector output by the special encoding model can accurately retrieve the target candidate audio belonging to the same audio as the audio to be queried, which improves the Accuracy of search results, that is, to improve the accuracy of search results of cover songs.
- Fig. 5 is a block diagram showing a device for generating a feature encoding model according to an exemplary embodiment of the present disclosure. As shown in Figure 5, the device 500 includes:
- the first obtaining module 510 is configured to obtain a plurality of sample audio marked with category labels
- the first extraction module 520 is configured to extract a plurality of audio features of the sample audio
- the encoding classification module 530 is configured to encode the audio features of the plurality of sample audios through the feature encoding model to obtain a plurality of encoding vectors of the plurality of sample audios, and according to the plurality of encoding vectors performing classification processing on a plurality of sample audios to obtain category prediction values of a plurality of sample audios;
- the first determining module 540 is configured to determine a target loss value of a target loss function according to a plurality of encoding vectors, a plurality of class prediction values of the sample audio, and a plurality of class labels of the sample audio , and update the parameters of the feature encoding model based on the target loss value, so as to reduce the difference between the encoding vectors of the sample audio belonging to the same category, and increase all the encoding vectors of the sample audio belonging to different categories.
- the difference between the encoding vectors, and the difference between the category prediction value and the label category of a plurality of the sample audios are reduced to obtain the trained feature encoding model.
- the first determining module 540 is further configured to:
- each of the training sample groups includes an anchor sample, a positive sample and a negative sample, wherein the anchor
- the sample is any sample audio in the preset sample set
- the positive sample is the sample audio belonging to the same category as the anchor sample in the preset sample set
- the negative sample is the preset sample set said sample audio does not belong to the same class as said anchor sample;
- the first loss function is used to reflect the encoding vector of the anchor sample and the The difference between the encoding vector of the positive sample and the encoding vector of the negative sample, and, based on the class prediction value of the plurality of audio samples and the class label of the audio samples difference, to determine the second loss value of the second loss function;
- the target loss value of the target loss function is determined based on the first loss value of the first loss function and the second loss value of the second loss function.
- the feature encoding model includes an encoding network
- the encoding classification module 530 is further configured to:
- the encoding network includes a residual network or a convolutional network; wherein, The encoding vector output by the encoding network of the trained feature encoding model can be used as the feature vector of the audio output by the feature encoding model.
- the residual network includes at least one of an IN layer and a BN layer.
- the encoding network further includes a GeM pooling layer
- the encoding classification module 530 is further configured to:
- the feature encoding model includes a BN layer and a classification layer
- the apparatus 500 further includes a regular processing module configured to perform a plurality of encoding vectors according to the BN layer Processing to obtain a plurality of encoded vectors after regularization;
- the coding classification module 530 is further configured to:
- the encoding vector output by the BN layer can be used as the feature vector of the audio output by the feature encoding model.
- Fig. 6 is a block diagram of an audio determination device according to an exemplary embodiment of the present disclosure. As shown in Figure 6, the device 600 includes:
- the second obtaining module 610 is configured to obtain the audio to be queried
- the second extraction module 620 extracts the audio features of the audio to be queried
- the processing module 630 is configured to process the audio to be queried according to the trained feature coding model to obtain a first feature vector of the audio to be queried;
- the second determination module 640 is configured to determine, from the reference feature library, the audio frequency to be queried based on the similarity between the first feature vector and the second feature vectors of multiple candidate audios in the reference feature library.
- the audio belongs to the target candidate audio of the same audio; the second feature vectors of multiple candidate audios are predetermined through the trained feature coding model; wherein, the feature coding model is determined according to an embodiment of the present disclosure obtained by the feature encoding model generation method described above.
- FIG. 7 it shows a schematic structural diagram of an electronic device 700 suitable for implementing the embodiments of the present disclosure.
- the terminal equipment in the embodiments of the present disclosure may include but not limited to mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), PMPs (Portable Multimedia Players), vehicle-mounted terminals (such as mobile terminals such as car navigation terminals) and fixed terminals such as digital TVs, desktop computers and the like.
- the electronic device shown in FIG. 7 is only an example, and should not limit the functions and application scope of the embodiments of the present disclosure.
- an electronic device 700 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) Various appropriate actions and processes are executed by programs in the memory (RAM) 703 . In the RAM 703, various programs and data necessary for the operation of the electronic device 700 are also stored.
- the processing device 701, ROM 702, and RAM 703 are connected to each other through a bus 704.
- An input/output (I/O) interface 705 is also connected to the bus 704 .
- the following devices can be connected to the I/O interface 705: input devices 706 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speaker, vibration an output device 707 such as a computer; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709.
- the communication means 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. While FIG. 7 shows electronic device 700 having various means, it should be understood that implementing or having all of the means shown is not a requirement. More or fewer means may alternatively be implemented or provided.
- embodiments of the present disclosure include a computer program product, which includes a computer program carried on a non-transitory computer readable medium, where the computer program includes program code for executing the method shown in the flowchart.
- the computer program may be downloaded and installed from a network via communication means 709, or from storage means 708, or from ROM 702.
- the processing device 701 the above-mentioned functions defined in the methods of the embodiments of the present disclosure are executed.
- the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two.
- a computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
- a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
- a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
- a computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device .
- Program code embodied on a computer readable medium may be transmitted by any appropriate medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
- any currently known or future network protocol such as HTTP (HyperText Transfer Protocol, Hypertext Transfer Protocol) can be used to communicate, and can communicate with digital data in any form or medium (for example, communication network) interconnection.
- Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network of.
- the above-mentioned computer-readable medium may be included in the above-mentioned electronic device, or may exist independently without being incorporated into the electronic device.
- the above-mentioned computer-readable medium carries at least one computer program, and when the above-mentioned at least one computer program is executed by the electronic device, the electronic device: acquires a plurality of audio samples marked with category tags; extracts a plurality of audio samples of the audio samples Features: encode the audio features of a plurality of sample audios by using the feature coding model to obtain a plurality of encoding vectors of the plurality of sample audios, and encode a plurality of the samples according to the plurality of encoding vectors performing audio classification processing to obtain a plurality of class prediction values of the sample audio; according to the plurality of encoding vectors, the plurality of class prediction values of the sample audio, and the plurality of class labels of the sample audio, determining a target loss value of the target loss function, and updating parameters of the feature coding model based on the target loss value, so as to reduce the difference between the coding vectors of the sample audio belonging to the same category, increase the difference between the coding vectors belonging to different The
- the above-mentioned computer-readable medium carries at least one computer program.
- the electronic device obtains the audio to be queried; extracts the audio features of the audio to be queried;
- the feature coding model of the audio to be queried is processed to obtain the first feature vector of the audio to be queried; based on the first feature vector and the second feature vector of a plurality of candidate audio in the reference feature library Similarity, determining from the reference feature library the target candidate audio that belongs to the same audio as the audio to be queried; the second feature vectors of multiple candidate audios are predetermined by the trained feature coding model wherein, the feature coding model is obtained according to the method for generating a feature coding model described in an embodiment of the present disclosure.
- Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, or combinations thereof, including but not limited to object-oriented programming languages—such as Java, Smalltalk, C++, and Includes conventional procedural programming languages - such as "C" or similar programming languages.
- the program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
- the remote computer may be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (for example, using an Internet service provider to connected via the Internet).
- LAN local area network
- WAN wide area network
- Internet service provider for example, using an Internet service provider to connected via the Internet.
- each block in a flowchart or block diagram may represent a module, program segment, or portion of code that contains one or more logical functions for implementing specified executable instructions.
- the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved.
- each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations can be implemented by a dedicated hardware-based system that performs the specified functions or operations , or may be implemented by a combination of dedicated hardware and computer instructions.
- modules involved in the embodiments described in the present disclosure may be implemented by software or by hardware. Wherein, the name of the module does not constitute a limitation on the module itself under certain circumstances.
- FPGAs Field Programmable Gate Arrays
- ASICs Application Specific Integrated Circuits
- ASSPs Application Specific Standard Products
- SOCs System on Chips
- CPLD Complex Programmable Logical device
- a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device.
- a machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium.
- a machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing.
- machine-readable storage media would include one or more wire-based electrical connections, portable computer discs, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.
- RAM random access memory
- ROM read only memory
- EPROM or flash memory erasable programmable read only memory
- CD-ROM compact disk read only memory
- magnetic storage or any suitable combination of the foregoing.
- Example 1 provides a method for generating a feature encoding model, including:
- a target loss value of a target loss function based on a plurality of encoding vectors, a plurality of class prediction values of the sample audio, and a plurality of class labels of the sample audio, and update based on the target loss value parameters of the feature coding model to reduce the difference between the coding vectors of the sample audio belonging to the same class, increase the difference between the coding vectors of the sample audio belonging to different classes, and reducing the difference between the category prediction value and the label category of the plurality of sample audios to obtain the trained feature coding model.
- Example 2 provides the method of Example 1, wherein the class prediction values of the plurality of sample audios and the class prediction values of the plurality of sample audios are The category label determines the target loss value of the target loss function, including:
- each of the training sample groups includes an anchor sample, a positive sample and a negative sample, wherein the anchor
- the sample is any sample audio in the preset sample set
- the positive sample is the sample audio belonging to the same category as the anchor sample in the preset sample set
- the negative sample is the preset sample set said sample audio does not belong to the same class as said anchor sample;
- the first loss function is used to reflect the encoding vector of the anchor sample and the The difference between the encoding vector of the positive sample and the encoding vector of the negative sample, and, based on the class prediction value of the plurality of audio samples and the class label of the audio samples difference, to determine the second loss value of the second loss function;
- the target loss value of the target loss function is determined based on the first loss value of the first loss function and the second loss value of the second loss function.
- Example 3 provides the method of Example 1, the feature encoding model includes an encoding network, and the audio features of a plurality of the sample audios are performed through the feature encoding model Encoding to obtain a plurality of encoding vectors of the sample audio, including:
- the encoding network includes a residual network or a convolutional network; wherein, The encoding vector output by the encoding network of the trained feature encoding model can be used as the feature vector of the audio output by the feature encoding model.
- Example 4 provides the method of Example 3, wherein the residual network includes at least one of an IN layer and a BN layer.
- Example 5 provides the method of Example 3, the encoding network further includes a GeM pooling layer, and the audio features of a plurality of the sample audios are analyzed according to the encoding network Encoding is performed to obtain a plurality of encoding vectors of the sample audio, including:
- Example 6 provides the method of any one of Examples 1-5, the feature encoding model includes a BN layer and a classification layer, and the method further includes:
- the classifying the plurality of sample audios according to the plurality of encoding vectors to obtain category prediction values of the plurality of sample audios includes:
- the encoding vector output by the BN layer can be used as the feature vector of the audio output by the feature encoding model.
- Example 7 provides an audio determination method, including:
- the second feature vectors of the multiple candidate audios are predetermined through the trained feature coding model
- the feature coding model is obtained according to the feature coding model generation method described in any one of Examples 1-6.
- Example 8 provides a training device for a feature encoding model, including:
- the first obtaining module is configured to obtain a plurality of audio samples marked with class labels
- a first extraction module configured to extract audio features of a plurality of said sample audios
- An encoding classification module configured to encode the audio features of the plurality of sample audios through the feature encoding model, obtain a plurality of encoding vectors of the plurality of sample audios, and pair the performing classification processing on a plurality of the sample audios to obtain category prediction values of the plurality of sample audios;
- the first determination module is configured to determine a target loss value of a target loss function according to a plurality of encoding vectors, a plurality of class prediction values of the sample audio, and a plurality of class labels of the sample audio, and update the parameters of the feature encoding model based on the target loss value, so as to reduce the difference between the encoding vectors of the sample audio belonging to the same category, and increase the encoding vectors of the sample audio belonging to different categories.
- the difference between encoding vectors, and reducing the difference between the category prediction value and the label category of a plurality of the sample audios are used to obtain the trained feature encoding model.
- Example 9 provides the apparatus of Example 8, the first determination module is further configured to:
- each of the training sample groups includes an anchor sample, a positive sample and a negative sample, wherein the anchor
- the sample is any sample audio in the preset sample set
- the positive sample is the sample audio belonging to the same category as the anchor sample in the preset sample set
- the negative sample is the preset sample set said sample audio does not belong to the same class as said anchor sample;
- the first loss function is used to reflect the encoding vector of the anchor sample and the The difference between the encoding vector of the positive sample and the encoding vector of the negative sample, and, based on the class prediction value of the plurality of audio samples and the class label of the audio samples difference, to determine the second loss value of the second loss function;
- the target loss value of the target loss function is determined based on the first loss value of the first loss function and the second loss value of the second loss function.
- Example 10 provides the apparatus of Example 8, the feature encoding model includes an encoding network, and the encoding classification module is further configured to:
- the encoding network includes a residual network or a convolutional network; wherein, The encoding vector output by the encoding network of the trained feature encoding model can be used as the feature vector of the audio output by the feature encoding model.
- Example 11 provides the apparatus of Example 10, wherein the residual network includes at least one of an IN layer and a BN layer.
- Example 12 provides the apparatus of Example 10, the encoding network further includes a GeM pooling layer, and the encoding classification module is further configured to:
- Example 13 provides the device of any one of Examples 8-12, the feature encoding model includes a BN layer and a classification layer, and the device further includes: a regularization processing module, the regularization The processing module is configured to: process the plurality of encoding vectors according to the BN layer to obtain the plurality of encoding vectors after regularization;
- the coding classification module is further configured to:
- the encoding vector output by the BN layer can be used as the feature vector of the audio output by the feature encoding model.
- Example 14 provides an audio determination device, comprising:
- the second obtaining module is configured to obtain the audio to be queried
- the second extraction module is configured to extract the audio features of the audio to be queried
- a processing module configured to process the audio features of the audio to be queried according to the trained feature coding model to obtain a first feature vector of the audio to be queried;
- the second determining module is configured to determine from the reference feature library the audio to be queried based on the similarity between the first feature vector and the second feature vectors of multiple candidate audios in the reference feature library
- the target candidate audio belonging to the same audio; the second feature vectors of a plurality of the candidate audios are predetermined by the trained feature coding model;
- the feature coding model is obtained according to the feature coding model generation method described in any one of Examples 1-6.
- Example 15 provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processing device, the steps of any one of the methods described in Examples 1-7 are implemented.
- Example 16 provides an electronic device, comprising:
- One or more processing devices configured to execute the one or more computer programs in the storage device to implement the steps of any one of the methods in Examples 1-7.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Biomedical Technology (AREA)
- Library & Information Science (AREA)
- Databases & Information Systems (AREA)
- Molecular Biology (AREA)
- Evolutionary Biology (AREA)
- Acoustics & Sound (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本公开涉及一种特征编码模型生成方法、音频确定方法以及相关装置,所述特征编码模型生成方法包括:获取标注有类别标签的多个样本音频;提取多个样本音频的音频特征;通过特征编码模型对多个样本音频的音频特征进行编码,得到多个样本音频的多个编码向量,以及根据多个编码向量对多个样本音频进行分类处理,得到多个样本音频的类别预测值;根据多个编码向量、多个样本音频的类别预测值以及多个样本音频的类别标签,确定目标损失函数的目标损失值,并基于目标损失值更新特征编码模型的参数,得到训练好的特征编码模型。本公开的特征编码模型生成方法得到的训练好的特征编码模型可以提高对音频输出的特征向量的可鉴别性以及特征编码模型的鲁棒性。
Description
相关申请的交叉引用
本申请要求于2022年01月14日提交的,申请号为202210045047.4、发明名称为“特征编码模型生成方法、音频确定方法以及相关装置”的中国专利申请的优先权,该申请的全部内容通过引用结合在本申请中。
本公开涉及人工智能技术领域,具体地,涉及一种特征编码模型生成方法、音频确定方法以及相关装置。
音乐作品通常包含了非常丰富的元素,例如,节奏、旋律、和声等,呈现出多层次的内部结构。因此,对于一个音乐作品的翻唱可以引入非常丰富的变化,使得该音乐作品在音调、音色、速度、结构、旋律以及歌词等多个方面发生变化。相关技术中,可以通过音频的特征向量确定音频之间是否属于同一音频,进而完成翻唱检索任务,然而,由于音频的变化方式多样,使得判断音频之间是否属于同一音频十分困难,因此,如何提高音频的特征向量的可鉴别性是亟需解决的技术问题。
发明内容
提供该部分内容以便以简要的形式介绍构思,这些构思将在后面的具体实施方式部分被详细描述。该部分内容并不旨在标识要求保护的技术方案的关键特征或必要特征,也不旨在用于限制所要求的保护的技术方案的范围。
第一方面,本公开提供一种特征编码模型生成方法,包括:
获取标注有类别标签的多个样本音频;
提取多个所述样本音频的音频特征;
通过所述特征编码模型对多个所述样本音频的所述音频特征进行编码,得到多个所述样本音频的多个编码向量,以及根据多个所述编码向量对多个所述样本音频进行分类处理,得到多个所述样本音频的类别预测值;
根据多个所述编码向量、多个所述样本音频的所述类别预测值以及多个所述样本音频的所述类别标签,确定目标损失函数的目标损失值,并基于所述目标损失值更新所述特征编码模型的参数,以减小属于同一类别的所述样本音频的所述编码向量之间的差异、增大属于不同类别的所述样本音频的所述编码向量之间的差异、以及减少多个所述样本音频的所述类别预测值和所述标签类别的差异,得到训练好的所述特征编码模型。
第二方面,本公开提供一种音频确定方法,包括:
获取待查询音频;
提取所述待查询音频的音频特征;
根据训练好的特征编码模型对所述待查询音频的所述音频特征进行处理,得到所述待查询音频的第一特征向量;
基于所述第一特征向量与参考特征库中的多个候选音频的第二特征向量之间的相似度,从所述参考特征库中确定与所述待查询音频属于同一音频的目标候选音频;多个所述候选音频的所述第二特征向量是通过训练好的所述特征编码模型预先确定的;
其中,所述特征编码模型是根据第一方面所述的特征编码模型生成方法得到的。
第三方面,本公开提供一种特征编码模型的训练装置,包括:
第一获取模块,被配置为获取标注有类别标签的多个样本音频;
第一提取模块,被配置为提取多个所述样本音频的音频特征;
编码分类模块,被配置为通过所述特征编码模型对多个所述样本音频的所述音频特征进行编码,得到多个所述样本音频的多个编码向量,以及根据多个所述编码向量对多个所述样本音频进行分类处理,得到多个所述样本音频的类别预测值;
第一确定模块,被配置为根据多个所述编码向量、多个所述样本音频的所述类别预测值以及多个所述样本音频的所述类别标签,确定目标损失函数的目标损失值,并基于所述目标损失值更新所述特征编码模型的参数,以减小属于同一类别的所述样本音频的所述编码向量之间的差异、增大属于不同类别的所述样本音频的所述编码向量之间的差异、以及减少多个所述样本音频的所述类别预测值和所述标签类别的差异,得到训练好的所述特征编码模型。
第四方面,本公开提供一种音频确定装置,包括:
第二获取模块,被配置为获取待查询音频;
第二提取模块,被配置为提取所述待查询音频的音频特征;
处理模块,被配置为根据训练好的特征编码模型对所述待查询音频进行处理,得到所述待查询音频的第一特征向量;
第二确定模块,被配置为基于所述第一特征向量与参考特征库中的多个候选音频的第二特征向量之间的相似度,从所述参考特征库中确定与所述待查询音频属于同一音频的目标候选音频;多个所述候选音频的所述第二特征向量是通过训练好的所述特征编码模型预先确定的;
其中,所述特征编码模型是根据第一方面所述的特征编码模型生成方法得到的。
第五方面,本公开提供一种计算机可读介质,其上存储有计算机程序,该程序被处理装置执行时实现第一方面和第二方面所述方法的步骤。
第六方面,本公开提供一种电子设备,包括:
存储装置,其上存储有至少一个计算机程序;
至少一个处理装置,用于执行所述存储装置中的所述至少一个计算机程序,以实现第一方面和第二方面所述方法的步骤。
通过上述技术方案,训练特征编码模型所使用的目标损失函数的目标损失值可以减小属于同一类别的样本音频的所述编码向量之间的差异、增大属于不同类别的样本音频的编码向量之间的差异、以及减少多个样本音频的类别预测值和标签类别的差异,使得特征编码模型在注意类间距离的同时也注意类内距离,提高了训练好的特征编码模型的鲁棒性以及训练好的特征编码模型对音频输出的特征向量的可鉴别性,进而提高翻唱检索结果的准确度。
本公开的其他特征和优点将在随后的具体实施方式部分予以详细说明。
结合附图并参考以下具体实施方式,本公开各实施例的上述和其他特征、优点及方面将变得更加明显。贯穿附图中,相同或相似的附图标记表示相同或相似的元素。应当理解附图是示意性的,原件和元素不一定按照比例绘制。在附图中:
图1是根据本公开一示例性实施例示出的一种特征编码模型生成方法的流程图。
图2是根据本公开一示例性实施例示出的确定目标损失函数的目标损失值的流程图。
图3是根据本公开一示例性实施例示出的特征编码模型的结构图。
图4是根据本公开一示例性实施例示出的一种音频确定方法的流程图。
图5是根据本公开一示例性实施例示出的一种特征编码模型生成装置的框图。
图6是根据本公开一示例性实施例示出的一种音频确定装置的框图。
图7是根据本公开一示例性实施例示出的电子设备的结构示意图。
下面将参照附图更详细地描述本公开的实施例。虽然附图中显示了本公开的某些实施例,然而应当理解的是,本公开可以通过各种形式来实现,而且不应该被解释为限于这里阐述的实施例,相反提供这些实施例是为了更加透彻和完整地理解本公开。应当理解的是,本公开的附图及实施例仅用于示例性作用,并非用于限制本公开的保护范围。
应当理解,本公开的方法实施方式中记载的各个步骤可以按照不同的顺序执行,和/或并行执行。此外,方法实施方式可以包括附加的步骤和/或省略执行示出的步骤。本公开的范围在此方面不受限制。
本文使用的术语“包括”及其变形是开放性包括,即“包括但不限于”。术语“基于”是“至少部分地基于”。术语“一个实施例”表示“至少一个实施例”;术语“另一实施例”表示“至少一个另外的实施例”;术语“一些实施例”表示“至少一些实施例”。其他术语的相关定义将在下文描述中给出。
需要注意,本公开中提及的“第一”、“第二”等概念仅用于对不同的装置、模块或单元进行区分,并非用于限定这些装置、模块或单元所执行的功能的顺序或者相互依存关系。
需要注意,本公开中提及的“一个”、“多个”的修饰是示意性而非限制性的,本领域技术人员应当理解,除非在上下文另有明确指出,否则应该理解为“一个或多个”。
本公开实施方式中的多个装置之间所交互的消息或者信息的名称仅用于说明性的目的,而并不是用于对这些消息或信息的范围进行限制。
翻唱检索任务可以是指针对给定的音频,从曲库中检索出与其属于同一音频的目标音频。相关技术中,将翻唱检索任务视为分类任务,根据分类损失函数训练得到翻唱模型,进而根据翻唱模型得到给定的音频的特征向量,基于该特征向量完成翻唱检索任务,其中,翻唱模型为包括卷积层、池化层和线性层的简单模型。
由于翻唱模型仅通过分类损失函数训练得到,分类损失函数强调类间距离,并不关注类内距离,导致类内距离较大,从而导致翻唱模型无法将属于同一类别的音频进行准确分类,进而利用该翻唱模型输出的特征向量无法有效区分音频,降低了特征向量的可鉴别性,从而降低了翻唱检索结果的准确度,且利用该方式训练得到的翻唱模型的鲁棒性差;除此之外,翻唱模型的结构简单,导致该翻唱模型得到的特征向量无法有效表示对应的音频的翻唱特征,进一步降低了翻唱检索结果的准确度。
因此,本公开实施例披露了一种特征编码模型生成方法,目标损失函数的目标损失值可以减小属于同一类别的样本音频的所述编码向量之间的差异、增大属于不同类别的样本音频的编码向量之间的差异、以及减少多个样本音频的类别预测值和标签类别的差异,使得模型在注意类间距离的同时也注意类内距离,提高了训练好的特征编码模型对音频输出的特征向量的可鉴别性,进而提高翻唱检索结果的准确度和特征编码模型的鲁棒性,且对特征编码模型的结构进行了优化,进一步提高了特征编码模型对音频输出的特征向量的可鉴别性以及翻唱检索结果的准确度。
以下将结合附图,以翻唱检索为例对本公开所披露的技术方案进行详细阐述。应当理解,本公开所披露的音频确定方法以及特征编码模型可以用于其他基于特征向量进行音频检索的场景,例如,根据特征向量进行音频消重,即消除一组音频中的重复音频。
图1是根据本公开一示例性实施例示出的一种特征编码模型生成方法的流程图。如图1所示,该方法包括:
步骤110,获取标注有类别标签的多个样本音频。
在一些实施例中,样本音频可以是输入至特征编码模型中用于训练特征编码模型的数据。样本音频可以包括音乐数据,例如歌曲。在一些实施例中,标签可以用于表征样本音频的某种真实信息,类别标签可以用于表征样本音频的类别。
在一些实施例中,可以将多个样本音频中属于同一音频的样本音频标注相同的类别标签。例如,以样本音频为歌曲为例,多个样本音频可以包括多首歌曲中每一首的不同版本,对同一歌曲的不同版本对 应的歌曲可以标注相同的类别标签。可以理解的,通过该类别标签可以区分多个样本音频中属于同一音频的样本音频,以及不属于同一音频的样本音频。
在一些实施例中,可以通过人工标注的方式对多个样本音频进行类别标签的标注。在一些实施例中,可以通过存储设备或调用相关的接口获取多个样本音频。
步骤120,提取多个样本音频的音频特征。
在一些实施例中,音频特征可以包括以下至少一种:频谱特征、梅尔频谱特征、语谱图特征、以及恒Q变换(Constant-Q Transform,CQT)特征。在一些实施例中,可以根据傅里叶变换提取多个样本音频的频谱特征、梅尔频谱特征、以及语谱图特征,根据恒Q滤波器提取多个样本音频的恒Q变换特征。在一些实施例中,可以根据对应音频处理库提取对应音频特征。在一些实施例中,还可以通过在特征编码模型中设置音频特征提取层,根据音频特征提取层提取多个样本音频的音频特征,值得说明的是,音频特征可以由特征编码模型得到,也可以独立于特征编码模型另外得到。
在一些实施例中,恒Q变换特征可以反映样本音频在各个时间单位处对应的音高位置的音高,由此得到的恒Q变换特征是一个二维的音高-时间矩阵,矩阵中每个元素表示对应时间以及对应音高位置的音高。在一些实施例中,时间单位可以根据实际情况具体设置,例如,0.22s。在一些实施例中,音高位置可以根据实际情况具体设置,例如,每个八度取12个音高位置。可以理解的,时间单位和音高位置还可以为其他数值,例如,时间单位为0.1,音高位置为每个八度取6个音高,本公开对此不作任何限制。
由于恒Q变换特征包含时间和音高信息,因此,恒Q变换特征可以间接反映样本音频的旋律信息。由于对音乐的改编(或翻唱)通常会保持音乐的旋律走向整体不变,因此,通过旋律信息更能反映音频之间是否属于同一音频,进而使得训练好的特征编码模型对音频输出的编码向量能有效表征该音频的翻唱特征,提高翻唱检索结果的准确率。且在音乐数据中,声音是以指数分布的,傅里叶变换得到的特征为线性分布的,两者的频率点不能一一对应,会使某些音阶频率的估计值产生误差,恒Q变换特征具备指数分布规律,其与音乐数据的声音的分布一致,更适合于进行翻唱检索,进而提高翻唱检索结果的准确度。
步骤130,通过特征编码模型对多个样本音频的音频特征进行编码,得到多个样本音频的多个编码向量,以及根据多个编码向量对多个样本音频进行分类处理,得到多个样本音频的类别预测值。
关于特征编码模型进行编码和分类处理的具体细节可以参见图3及其相关描述,在此不再赘述。
步骤140,根据多个编码向量、多个样本音频的类别预测值以及多个样本音频的类别标签,确定目标损失函数的目标损失值,并基于目标损失值更新特征编码模型的参数,以减小属于同一类别的样本音频的编码向量之间的差异、增大属于不同类别的样本音频的编码向量之间的差异、以及减少多个样本音频的类别预测值和标签类别的差异,得到训练好的特征编码模型。
在一些实施例中,可以基于目标损失值更新特征编码模型的参数,直至目标损失值满足预设条件。 例如,目标损失值收敛,或目标损失值小于预设值。当目标损失值满足预设条件时,特征编码模型训练完成,得到训练好的特征编码模型。关于确定目标损失函数的目标损失值的具体细节可以参见图2及其相关描述,在此不再赘述。
在一些实施例中,同一类别的样本音频的编码向量之间的差异以及不同类别的样本音频的编码向量之间的差异可以通过各自对应的编码向量之间的距离表征。可以理解的,距离越小,差异越小。在一些实施例中,距离可以包括但不限于余弦距离、欧氏距离、曼哈顿距离、马氏距离或闵可夫斯基距离等。
在一些实施例中,同一类别的样本音频的编码向量之间的差异可以表征类内距离,不同类别的样本音频的编码向量之间的差异以及多个样本音频的类别预测值和标签类别的差异可以表征类间距离。由此可知,目标损失函数的损失值与类间距离和类内距离均相关,特征编码模型的训练过程中同时关注了类间距离和类内距离,提高了训练好的特征编码模型的鲁棒性以及特征编码模型输出的特征向量(即编码向量)的可鉴别性。
在一些实施例中,通过减小属于同一类别的样本音频的编码向量之间的差异、增大属于不同类别的样本音频的编码向量之间的差异,使得特征编码模型对同一类的样本音频输出的编码向量越相似,对不同类的样本音频输出的编码向量越不相似。由此可知,通过训练好的特征编码模型输出的编码向量可以有效区分不同音频,进一步提高特征编码模型对音频输出的特征向量的可鉴别性,通过该特征编码模型输出的特征向量进行翻唱检索,可以提高翻唱检索结果的准确度。
图2是根据本公开一示例性实施例示出的确定目标损失函数的目标损失值的流程图。如图2所示,该方法包括:
步骤210,根据多个样本音频确定预设样本集,并根据预设样本集构造多个训练样本组,每一训练样本组包括锚样本、正样本以及负样本。
在一些实施例中,预设样本集可以是由多个样本音频中的部分样本音频或全部样本音频构成的样本集。在一些实施例中,预设样本集可以由随机选取的预设数量的样本音频构成。示例地,预设样本集可以由多个样本音频中选取的P*K个样本音频构成,其中,P表示类别数量,该类别数量可以是指预设样本集中的样本音频包括的不同类别标签的数量,K表示P个类别中每个类别对应的样本音频数量,P和K均为大于1的正整数。
在一些实施例中,锚样本为预设样本集中的任一样本音频,正样本为预设样本集中与锚样本属于同一类别的样本音频,负样本为预设样本集中与锚样本不属于同一类别的所述样本音频。示例地,仍以上述预设样本集包括P*K个样本音频为例,则通过该预设样本集可以构造P*K个训练样本组。
步骤220,根据每一训练样本组包括的样本对应的编码向量,确定第一损失函数的第一损失值,以及,根据多个样本音频的类别预测值以及多个样本音频的类别标签的差异,确定第二损失函数的第二损失值。
在一些实施例中,每一训练样本组包括的样本对应的编码向量可以是指每一训练样本组包括的锚样 本、正样本以及负样本对应的编码向量。在一些实施例中,第一损失函数用于反映锚样本的编码向量与正样本的编码向量和负样本的编码向量之间的差异。如前所述,编码向量之间的差异可以通过距离表征,因此,在一些实施例中,可以通过锚样本的编码向量与正样本的编码向量之间的距离,以及锚样本的编码向量与负样本的编码向量之间的距离构造第一损失函数。
在一些实施例中,第一损失函数可以是三元组损失函数,三元组损失函数的损失值(即第一损失函数的第一损失值)可以通过下述公式(1)得到:
其中,loss
tri表示三元组损失函数的损失值,
表示锚样本,
表示正样本,
表示锚样本与正样本的距离,
表示负样本,
表示锚样本与负样本的距离,∝表示阈值,可根据实际情况具体设置,[]
+表示“[]”内的值大于0时,取该值为损失值,小于0时,损失值为0。
在一些实施例中,第二损失函数可以是分类损失函数,例如,交叉熵损失函数,对应的,第二损失函数的第二损失值可以是交叉熵损失函数的损失值。关于交叉熵损失函数可以参见本领域相关知识,在此不再赘述。
步骤230,基于第一损失函数的第一损失值和第二损失函数的第二损失值,确定目标损失函数的目标损失值。
在一些实施例中,可以根据第一损失值和第二损失值的加权求和的结果,确定目标损失值。在本公开实施例中,通过采用三元组损失函数和分类损失函数构建训练特征编码模型的目标损失函数,即使用多个损失函数对特征编码模型进行训练,使得类内距离得到了良好的控制,不同类别的边界更加明显,从而提升了特征编码模型对音频输出的特征向量的可鉴别性。且,特征编码模型采用端到端的方式训练得到,提升了模型训练的便利性。
图3是根据本公开一示例性实施例示出的特征编码模型的结构图。如图3所示,特征编码模型可以包括编码网络310。在一些实施例中,通过特征编码模型对多个样本音频的音频特征进行编码,得到多个样本音频的多个编码向量,包括:根据编码网络310对多个样本音频的音频特征进行编码,得到多个样本音频的多个编码向量。
在一些实施例中,编码网络310可以包括残差网络或卷积网络。残差网络或卷积网络可以根据实际情况具体确定,例如,残差网络可以包括ResNet50或ResNet50-IBN,卷积网络可以包括VGG16等。
在一些实施例中,残差网络可以包括IN(Instance Normalization,IN)层和BN(Batch Normailzation,BN)层中的至少之一。在一些实施例中,ResNet50-IBN可以包括IN层和BN层。通过IN层使得特征编码网络能学习到音乐的风格不变特征,更好地利用多个样本音频对应的风格多样化的音乐,BN层更容易提取样本音频的内容信息,例如,音调、节奏、音色、音量、流派等。通过ResNet50-IBN网络中的IN层和BN层更容易提取音频特征中的信息,使得编码网络310输出的编码向量能有效表示对应的样本 音频的翻唱特征。
在一些实施例中,编码网络310还可以包括GeM(Generalized mean,GeM)池化层。根据编码网络310对多个样本音频的音频特征进行编码,得到多个样本音频的多个编码向量,包括:根据残差网络或卷积网络对多个样本音频的音频特征进行编码,得到多个样本音频的多个初始编码向量;根据GeM池化层对多个初始编码向量进行处理,得到多个样本音频的多个编码向量。通过GeM池化层可以减少从残差网络或卷积网络对音频特征进行编码后的特征的损耗,例如,GeM池化层可以减少从ResNet50-IBN网络进行编码后的特征的损耗,进而提高样本特征的编码向量所表征的翻唱特征的有效性。
在一些实施例中,训练好的特征编码模型的编码网络310输出的编码向量能够作为该特征编码模型输出的音频的特征向量。在一些实施例中,可以将编码网络310中的残差网络或卷积网络输出的编码向量作为训练好的特征编码模型输出的音频的特征向量,或者将编码网络310中的GeM池化层输出的编码向量作为训练好的特征编码模型输出的音频的特征向量。
在一些实施例中,特征编码模型包括BN层320和分类层330,所述特征编码模型生成方法还包括:根据BN层320对多个编码向量进行处理,得到正则化后的多个编码向量;根据多个编码向量对多个样本音频进行分类处理,得到多个样本音频的类别预测值,包括:根据分类层330对正则化后的多个编码向量进行分类处理,得到多个样本音频的类别预测值;其中,训练好的特征编码模型的BN层320输出的编码向量能够作为该特征编码模型输出的音频的特征向量。
在一些实施例中,BN层320可以设置于编码网络310或GeM池化层,与分类层330之间,该BN层320和分类层330构成了BNNeck。通过编码网络310或GeM池化层输出的编码向量可以用于计算第一损失值,通过BN层320对多个编码向量进行处理得到正则化后的多个编码向量,正则化平衡了编码向量中各个维度的特征,由此通过正则化后的多个编码向量进行分类处理得到的类别预测值所计算的第二损失值更容易收敛。BNNeck减少了第二损失值在BN层之前的编码向量(即通过编码网络或GeM池化层输出的编码向量)的限制,第二损失值更少的约束使得第一损失值同时更容易收敛,进而,通过BNNeck能提高特征编码模型的训练效率。除此之外,BNNeck能更好地维护类间边界,使特征编码模型以及特征编码模型对音频输出的特征向量显著增强了可鉴别性和鲁棒性。
图4是根据本公开一示例性实施例示出的一种音频确定方法的流程图。如图4所示,该方法包括:
步骤410,获取待查询音频。
步骤420,提取待查询音频的音频特征。
在一些实施例中,待查询音频可以是需要查询其翻唱版本的音频。例如,需要查询其翻唱歌曲的歌曲。关于步骤410和420的具体细节与上述步骤110和120类似,具体可参见上述步骤110和120,在此不再赘述。
步骤430,根据训练好的特征编码模型对待查询音频的音频特征进行处理,得到待查询音频的第一特征向量。
在一些实施例中,待查询音频的第一特征向量可以是训练好的特征编码模型对待查询音频进行处理后,其编码网络(例如,残差网络、卷积网络或GeM池化层)或BN层输出的编码向量。关于步骤430的具体细节可以参见上述图3中的相关描述,在此不再赘述。
步骤440,基于第一特征向量与参考特征库中的多个候选音频的第二特征向量之间的相似度,从参考特征库中确定与待查询音频属于同一音频的目标候选音频;多个候选音频的第二特征向量是通过训练好的特征编码模型预先确定的。
在一些实施例中,特征编码模型是根据上述步骤110-140所述的特征编码模型生成方法得到。在一些实施例中,属于同一音频可以是指待查询音频和目标候选音频是对同一音频的不同演绎,例如,待查询音频和目标候选音频是同一首歌曲的不同翻唱版本。
在一些实施例中,可以将相似度大于预设阈值的候选音频确定为目标候选音频。预设阈值可以根据实际情况具体设置,例如,0.95或0.98等。在本公开实施例中,由于特征编码模型输出的特征向量的可鉴别性高,因此,通过特别编码模型输出的特征向量可以准确检索出与待查询音频属于同一音频的目标候选音频时,提高了检索结果的准确度,即提高翻唱检索结果的准确度。
图5是根据本公开一示例性实施例示出的一种特征编码模型生成装置的框图。如图5所示,该装置500包括:
第一获取模块510,被配置为获取标注有类别标签的多个样本音频;
第一提取模块520,被配置为提取多个所述样本音频的音频特征;
编码分类模块530,被配置为通过所述特征编码模型对多个所述样本音频的所述音频特征进行编码,得到多个所述样本音频的多个编码向量,以及根据多个所述编码向量对多个所述样本音频进行分类处理,得到多个所述样本音频的类别预测值;
第一确定模块540,被配置为根据多个所述编码向量、多个所述样本音频的所述类别预测值以及多个所述样本音频的所述类别标签,确定目标损失函数的目标损失值,并基于所述目标损失值更新所述特征编码模型的参数,以减小属于同一类别的所述样本音频的所述编码向量之间的差异、增大属于不同类别的所述样本音频的所述编码向量之间的差异、以及减少多个所述样本音频的所述类别预测值和所述标签类别的差异,得到训练好的所述特征编码模型。
在一些实施例中,所述第一确定模块540进一步被配置为:
根据多个所述样本音频确定预设样本集,并根据所述预设样本集构造多个训练样本组,每一所述训练样本组包括锚样本、正样本以及负样本,其中,所述锚样本为所述预设样本集中的任一样本音频,所述正样本为所述预设样本集中与所述锚样本属于同一类别的所述样本音频,所述负样本为所述预设样本集中与所述锚样本不属于同一类别的所述样本音频;
根据每一所述训练样本组包括的样本对应的所述编码向量,确定第一损失函数的第一损失值,所述第一损失函数用于反映所述锚样本的所述编码向量与所述正样本的所述编码向量和所述负样本的所述 编码向量之间的差异,以及,根据多个所述样本音频的所述类别预测值以及多个所述样本音频的所述类别标签的差异,确定第二损失函数的第二损失值;
基于所述第一损失函数的所述第一损失值和所述第二损失函数的所述第二损失值,确定所述目标损失函数的所述目标损失值。
在一些实施例中,所述特征编码模型包括编码网络,所述编码分类模块530进一步被配置为:
根据所述编码网络对多个所述样本音频的所述音频特征进行编码,得到多个所述样本音频的多个所述编码向量;所述编码网络包括残差网络或卷积网络;其中,训练好的所述特征编码模型的所述编码网络输出的编码向量能够作为该特征编码模型输出的音频的特征向量。
在一些实施例中,所述残差网络包括有IN层和BN层中的至少之一。
在一些实施例中,所述编码网络还包括GeM池化层,所述编码分类模块530进一步被配置为:
根据所述残差网络或所述卷积网络对多个所述样本音频的所述音频特征进行编码,得到多个所述样本音频的多个初始编码向量;
根据所述GeM池化层对多个所述初始编码向量进行处理,得到多个所述样本音频的多个所述编码向量。
在一些实施例中,所述特征编码模型包括BN层和分类层,所述装置500还包括正则处理模块,所述正则处理模块被配置为:根据所述BN层对多个所述编码向量进行处理,得到正则化后的多个所述编码向量;
所述编码分类模块530进一步被配置为:
根据所述分类层对所述正则化后的多个所述编码向量进行所述分类处理,得到多个所述样本音频的所述类别预测值;其中,训练好的所述特征编码模型的所述BN层输出的编码向量能够作为该特征编码模型输出的音频的特征向量。
图6是根据本公开一示例性实施例示出的一种音频确定装置的框图。如图6所示,该装置600包括:
第二获取模块610,被配置为获取待查询音频;
第二提取模块620,提取所述待查询音频的音频特征;
处理模块630,被配置为根据训练好的特征编码模型对所述待查询音频进行处理,得到所述待查询音频的第一特征向量;
第二确定模块640,被配置为基于所述第一特征向量与参考特征库中的多个候选音频的第二特征向量之间的相似度,从所述参考特征库中确定与所述待查询音频属于同一音频的目标候选音频;多个所述候选音频的所述第二特征向量是通过训练好的所述特征编码模型预先确定的;其中,所述特征编码模型是根据本公开实施例所述的特征编码模型生成方法得到的。
下面参考图7,其示出了适于用来实现本公开实施例的电子设备700的结构示意图。本公开实施例中的终端设备可以包括但不限于诸如移动电话、笔记本电脑、数字广播接收器、PDA(个人数字助理)、 PAD(平板电脑)、PMP(便携式多媒体播放器)、车载终端(例如车载导航终端)等等的移动终端以及诸如数字TV、台式计算机等等的固定终端。图7示出的电子设备仅仅是一个示例,不应对本公开实施例的功能和使用范围带来任何限制。
如图7所示,电子设备700可以包括处理装置(例如中央处理器、图形处理器等)701,其可以根据存储在只读存储器(ROM)702中的程序或者从存储装置708加载到随机访问存储器(RAM)703中的程序而执行各种适当的动作和处理。在RAM 703中,还存储有电子设备700操作所需的各种程序和数据。处理装置701、ROM 702以及RAM 703通过总线704彼此相连。输入/输出(I/O)接口705也连接至总线704。
通常,以下装置可以连接至I/O接口705:包括例如触摸屏、触摸板、键盘、鼠标、摄像头、麦克风、加速度计、陀螺仪等的输入装置706;包括例如液晶显示器(LCD)、扬声器、振动器等的输出装置707;包括例如磁带、硬盘等的存储装置708;以及通信装置709。通信装置709可以允许电子设备700与其他设备进行无线或有线通信以交换数据。虽然图7示出了具有各种装置的电子设备700,但是应理解的是,并不要求实施或具备所有示出的装置。可以替代地实施或具备更多或更少的装置。
特别地,根据本公开的实施例,上文参考流程图描述的过程可以被实现为计算机软件程序。例如,本公开的实施例包括一种计算机程序产品,其包括承载在非暂态计算机可读介质上的计算机程序,该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中,该计算机程序可以通过通信装置709从网络上被下载和安装,或者从存储装置708被安装,或者从ROM 702被安装。在该计算机程序被处理装置701执行时,执行本公开实施例的方法中限定的上述功能。
需要说明的是,本公开上述的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的***、装置或器件,或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于:具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本公开中,计算机可读存储介质可以是任何包含或存储程序的有形介质,该程序可以被指令执行***、装置或者器件使用或者与其结合使用。而在本公开中,计算机可读信号介质可以包括在基带中或者作为载波一部分传播的数据信号,其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式,包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读信号介质还可以是计算机可读存储介质以外的任何计算机可读介质,该计算机可读信号介质可以发送、传播或者传输用于由指令执行***、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输,包括但不限于:电线、光缆、RF(射频)等等,或者上述的任意合适的组合。
在一些实施方式中,可以利用诸如HTTP(HyperText Transfer Protocol,超文本传输协议)之类的任 何当前已知或未来研发的网络协议进行通信,并且可以与任意形式或介质的数字数据通信(例如,通信网络)互连。通信网络的示例包括局域网(“LAN”),广域网(“WAN”),网际网(例如,互联网)以及端对端网络(例如,ad hoc端对端网络),以及任何当前已知或未来研发的网络。
上述计算机可读介质可以是上述电子设备中所包含的;也可以是单独存在,而未装配入该电子设备中。
上述计算机可读介质承载有至少一个计算机程序,当上述至少一个计算机程序被该电子设备执行时,使得该电子设备:获取标注有类别标签的多个样本音频;提取多个所述样本音频的音频特征;通过所述特征编码模型对多个所述样本音频的所述音频特征进行编码,得到多个所述样本音频的多个编码向量,以及根据多个所述编码向量对多个所述样本音频进行分类处理,得到多个所述样本音频的类别预测值;根据多个所述编码向量、多个所述样本音频的所述类别预测值以及多个所述样本音频的所述类别标签,确定目标损失函数的目标损失值,并基于所述目标损失值更新所述特征编码模型的参数,以减小属于同一类别的所述样本音频的所述编码向量之间的差异、增大属于不同类别的所述样本音频的所述编码向量之间的差异、以及减少多个所述样本音频的所述类别预测值和所述标签类别的差异,得到训练好的所述特征编码模型。
或者,上述计算机可读介质承载有至少一个计算机程序,当上述至少一个计算机程序被该电子设备执行时,使得该电子设备:获取待查询音频;提取所述待查询音频的音频特征;根据训练好的特征编码模型对所述待查询音频进行处理,得到所述待查询音频的第一特征向量;基于所述第一特征向量与参考特征库中的多个候选音频的第二特征向量之间的相似度,从所述参考特征库中确定与所述待查询音频属于同一音频的目标候选音频;多个所述候选音频的所述第二特征向量是通过训练好的所述特征编码模型预先确定的;其中,所述特征编码模型是根据本公开实施例所述的特征编码模型生成方法得到的。
可以以一种或多种程序设计语言或其组合来编写用于执行本公开的操作的计算机程序代码,上述程序设计语言包括但不限于面向对象的程序设计语言—诸如Java、Smalltalk、C++,还包括常规的过程式程序设计语言——诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中,远程计算机可以通过任意种类的网络——包括局域网(LAN)或广域网(WAN)——连接到用户计算机,或者,可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
附图中的流程图和框图,图示了按照本公开各种实施例的***、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上,流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分,该模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意,在有些作为替换的实现中,方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如,两个接连地表示的方框实际上可以基本并行地执行,它们有时也可以按相反的顺序执行,这依所 涉及的功能而定。也要注意的是,框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合,可以用执行规定的功能或操作的专用的基于硬件的***来实现,或者可以用专用硬件与计算机指令的组合来实现。
描述于本公开实施例中所涉及到的模块可以通过软件的方式实现,也可以通过硬件的方式来实现。其中,模块的名称在某种情况下并不构成对该模块本身的限定。
本文中以上描述的功能可以至少部分地由一个或多个硬件逻辑部件来执行。例如,非限制性地,可以使用的示范类型的硬件逻辑部件包括:现场可编程门阵列(FPGA)、专用集成电路(ASIC)、专用标准产品(ASSP)、片上***(SOC)、复杂可编程逻辑设备(CPLD)等等。
在本公开的上下文中,机器可读介质可以是有形的介质,其可以包含或存储以供指令执行***、装置或设备使用或与指令执行***、装置或设备结合地使用的程序。机器可读介质可以是机器可读信号介质或机器可读储存介质。机器可读介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体***、装置或设备,或者上述内容的任何合适组合。机器可读存储介质的更具体示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM或快闪存储器)、光纤、便捷式紧凑盘只读存储器(CD-ROM)、光学储存设备、磁储存设备、或上述内容的任何合适组合。
根据本公开的一个或多个实施例,示例1提供了一种特征编码模型生成方法,包括:
获取标注有类别标签的多个样本音频;
提取多个所述样本音频的音频特征;
通过所述特征编码模型对多个所述样本音频的所述音频特征进行编码,得到多个所述样本音频的多个编码向量,以及根据多个所述编码向量对多个所述样本音频进行分类处理,得到多个所述样本音频的类别预测值;
根据多个所述编码向量、多个所述样本音频的所述类别预测值以及多个所述样本音频的所述类别标签,确定目标损失函数的目标损失值,并基于所述目标损失值更新所述特征编码模型的参数,以减小属于同一类别的所述样本音频的所述编码向量之间的差异、增大属于不同类别的所述样本音频的所述编码向量之间的差异、以及减少多个所述样本音频的所述类别预测值和所述标签类别的差异,得到训练好的所述特征编码模型。
根据本公开的一个或多个实施例,示例2提供了示例1的方法,所述根据多个所述编码向量、多个所述样本音频的所述类别预测值以及多个所述样本音频的所述类别标签,确定目标损失函数的目标损失值,包括:
根据多个所述样本音频确定预设样本集,并根据所述预设样本集构造多个训练样本组,每一所述训练样本组包括锚样本、正样本以及负样本,其中,所述锚样本为所述预设样本集中的任一样本音频,所述正样本为所述预设样本集中与所述锚样本属于同一类别的所述样本音频,所述负样本为所述预设样本 集中与所述锚样本不属于同一类别的所述样本音频;
根据每一所述训练样本组包括的样本对应的所述编码向量,确定第一损失函数的第一损失值,所述第一损失函数用于反映所述锚样本的所述编码向量与所述正样本的所述编码向量和所述负样本的所述编码向量之间的差异,以及,根据多个所述样本音频的所述类别预测值以及多个所述样本音频的所述类别标签的差异,确定第二损失函数的第二损失值;
基于所述第一损失函数的所述第一损失值和所述第二损失函数的所述第二损失值,确定所述目标损失函数的所述目标损失值。
根据本公开的一个或多个实施例,示例3提供了示例1的方法,所述特征编码模型包括编码网络,所述通过所述特征编码模型对多个所述样本音频的所述音频特征进行编码,得到多个所述样本音频的多个编码向量,包括:
根据所述编码网络对多个所述样本音频的所述音频特征进行编码,得到多个所述样本音频的多个所述编码向量;所述编码网络包括残差网络或卷积网络;其中,训练好的所述特征编码模型的所述编码网络输出的编码向量能够作为该特征编码模型输出的音频的特征向量。
根据本公开的一个或多个实施例,示例4提供了示例3的方法,所述残差网络包括有IN层和BN层中的至少之一。
根据本公开的一个或多个实施例,示例5提供了示例3的方法,所述编码网络还包括GeM池化层,所述根据所述编码网络对多个所述样本音频的所述音频特征进行编码,得到多个所述样本音频的多个所述编码向量,包括:
根据所述残差网络或所述卷积网络对多个所述样本音频的所述音频特征进行编码,得到多个所述样本音频的多个初始编码向量;
根据所述GeM池化层对多个所述初始编码向量进行处理,得到多个所述样本音频的多个所述编码向量。
根据本公开的一个或多个实施例,示例6提供了示例1-5中任一项的方法,所述特征编码模型包括BN层和分类层,所述方法还包括:
根据所述BN层对多个所述编码向量进行处理,得到正则化后的多个所述编码向量;
所述根据多个所述编码向量对多个所述样本音频进行分类处理,得到多个所述样本音频的类别预测值,包括:
根据所述分类层对所述正则化后的多个所述编码向量进行所述分类处理,得到多个所述样本音频的所述类别预测值;其中,训练好的所述特征编码模型的所述BN层输出的编码向量能够作为该特征编码模型输出的音频的特征向量。
根据本公开的一个或多个实施例,示例7提供一种音频确定方法,包括:
获取待查询音频;
提取所述待查询音频的音频特征;
根据训练好的特征编码模型对所述待查询音频的所述音频特征进行处理,得到所述待查询音频的第一特征向量;
基于所述第一特征向量与参考特征库中的多个候选音频的第二特征向量之间的相似度,从所述参考特征库中确定与所述待查询音频属于同一音频的目标候选音频;多个所述候选音频的所述第二特征向量是通过训练好的所述特征编码模型预先确定的;
其中,所述特征编码模型是根据示例1-6中任一项所述的特征编码模型生成方法得到的。
根据本公开的一个或多个实施例,示例8提供一种特征编码模型的训练装置,包括:
第一获取模块,被配置为获取标注有类别标签的多个样本音频;
第一提取模块,被配置为提取多个所述样本音频的音频特征;
编码分类模块,被配置为通过所述特征编码模型对多个所述样本音频的所述音频特征进行编码,得到多个所述样本音频的多个编码向量,以及根据多个所述编码向量对多个所述样本音频进行分类处理,得到多个所述样本音频的类别预测值;
第一确定模块,被配置为根据多个所述编码向量、多个所述样本音频的所述类别预测值以及多个所述样本音频的所述类别标签,确定目标损失函数的目标损失值,并基于所述目标损失值更新所述特征编码模型的参数,以减小属于同一类别的所述样本音频的所述编码向量之间的差异、增大属于不同类别的所述样本音频的所述编码向量之间的差异、以及减少多个所述样本音频的所述类别预测值和所述标签类别的差异,得到训练好的所述特征编码模型。
根据本公开的一个或多个实施例,示例9提供了示例8的装置,所述第一确定模块进一步被配置为:
根据多个所述样本音频确定预设样本集,并根据所述预设样本集构造多个训练样本组,每一所述训练样本组包括锚样本、正样本以及负样本,其中,所述锚样本为所述预设样本集中的任一样本音频,所述正样本为所述预设样本集中与所述锚样本属于同一类别的所述样本音频,所述负样本为所述预设样本集中与所述锚样本不属于同一类别的所述样本音频;
根据每一所述训练样本组包括的样本对应的所述编码向量,确定第一损失函数的第一损失值,所述第一损失函数用于反映所述锚样本的所述编码向量与所述正样本的所述编码向量和所述负样本的所述编码向量之间的差异,以及,根据多个所述样本音频的所述类别预测值以及多个所述样本音频的所述类别标签的差异,确定第二损失函数的第二损失值;
基于所述第一损失函数的所述第一损失值和所述第二损失函数的所述第二损失值,确定所述目标损失函数的所述目标损失值。
根据本公开的一个或多个实施例,示例10提供了示例8的装置,所述特征编码模型包括编码网络,所述编码分类模块进一步被配置为:
根据所述编码网络对多个所述样本音频的所述音频特征进行编码,得到多个所述样本音频的多个所 述编码向量;所述编码网络包括残差网络或卷积网络;其中,训练好的所述特征编码模型的所述编码网络输出的编码向量能够作为该特征编码模型输出的音频的特征向量。
根据本公开的一个或多个实施例,示例11提供了示例10的装置,所述残差网络包括有IN层和BN层中的至少之一。
根据本公开的一个或多个实施例,示例12提供了示例10的装置,所述编码网络还包括GeM池化层,所述编码分类模块进一步被配置为:
根据所述残差网络或所述卷积网络对多个所述样本音频的所述音频特征进行编码,得到多个所述样本音频的多个初始编码向量;
根据所述GeM池化层对多个所述初始编码向量进行处理,得到多个所述样本音频的多个所述编码向量。
根据本公开的一个或多个实施例,示例13提供了示例8-12任一项的装置,所述特征编码模型包括BN层和分类层,所述装置还包括:正则处理模块,所述正则处理模块被配置为:根据所述BN层对多个所述编码向量进行处理,得到正则化后的多个所述编码向量;
所述编码分类模块进一步被配置为:
根据所述分类层对所述正则化后的多个所述编码向量进行所述分类处理,得到多个所述样本音频的所述类别预测值;其中,训练好的所述特征编码模型的所述BN层输出的编码向量能够作为该特征编码模型输出的音频的特征向量。
根据本公开的一个或多个实施例,示例14提供一种音频确定装置,包括:
第二获取模块,被配置为获取待查询音频;
第二提取模块,被配置为提取所述待查询音频的音频特征;
处理模块,被配置为根据训练好的特征编码模型对所述待查询音频的所述音频特征进行处理,得到所述待查询音频的第一特征向量;
第二确定模块,被配置为基于所述第一特征向量与参考特征库中的多个候选音频的第二特征向量之间的相似度,从所述参考特征库中确定与所述待查询音频属于同一音频的目标候选音频;多个所述候选音频的所述第二特征向量是通过训练好的所述特征编码模型预先确定的;
其中,所述特征编码模型是根据示例1-6中任一项所述的特征编码模型生成方法得到的。
根据本公开的一个或多个实施例,示例15提供一种计算机可读介质,其上存储有计算机程序,该程序被处理装置执行时实现示例1-7中任一项所述方法的步骤。
根据本公开的一个或多个实施例,示例16提供一种电子设备,包括:
存储装置,其上存储有一个或多个计算机程序;
一个或多个处理装置,用于执行所述存储装置中的所述一个或多个计算机程序,以实现示例1-7中任一项所述方法的步骤。
以上描述仅为本公开的较佳实施例以及对所运用技术原理的说明。本领域技术人员应当理解,本公开中所涉及的公开范围,并不限于上述技术特征的特定组合而成的技术方案,同时也应涵盖在不脱离上述公开构思的情况下,由上述技术特征或其等同特征进行任意组合而形成的其它技术方案。例如上述特征与本公开中公开的(但不限于)具有类似功能的技术特征进行互相替换而形成的技术方案。
此外,虽然采用特定次序描绘了各操作,但是这不应当理解为要求这些操作以所示出的特定次序或以顺序次序执行来执行。在一定环境下,多任务和并行处理可能是有利的。同样地,虽然在上面论述中包含了若干具体实现细节,但是这些不应当被解释为对本公开的范围的限制。在单独的实施例的上下文中描述的某些特征还可以组合地实现在单个实施例中。相反地,在单个实施例的上下文中描述的各种特征也可以单独地或以任何合适的子组合的方式实现在多个实施例中。
尽管已经采用特定于结构特征和/或方法逻辑动作的语言描述了本主题,但是应当理解所附权利要求书中所限定的主题未必局限于上面描述的特定特征或动作。相反,上面所描述的特定特征和动作仅仅是实现权利要求书的示例形式。关于上述实施例中的装置,其中各个模块执行操作的具体方式已经在有关该方法的实施例中进行了详细描述,此处将不做详细阐述说明。
Claims (11)
- 一种特征编码模型生成方法,其包括:获取标注有类别标签的多个样本音频;提取多个所述样本音频的音频特征;通过所述特征编码模型对多个所述样本音频的所述音频特征进行编码,得到多个所述样本音频的多个编码向量,以及根据多个所述编码向量对多个所述样本音频进行分类处理,得到多个所述样本音频的类别预测值;根据多个所述编码向量、多个所述样本音频的所述类别预测值以及多个所述样本音频的所述类别标签,确定目标损失函数的目标损失值,并基于所述目标损失值更新所述特征编码模型的参数,以减小属于同一类别的所述样本音频的所述编码向量之间的差异、增大属于不同类别的所述样本音频的所述编码向量之间的差异、以及减少多个所述样本音频的所述类别预测值和所述标签类别的差异,得到训练好的所述特征编码模型。
- 根据权利要求1所述的方法,其中,所述根据多个所述编码向量、多个所述样本音频的所述类别预测值以及多个所述样本音频的所述类别标签,确定目标损失函数的目标损失值,包括:根据多个所述样本音频确定预设样本集,并根据所述预设样本集构造多个训练样本组,每一所述训练样本组包括锚样本、正样本以及负样本,其中,所述锚样本为所述预设样本集中的任一样本音频,所述正样本为所述预设样本集中与所述锚样本属于同一类别的所述样本音频,所述负样本为所述预设样本集中与所述锚样本不属于同一类别的所述样本音频;根据每一所述训练样本组包括的样本对应的所述编码向量,确定第一损失函数的第一损失值,所述第一损失函数用于反映所述锚样本的所述编码向量与所述正样本的所述编码向量和所述负样本的所述编码向量之间的差异,以及,根据多个所述样本音频的所述类别预测值以及多个所述样本音频的所述类别标签的差异,确定第二损失函数的第二损失值;基于所述第一损失函数的所述第一损失值和所述第二损失函数的所述第二损失值,确定所述目标损失函数的所述目标损失值。
- 根据权利要求1所述的方法,其中,所述特征编码模型包括编码网络,所述通过所述特征编码模型对多个所述样本音频的所述音频特征进行编码,得到多个所述样本音频的多个编码向量,包括:根据所述编码网络对多个所述样本音频的所述音频特征进行编码,得到多个所述样本音频的多个所述编码向量;所述编码网络包括残差网络或卷积网络;其中,训练好的所述特征编码模型的所述编码网络输出的编码向量能够作为该特征编码模型输出的音频的特征向量。
- 根据权利要求3所述的方法,其中,所述残差网络包括有IN层和BN层中的至少之一。
- 根据权利要求3所述的方法,其中,所述编码网络还包括GeM池化层,所述根据所述编码网络对多个所述样本音频的所述音频特征进行编码,得到多个所述样本音频的多个所述编码向量,包括:根据所述残差网络或所述卷积网络对多个所述样本音频的所述音频特征进行编码,得到多个所述样本音频的多个初始编码向量;根据所述GeM池化层对多个所述初始编码向量进行处理,得到多个所述样本音频的多个所述编码向量。
- 根据权利要求1-5中任一项所述的方法,其中,所述特征编码模型包括BN层和分类层,所述方法还包括:根据所述BN层对多个所述编码向量进行处理,得到正则化后的多个所述编码向量;所述根据多个所述编码向量对多个所述样本音频进行分类处理,得到多个所述样本音频的类别预测值,包括:根据所述分类层对所述正则化后的多个所述编码向量进行所述分类处理,得到多个所述样本音频的所述类别预测值;其中,训练好的所述特征编码模型的所述BN层输出的编码向量能够作为该特征编码模型输出的音频的特征向量。
- 一种音频确定方法,其包括:获取待查询音频;提取所述待查询音频的音频特征;根据训练好的特征编码模型对所述待查询音频的所述音频特征进行处理,得到所述待查询音频的第一特征向量;基于所述第一特征向量与参考特征库中的多个候选音频的第二特征向量之间的相似度,从所述参考特征库中确定与所述待查询音频属于同一音频的目标候选音频;多个所述候选音频的所述第二特征向量是通过训练好的所述特征编码模型预先确定的;其中,所述特征编码模型是根据权利要求1-6中任一项所述的特征编码模型生成方法得到的。
- 一种特征编码模型的训练装置,其包括:第一获取模块,被配置为获取标注有类别标签的多个样本音频;第一提取模块,被配置为提取多个所述样本音频的音频特征;编码分类模块,被配置为通过所述特征编码模型对多个所述样本音频的所述音频特征进行编码,得到多个所述样本音频的多个编码向量,以及根据多个所述编码向量对多个所述样本音频进行分类处理, 得到多个所述样本音频的类别预测值;第一确定模块,被配置为根据多个所述编码向量、多个所述样本音频的所述类别预测值以及多个所述样本音频的所述类别标签,确定目标损失函数的目标损失值,并基于所述目标损失值更新所述特征编码模型的参数,以减小属于同一类别的所述样本音频的所述编码向量之间的差异、增大属于不同类别的所述样本音频的所述编码向量之间的差异、以及减少多个所述样本音频的所述类别预测值和所述标签类别的差异,得到训练好的所述特征编码模型。
- 一种音频确定装置,其包括:第二获取模块,被配置为获取待查询音频;第二提取模块,被配置为提取所述待查询音频的音频特征;处理模块,被配置为根据训练好的特征编码模型对所述待查询音频进行处理,得到所述待查询音频的第一特征向量;第二确定模块,被配置为基于所述第一特征向量与参考特征库中的多个候选音频的第二特征向量之间的相似度,从所述参考特征库中确定与所述待查询音频属于同一音频的目标候选音频;多个所述候选音频的所述第二特征向量是通过训练好的所述特征编码模型预先确定的;其中,所述特征编码模型是根据权利要求1-6中任一项所述的特征编码模型生成方法得到的。
- 一种计算机可读介质,其上存储有计算机程序,其中,该程序被处理装置执行时实现权利要求1-7中任一项所述方法的步骤。
- 一种电子设备,其包括:存储装置,其上存储有至少一个计算机程序;至少一个处理装置,用于执行所述存储装置中的所述至少一个计算机程序,以实现权利要求1-7中任一项所述方法的步骤。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210045047.4A CN114510599A (zh) | 2022-01-14 | 2022-01-14 | 特征编码模型生成方法、音频确定方法以及相关装置 |
CN202210045047.4 | 2022-01-14 |
Publications (2)
Publication Number | Publication Date |
---|---|
WO2023134550A1 true WO2023134550A1 (zh) | 2023-07-20 |
WO2023134550A9 WO2023134550A9 (zh) | 2023-08-31 |
Family
ID=81550533
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2023/070800 WO2023134550A1 (zh) | 2022-01-14 | 2023-01-06 | 特征编码模型生成方法、音频确定方法以及相关装置 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN114510599A (zh) |
WO (1) | WO2023134550A1 (zh) |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114510599A (zh) * | 2022-01-14 | 2022-05-17 | 北京有竹居网络技术有限公司 | 特征编码模型生成方法、音频确定方法以及相关装置 |
CN115134338B (zh) * | 2022-05-20 | 2023-08-11 | 腾讯科技(深圳)有限公司 | 多媒体信息编码方法、对象检索方法及装置 |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111091835A (zh) * | 2019-12-10 | 2020-05-01 | 携程计算机技术(上海)有限公司 | 模型训练的方法、声纹识别的方法、***、设备及介质 |
CN113327621A (zh) * | 2021-06-09 | 2021-08-31 | 携程旅游信息技术(上海)有限公司 | 模型训练方法、用户识别方法、***、设备及介质 |
CN113593611A (zh) * | 2021-07-26 | 2021-11-02 | 平安科技(深圳)有限公司 | 语音分类网络训练方法、装置、计算设备及存储介质 |
CN113822428A (zh) * | 2021-08-06 | 2021-12-21 | 中国工商银行股份有限公司 | 神经网络训练方法及装置、图像分割方法 |
CN114510599A (zh) * | 2022-01-14 | 2022-05-17 | 北京有竹居网络技术有限公司 | 特征编码模型生成方法、音频确定方法以及相关装置 |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10110187B1 (en) * | 2017-06-26 | 2018-10-23 | Google Llc | Mixture model based soft-clipping detection |
CN113392868A (zh) * | 2021-01-14 | 2021-09-14 | 腾讯科技(深圳)有限公司 | 一种模型训练的方法、相关装置、设备及存储介质 |
-
2022
- 2022-01-14 CN CN202210045047.4A patent/CN114510599A/zh active Pending
-
2023
- 2023-01-06 WO PCT/CN2023/070800 patent/WO2023134550A1/zh unknown
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111091835A (zh) * | 2019-12-10 | 2020-05-01 | 携程计算机技术(上海)有限公司 | 模型训练的方法、声纹识别的方法、***、设备及介质 |
CN113327621A (zh) * | 2021-06-09 | 2021-08-31 | 携程旅游信息技术(上海)有限公司 | 模型训练方法、用户识别方法、***、设备及介质 |
CN113593611A (zh) * | 2021-07-26 | 2021-11-02 | 平安科技(深圳)有限公司 | 语音分类网络训练方法、装置、计算设备及存储介质 |
CN113822428A (zh) * | 2021-08-06 | 2021-12-21 | 中国工商银行股份有限公司 | 神经网络训练方法及装置、图像分割方法 |
CN114510599A (zh) * | 2022-01-14 | 2022-05-17 | 北京有竹居网络技术有限公司 | 特征编码模型生成方法、音频确定方法以及相关装置 |
Also Published As
Publication number | Publication date |
---|---|
WO2023134550A9 (zh) | 2023-08-31 |
CN114510599A (zh) | 2022-05-17 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2023134550A1 (zh) | 特征编码模型生成方法、音频确定方法以及相关装置 | |
WO2022105545A1 (zh) | 语音合成方法、装置、可读介质及电子设备 | |
US20180276540A1 (en) | Modeling of the latent embedding of music using deep neural network | |
WO2022121801A1 (zh) | 信息处理方法、装置和电子设备 | |
WO2022105553A1 (zh) | 语音合成方法、装置、可读介质及电子设备 | |
WO2019214289A1 (zh) | 图像处理方法、装置、电子设备及存储介质 | |
WO2022111242A1 (zh) | 旋律生成方法、装置、可读介质及电子设备 | |
WO2022247562A1 (zh) | 多模态数据检索方法、装置、介质及电子设备 | |
CN114443891B (zh) | 编码器的生成方法、指纹提取方法、介质及电子设备 | |
WO2023143016A1 (zh) | 特征提取模型的生成方法、图像特征提取方法和装置 | |
CN112153460B (zh) | 一种视频的配乐方法、装置、电子设备和存储介质 | |
WO2023273596A1 (zh) | 确定文本相关性的方法、装置、可读介质及电子设备 | |
CN111370019A (zh) | 声源分离方法及装置、神经网络的模型训练方法及装置 | |
WO2022166613A1 (zh) | 文本中角色的识别方法、装置、可读介质和电子设备 | |
WO2022156413A1 (zh) | 语音风格的迁移方法、装置、可读介质和电子设备 | |
CN111192601A (zh) | 音乐标注方法、装置、电子设备及介质 | |
WO2023142914A1 (zh) | 日期识别方法、装置、可读介质及电子设备 | |
CN111625649A (zh) | 文本处理方法、装置、电子设备及介质 | |
CN111428078B (zh) | 音频指纹编码方法、装置、计算机设备及存储介质 | |
CN111291715B (zh) | 基于多尺度卷积神经网络的车型识别方法、电子设备及存储介质 | |
WO2021012691A1 (zh) | 用于检索图像的方法和装置 | |
CN111898753B (zh) | 音乐转录模型的训练方法、音乐转录方法以及对应的装置 | |
WO2024001548A1 (zh) | 歌单生成方法、装置、电子设备及存储介质 | |
Sturm et al. | Formalizing the problem of music description | |
WO2023130925A1 (zh) | 字体识别方法、装置、可读介质及电子设备 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 23739894 Country of ref document: EP Kind code of ref document: A1 |