WO2022051856A1 - Method and system for training a neural network model using adversarial learning and knowledge distillation - Google Patents
Method and system for training a neural network model using adversarial learning and knowledge distillation Download PDFInfo
- Publication number
- WO2022051856A1 WO2022051856A1 PCT/CA2021/051249 CA2021051249W WO2022051856A1 WO 2022051856 A1 WO2022051856 A1 WO 2022051856A1 CA 2021051249 W CA2021051249 W CA 2021051249W WO 2022051856 A1 WO2022051856 A1 WO 2022051856A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- neural network
- network model
- data samples
- student
- training
- Prior art date
Links
- 238000003062 neural network model Methods 0.000 title claims abstract description 166
- 238000012549 training Methods 0.000 title claims abstract description 75
- 238000000034 method Methods 0.000 title claims abstract description 67
- 238000013140 knowledge distillation Methods 0.000 title claims abstract description 39
- 238000013528 artificial neural network Methods 0.000 claims abstract description 39
- 230000000873 masking effect Effects 0.000 claims abstract description 11
- 238000012545 processing Methods 0.000 claims description 39
- 230000006870 function Effects 0.000 claims description 21
- 238000003058 natural language processing Methods 0.000 claims description 15
- 238000009826 distribution Methods 0.000 claims description 4
- 238000005070 sampling Methods 0.000 claims description 3
- 230000015654 memory Effects 0.000 description 20
- 238000010801 machine learning Methods 0.000 description 14
- 230000008569 process Effects 0.000 description 13
- 238000010586 diagram Methods 0.000 description 8
- 239000013598 vector Substances 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 238000013135 deep learning Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000000926 separation method Methods 0.000 description 3
- 230000003190 augmentative effect Effects 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000001537 neural effect Effects 0.000 description 2
- 241000282326 Felis catus Species 0.000 description 1
- 230000000712 assembly Effects 0.000 description 1
- 238000000429 assembly Methods 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000008014 freezing Effects 0.000 description 1
- 238000007710 freezing Methods 0.000 description 1
- 238000012886 linear function Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 230000007935 neutral effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000003094 perturbing effect Effects 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000006403 short-term memory Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/094—Adversarial learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/088—Non-supervised learning, e.g. competitive learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Definitions
- the present application relates to methods and systems for training machine learning models, and, in particular, methods and systems for training a neural network model using adversarial learning and knowledge distillation.
- Deep learning based algorithms are machine learning methods used for many machine learning applications in natural language processing (NLP) and computer vision (CV) fields. Deep learning consists of composing layers of nonlinear parametric functions or "neurons” together and training the parameters or "weights", typically using gradient-based optimization algorithms, to minimize a loss function.
- NLP natural language processing
- CV computer vision
- One key reason of the success of these methods is the ability to improve performance with an increase in parameters and data.
- NLP this has led to deep learning architectures with billions of parameters (Brown et. al 2020). Research has shown that large architectures or "models" are easier to optimize as well. Model compression is thus imperative for any practical application such as deploying a trained machine learning model on a phone for a personal assistant.
- KD Knowledge distillation
- complex neural network model refers to a neural network model with a relatively high number of computing resources such as GPU/CPU power and computer memory space and/or those neural network models including a relatively high number of hidden layers.
- the complex neural network model for the purposes of KD, is sometimes referred to as a teacher neural network model (T) or a teacher for short.
- T teacher neural network model
- a typical drawback of the teacher is that it may require significant computing resources that may not be available in consumer electronic devices, such as mobile communication devices or edge computing devices.
- the teacher neural network model typically requires a significant amount of time to infer (i.e. predict) a particular output for an input due to the complexity of the teacher neural network model itself, and hence the teacher neural network model may not be suitable for deployment to a consumer computing device for use therein.
- KD techniques are applied to extract, or distill the learned parameters, or knowledge, of a teacher neural network model and impart such knowledge to a less sophisticated neural network model with faster inference time and reduced computing resource and memory space cost that may with less effort on consumer computing devices, such as edge devices.
- the less complex neural network model is often referred to as the student neural network model (S) or a student for short.
- the KD techniques involve training the student using not only the labeled training data samples of the training dataset but also using the outputs generated by the teacher neural network model, known as logits.
- H the cross-entropy loss function (other loss functions may also be used)
- o the softmax function
- parameter T is a temperature parameter
- a a hyperparamter that controls the amount of contribution from the cross entropy loss function and KD loss
- zt and zs are the logits (i.e. the output of the neural network before the last softmax layer) of the teacher neural network model (T) and student neural network model (S) respectively.
- KD techniques are widely used because they are agnostic to the architectures of the neural networks of the teacher and the student neural network models and require only access to the outputs generated by the teacher neural network model in order to train the student neural network model to effectively imitate the behavior of the teacher neural network model. Still, for many applications there is a significant gap between the performance of the teacher neural network model and the performance of the student neural network model and various KD techniques have been proposed to reduce this gap. For example, in the NLP field, KD techniques have been proposed whereby knowledge transfer can be effected by learning parameters of the teacher and student neural network models in stages and freezing the parameters, or by defining intermediate neural networks which learns from the teacher neural network model and teaches the student neural network model.
- Known KD techniques present a number of challenges.
- many known KD techniques have been demonstrated on a specific neural network model architecture, such as a transformer, in the scenario where the teacher and student neural network models use the same architecture.
- the training process used for many known KD techniques typically applies arbitrary design choices which work for one task but are difficult to generalize to other tasks.
- training of the teacher neural network model can be computationally expensive (i.e. requires significant processing and memory resources that consume power).
- many known KD techniques require access to the parameters (e.g., weights) of the teacher neural network model as well as the output of the teacher neural network model.
- the difference between the accuracy of the teacher neural network model and the accuracy of the student neural model can be large.
- the present disclosure describes a method of training a student neural network model using adversarial learning and knowledge distillation.
- the method includes training a generator to generate a respective adversarial data sample for each of a plurality of input data samples by replacing selected parts of the input data samples, the training of the generator being based on an objective of maximizing divergence between output predictions generated by the student neural network and a teacher neural network model for the adversarial data samples.
- the method further includes training the student neural network model based on objectives of: (i) minimizing divergence between output predictions generated by the student neural network model and the teacher neural network model for the adversarial data samples generated by the generator, and (ii) minimizing divergence between output predictions generated by the student neural network model and the teacher neural network model for the input data samples.
- the introduction of perturbed input data samples that are specifically generated to cause a divergence of teacher neural network and student neural network model output predictions can improve the training and accuracy of the student neural network model.
- training the generator comprises: for each of the input data samples, randomly selecting and masking the selected parts of the input data sample and generating, using a generator neural network model, the respective adversarial data sample with replacement data for the selected parts of the masked input data; obtaining output predictions for the respective adversarial data samples from the teacher neural network model and the student teacher neural network model; computing a generator loss using a generator loss function that is minimized when divergence between the output predictions generated by the student neural network and a teacher neural network model for the respective adversarial data samples is maximized; and updating parameters of the generator neural network model using gradient decent based on the computed loss.
- randomly selecting the selected parts of the input data sample is performed by randomly determining, based on a defined probability, for each part of a input data sample, if the part is to be selected as one of the selected parts.
- generating the replacement data for the selected parts comprises sampling a Gumbel softmax distribution of logits generated by the generator neural network model.
- the student neural network model is trained to perform a natural language processing task and each of the input data samples is a text data sample comprising a set of tokens that each correspond to a discrete text element of the text data sample, wherein the selected parts of the input data samples correspond to individual tokens.
- the student neural network model is trained to perform an image processing and each of the input data samples is an image data sample comprising a set of pixels, wherein the selected parts of the input data samples correspond to pixels.
- divergence between the output predictions generated by the student neural network and a teacher neural network model for the respective adversarial data samples corresponds to a Kullback-Leibler (KL) divergence.
- KL Kullback-Leibler
- the divergence between the output predictions generated by the student neural network and a teacher neural network model for the input data samples corresponds to a Kullback-Leibler (KL) divergence.
- KL Kullback-Leibler
- training the student neural network model is also based on an objective of minimizing divergence between output predictions generated by the student neural network for the input data samples and ground truth labels for the input data samples.
- divergence between the output predictions generated by the student neural network and the ground truth labels for the input data samples corresponds to a Cross-Entropy loss.
- a system for training a student neural network model comprises one or more processors and a non-transitory storage medium storing software instructions that, when executed by the one or more processors, configure the system to perform the method of one or more of the preceding aspects.
- a non-transitory computer readable medium storing software instructions.
- the software instructions when executed by the one or more processors, configure the one or more processors to perform the method of one or more of the preceding aspects.
- the present disclosure provides methods and systems for training a student neural network using adversarial learning and knowledge distillation.
- a generator is trained to generate adversarial data samples for respective input data samples by masking parts of the input data samples with an objective of maximizing a divergence between output predictions generated by the student neural network and a teacher neural network model for the adversarial data samples.
- the student neural network is trained based on objectives of (i) minimizing a divergence between output predictions generated by the student neural network and the teacher neural network model for the adversarial data samples, and (ii) minimizing a divergence between output predictions generated by the student neural network and the teacher neural network model for the input data samples.
- training the student neural network is also based on an objective of minimizing a divergence between output predictions generated by the student neural network for the input data samples and ground truth labels for the training data samples.
- FIG. 1 illustrates a schematic block diagram of a system for training a neural network model for a NLP task using adversarial learning and knowledge distillation in accordance with an embodiment of the present disclosure
- FIG. 2 is a flow diagram of a process for training a text generator of the system of FIG. 1;
- FIG. 3 is a flow diagram of a process for training a student neural network model of the system of FIG. 1;
- FIG. 4 illustrates pseudo code of a method of training a neural network model (i.e. a student neural network model) using adversarial learning and knowledge distillation in accordance with an embodiment of the present disclosure
- FIG. 5 illustrates a schematic block diagram of a system for training a neural network model for an image processing task using adversarial learning and knowledge distillation in accordance with an embodiment of the present disclosure
- FIG. 6 shows a block diagram of an example simplified processing system which may be used to implement embodiments disclosed herein.
- the present disclosure relates to a method and system for training a neural network model using adversarial learning and knowledge distillation in order to reduce a difference between the accuracy of a teacher neural network model and the accuracy of a student neural network model.
- a set of adversarial data samples are generated by perturbing a set of original training data samples. This can be performed by masking parts of the original training data samples and generating respective adversarial data samples in which the masked parts are replaced with generated content.
- a generator is trained to generate the adversarial data samples so that the output predictions of the teacher and student neural network models diverge for the adversarial data samples.
- the student neural network model is then trained using KD with both the original training data samples and the adversarial data samples.
- the method and system of the present disclosure can improve knowledge distillation between the teacher neural network model and the student neural network for both discrete data processing tasks, such as embedding vectors representative of text, and continuous data processing tasks, such as image data.
- FIG. 1 illustrates a schematic block diagram of a machine learning system 100 (hereinafter referred to as system 100) for training a neural network model for a NLP task using adversarial learning and knowledge distillation in accordance with an embodiment of the present disclosure.
- the system 100 includes a text generator 102 (illustrated as generator), a teacher neural network model 104 (illustrated as teacher), and a student neural network model 106 (illustrated as student).
- the teacher neural network model 104 is a large trained neural network model.
- the teacher neural network model 104 has been trained to perform an NLP task to generate an output prediction for an input text data sample (x), which in example embodiments can be a vector of tokens, with each token representing a discrete element within the input text data sample (x).
- the discrete elements that are used as tokens can be words, terms, sentences, symbols, or some other meaningful elements.
- each token can be considered a discrete part of input text data sample (x).
- the teacher neural network model 104 has been trained to map an input text data sample (x) to an output prediction (y) that is appropriate for the application. For example, in the case of sentiment analysis, an input text data sample (x) could be mapped to an output prediction (y) that is selected from the candidate prediction set of "Good”, “Bad”, and "Neutral".
- the student neural network model 106 is smaller than the teacher neural network model 104 (i.e., has fewer and/or compressed parameters, and/or fewer hidden layers, and/or requires fewer computations to generate a prediction). Student neural network model 106 is to be trained to imitate the behavior of the teacher neural network model for an NLP task.
- the text generator 102 is configured to receive input text data samples (x) of an original training dataset (X) in which each input text data sample (x) includes a vector of discrete parts, namely tokens.
- the text generator 102 is configured to generate adversarial text data samples (x') by randomly replacing tokens in the input text data samples (x) with generated tokens to provide an augmented training dataset (X U X') that is used for KD training of the student neural network model 106.
- the system 100 performs a method of the present disclosure that includes two stages. In a first stage, the text generator 102 is trained and in the second stage the student neural network model 106 is trained using knowledge distilled from the teacher neural network model 104.
- the first stage of the method of the present disclosure will now be described with reference to FIG. 1 and the flowchart of FIG. 2, which illustrates a process 200 for training the text generator 102 to generate adversarial data samples x' (also referred to as adversarial examples).
- training text data samples x from original training dataset X are provided as input text data samples x to text generator 102.
- each input text data sample x comprises a vector of tokens 108.
- each token 108 corresponds to an individual word.
- text generator 102 then randomly selects tokens from the vector of tokens 108 included in each text data sample x for masking.
- text generator 102 includes masking operation 109 that is configured to randomly determine for each token, based on a defined probability p (for example 30%), if the token should be masked or not.
- p for example 30%
- text generator 102 is configured to generate replacement tokens to replace the masked tokens included in masked text data samples x m and thereby generate respective adversarial text data samples x'.
- the text generator 102 includes a generator neural network model 110 that is configured to receive a masked text data sample x m as input and generate replacement tokens 112 for the masked tokens included in the masked text data sample x m .
- the text generator 102 generates the replacement tokens 112 for the masked tokens by sampling a Gumbel softmax distribution of the logits generated by generator neural network model 110. This can enable end-to-end differentiation to support backpropagation based training.
- the replacement tokens 112 are inserted at their respective locations into the masked text data sample x m (i.e. the masked version of the input data sample x ) to replace the masked tokens, thereby generating an adversarial data sample x'.
- the adversarial text data samples x' are provided to the teacher neural network model 104 and the student neural network model 106 to obtain their respective output predictions.
- the output predictions of the teacher neural network model 104 and the student neural network model 106 that are obtained are groups of logits that are generated by each of the teacher neural network model 104 and the student neural network model 106 for each of the adversarial text data samples x'.
- KL Kullback-Leibler
- the KL divergence between two discrete distributions P and Q over a probability space is defined as: P(x) DL(P
- (?) ) p (x) log(- ) 4- >xex Qw
- the parameters of the text generator 102 are then updated using gradient decent with the objective of maximizing the KL divergence in future training iterations.
- FIG. 3 illustrates a process 300 for KD training of the student neural network model 106 using original training dataset X and the adversarial training dataset X'.
- the adversarial loss LADV is the opposite of the generator loss L G and in this regard is a measure of the KL divergence between the output prediction (i.e. the inferred or predicted output) of the teacher neural network model 104 and the output prediction (i.e. the inferred or predicted output) of the student neural network model 106 on the adversarial text data sample x'.
- logits i.e., the output prediction of the final layer of a neural network before the softmax layer of the neural network
- logits generated by the teacher neural network model 104 and the student neural network model 106 are used as the respective predicted outputs for determining the adversarial loss.
- logits i.e., the output of the final layer of the neural network before the softmax layer of the neural network
- the teacher neural network model 104 and the student neural network model 106 are used as the respective output predictions for determining the KD loss.
- a Cross Entropy (CE) Loss Lcs (.y ⁇ ⁇ s M) is calculated based on a comparison of the true labels and output predictions from the student network model for the original training samples x.
- the Cross Entropy loss between the student neural network model output predictions can be defined as: where II is an indicator function, C is the number of output prediction classes, and P is the probability of the student neural network model and fi is the ith student output.
- teacher neural network output predictions may be used as the true labels for determining CE loss.
- the adversarial loss L ADV , KD loss L KD , and CE loss L CE are combined to generate a total loss Ltotai-
- the total loss can be an average of the three losses as indicated below, however in other examples the relative weighting of the losses can be adjusted as hyperparameters:
- the parameters of the student neural network model 106 are then updated using gradient decent with the objective of minimizing the total loss Ltotai-
- stage 1 process 200 and stage 2 process 300 can be iteratively repeated until predefined training criteria are reached.
- FIG. 4 illustrates an example of pseudo code 400 that corresponds to the stage 1 process 200 and stage 2 process 300.
- the method and system of the present disclosure can in some applications reduce a difference between the accuracy of a teacher neural network model 104 and the accuracy of a student neural network model 106. Further, the method and system of the present disclosure only require access to only the output predictions or logits of the teacher neural network model 104. Moreover, the teacher and student neural network models 104, 106 may have different architectures and the method and system of the present disclosure may be used for training teacher and student neural network models for any particular NLP task.
- FIG. 5 illustrates a schematic block diagram of a machine learning system 500 for training a neural network model for an image processing task using adversarial learning and knowledge distillation in accordance with another embodiment of the present disclosure.
- the image processing based machine learning system 500 is similar to the NPL processing based machine learning system 100 of FIG. 1 with the exception of differences that will be apparent from the following description and FIG. 5.
- the machine learning system 500 includes an image generator 502, a teacher neural network model 504, and a student neural network model 506.
- Teacher neural network 504 is trained to map input image data samples x to respective output predictions y for an image processing task, where the input data sample is an input image.
- teacher neural network 504 may be configured to assign a classification label to an input image data sample to identify the subject matter of the input image data sample x (e.g., image is of a "cat" or "dog").
- Image generator 502 receives, as inputs, image data samples x from an original training dataset X. Each image data sample is an array of pixels, each of which define a respective light intensity value as one or more variables. Whereas the text generator 102 of the NLP based machine learning system 100 of FIG. 1 is configured to perturb data samples by masking and replacing discrete elements or parts of a text data sample that correspond to tokens, the image generator 502 is configured to perturb data samples by masking and replacing discrete elements or parts of an image data sample that correspond to pixels 508. Image generator 502 incudes a mask operation 509 that randomly selects pixels 508 each image data sample x for masking and outputs a masked image data sample x m .
- image generator 502 is configured to randomly determine for each pixel 508, based on a defined probability p (for example 30%), if the pixel 508 should be masked or not.
- Image generator 502 is configured to generate replacement pixels 508' to replace the masked pixels 508 included in masked image data samples x m and thereby generate respective adversarial image data samples x'.
- the image generator 502 includes a neural network model 506 that is configured to receive a masked image data sample x m as input and generate replacement pixels 508' for the masked pixels 508. For each input image data sample x, the replacement pixels are inserted at their respective locations into the masked version (i.e., masked image data sample x m ) of the input image data sample x to output adversarial image data samples x'.
- the adversarial image data samples x' are provided to the teacher neural network model 504 and the student neural network model 506 to obtain their respective output predictions.
- a loss function L c -D KL (T(x') ⁇ S(x’')) is computed to minimize a negative KL objective, or in other words, to maximize the KL divergence between the teacher and student output predictions on the adversarial image data samples x'.
- the parameters of the image generator 502 are then updated using gradient decent with the objective of maximizing the KL divergence in future training iterations.
- the student neural network model 506 is then trained using original training dataset X and the adversarial training dataset X'.
- a Cross Entropy (CE) Loss L CE (y ⁇
- the output predictions output by teacher neural network 504 may be used as the true labels for determining CE loss.
- the adversarial loss L ADV , KD loss L KD , and CE loss L CE are combined to generate a total loss Ltotai-
- the parameters of the student neural network model 506 are then updated using gradient decent with the objective of minimizing the total loss Ltotai-
- the image generator training process and student neural network training process can be iteratively repeated until predefined training criteria are reached.
- the image generator 502 of the system of FIG. 5 is configured to mask pixels 508 of an image data sample and generate adversarial image data samples by maximizing a divergence between the teacher and student neural network models 504, 506.
- the student neural network model 506 is then trained using KD using an augmented training dataset comprising both the original images in the training dataset and the adversarial images generated by the image generator 502.
- the differences between the machine learning system 500 shown in FIG. 5 and the machine learning system 100 shown in FIG. 1 is that pixels of an image are masked rather than tokens and the image generator 502 does not use Gumbel-softmax method for replacement pixel generation.
- the image generator 502 may use any non-linear function for pixel generation because pixels are continuous variables and are end-to-end differentiable.
- sets of neighboring pixels may be grouped and used as discrete parts of the input data samples for masking, rather than just individual pixels.
- the methods and systems described above can be implemented on one or more computing devices that includes a processing unit (for example a CPU or GPU or special purpose Al processing unit) and persistent storage storing suitable instructions of the methods and systems that can be executed by the processing unit to configure the computing device to perform the functions described above.
- a processing unit for example a CPU or GPU or special purpose Al processing unit
- persistent storage storing suitable instructions of the methods and systems that can be executed by the processing unit to configure the computing device to perform the functions described above.
- FIG. 6 a block diagram of an example simplified processing system 1200, which may be used to implement embodiments disclosed herein, and provides a higher level implementation example.
- One or more of the teacher neural network model 104,504, the student neural network model 106, 506 and adversarial sample generator 102,502 as well as other functions included in the machine learning systems 100 and 500 may be implemented on the example processing system 1200, or variations of the processing system 1200.
- the processing system 1200 could be a terminal, for example, a desktop terminal, a tablet computer, a notebook computer, AR/VR, or an in-vehicle terminal, or may be a server, a cloud end, a smart phone or any suitable processing system.
- Other processing systems suitable for implementing embodiments described in the present disclosure may be used, which may include components different from those discussed below.
- Figure 12 shows a single instance of each component, there may be multiple instances of each component in the processing system 1200.
- the processing system 1200 may include one or more processing devices 1202, such as a graphics processing unit, a processor, a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a dedicated logic circuitry, accelerator, a tensor processing unit (TPU), a neural processing unit (NPU), or combinations thereof.
- the processing system 1200 may also include one or more input/output (I/O) interfaces 1204, which may enable interfacing with one or more appropriate input devices 1214 and/or output devices 1216.
- the processing system 1200 may include one or more network interfaces 1206 for wired or wireless communication with a network.
- the processing system 1200 may also include one or more storage units 1208, which may include a mass storage unit such as a solid state drive, a hard disk drive, a magnetic disk drive and/or an optical disk drive.
- the processing system 1200 may include one or more memories 1210, which may include volatile or non-volatile memory (e.g., a flash memory, a random access memory (RAM), and/or a read-only memory (ROM)).
- the non-transitory memory of memory 1210 may store instructions for execution by the processing device(s) 1202, such as to carry out examples of the methods and systems described in the present disclosure, for example, CKD/CKD* instructions and data 1212, such as the training datasets, for machine learning system 100 or 500.
- the memory(ies) 1210 may include other software instructions, such as for implementing an operating system for the processing system 1200 and other applications/functions.
- one or more data sets and/or modules may be provided by an external memory (e.g., an external drive in wired or wireless communication with the processing system 1200) or may be provided by a transitory or non-transitory computer-readable medium.
- Examples of non- transitory computer readable media include a RAM, a ROM, an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a CD-ROM, or other portable memory storage.
- the processing system 1200 may also include a bus 1218 providing communication among components of the processing system 1200, including the processing device(s) 1202, I/O interface(s) 1204, network interface(s) 1206, storage unit(s) 1208 and/or memory(ies) 1210.
- the bus 1218 may be any suitable bus architecture including, for example, a memory bus, a peripheral bus or a video bus.
- teacher neural network model 104,504 may be performed by any suitable processing device 1202 of the processing system 1200 or variant thereof.
- teacher neural network model 104,504, student neural network model 106, 506 and adversarial sample generator 102,502 may be use suitable neural network model, including variations such as recurrent neural network models, long short-term memory (LSTM) neural network models.
- LSTM long short-term memory
- any module, component, or device disclosed herein that executes instructions may include or otherwise have access to a non-transitory computer/processor readable storage medium or media for storage of information, such as computer/processor readable instructions, data structures, program modules, and/or other data.
- a non-transitory computer/processor readable storage medium includes magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, optical disks such as compact disc read-only memory (CD-ROM), digital video discs or digital versatile discs (i.e.
- Non-transitory computer/processor storage media may be part of a device or accessible or connectable thereto.
- Computer/processor readable/executable instructions to implement an application or module described herein may be stored or otherwise held by such non-transitory computer/processor readable storage media.
- the present disclosure may be described, at least in part, in terms of methods, a person of ordinary skill in the art will understand that the present disclosure is also directed to the various components for performing at least some of the aspects and features of the described methods, be it by way of hardware components, software or any combination of the two. Accordingly, the technical solution of the present disclosure may be embodied in the form of a software product.
- a suitable software product may be stored in a pre-recorded storage device or other similar non-volatile or non-transitory computer readable storage medium.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Molecular Biology (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
- Machine Translation (AREA)
Abstract
Method and system of training a student neural network using adversarial learning and knowledge distillation, including: training a generator to generate adversarial data samples for respective training data samples by masking parts of the training data samples with an objective of maximizing a divergence between output predictions generated by the student neural network and a teacher neural network model for the adversarial data samples; and training the student neural network based on objectives of (i) minimizing a divergence between output predictions generated by the student neural network and the teacher neural network model for the adversarial data samples, and (ii) minimizing a divergence between output predictions generated by the student neural network and the teacher neural network model for the training data samples.
Description
METHOD AND SYSTEM FOR TRAINING A NEURAL NETWORK MODEL USING ADVERSARIAL LEARNING AND KNOWLEDGE DISTILLATION
Related Application Data
[0001] The present application claims prior to, and the benefit of, provisional U.S. patent application no. 63/076,374, filed September 9, 2020, the content of which is incorporated herein by reference.
FIELD
[0002] The present application relates to methods and systems for training machine learning models, and, in particular, methods and systems for training a neural network model using adversarial learning and knowledge distillation.
BACKGROUND
[0003] Deep learning based algorithms are machine learning methods used for many machine learning applications in natural language processing (NLP) and computer vision (CV) fields. Deep learning consists of composing layers of nonlinear parametric functions or "neurons" together and training the parameters or "weights", typically using gradient-based optimization algorithms, to minimize a loss function. One key reason of the success of these methods is the ability to improve performance with an increase in parameters and data. In NLP this has led to deep learning architectures with billions of parameters (Brown et. al 2020). Research has shown that large architectures or "models" are easier to optimize as well. Model compression is thus imperative for any practical application such as deploying a trained machine learning model on a phone for a personal assistant.
[0004] Knowledge distillation (KD) is a neural network compression technique whereby the generalizations of a complex neural network model are transferred to a less complex neural network model that is able to make comparable inferences (i.e. predictions) as the complex model at less computing resource cost and time. Here, complex neural network model refers to a neural network model with a relatively high number of computing resources such as GPU/CPU power and computer memory space and/or those neural network
models including a relatively high number of hidden layers. The complex neural network model, for the purposes of KD, is sometimes referred to as a teacher neural network model (T) or a teacher for short. A typical drawback of the teacher is that it may require significant computing resources that may not be available in consumer electronic devices, such as mobile communication devices or edge computing devices. Furthermore, the teacher neural network model typically requires a significant amount of time to infer (i.e. predict) a particular output for an input due to the complexity of the teacher neural network model itself, and hence the teacher neural network model may not be suitable for deployment to a consumer computing device for use therein. Thus, KD techniques are applied to extract, or distill the learned parameters, or knowledge, of a teacher neural network model and impart such knowledge to a less sophisticated neural network model with faster inference time and reduced computing resource and memory space cost that may with less effort on consumer computing devices, such as edge devices. The less complex neural network model is often referred to as the student neural network model (S) or a student for short. The KD techniques involve training the student using not only the labeled training data samples of the training dataset but also using the outputs generated by the teacher neural network model, known as logits.
[0005] An example of a KD loss function used for training a student neural network model is as follows:
LKD = a * H(y, <r(zs; T = 1)) + (1 — a) * H(cr(zt; T = T), CT(ZS; T = T))(1) where H is the cross-entropy loss function (other loss functions may also be used), o is the softmax function, parameter T is a temperature parameter, a is a hyperparamter that controls the amount of contribution from the cross entropy loss function and KD loss, and zt and zs are the logits (i.e. the output of the neural network before the last softmax layer) of the teacher neural network model (T) and student neural network model (S) respectively.
[0006] KD techniques are widely used because they are agnostic to the architectures of the neural networks of the teacher and the student neural
network models and require only access to the outputs generated by the teacher neural network model in order to train the student neural network model to effectively imitate the behavior of the teacher neural network model. Still, for many applications there is a significant gap between the performance of the teacher neural network model and the performance of the student neural network model and various KD techniques have been proposed to reduce this gap. For example, in the NLP field, KD techniques have been proposed whereby knowledge transfer can be effected by learning parameters of the teacher and student neural network models in stages and freezing the parameters, or by defining intermediate neural networks which learns from the teacher neural network model and teaches the student neural network model.
[0007] Known KD techniques present a number of challenges. First, many known KD techniques have been demonstrated on a specific neural network model architecture, such as a transformer, in the scenario where the teacher and student neural network models use the same architecture. Further, the training process used for many known KD techniques typically applies arbitrary design choices which work for one task but are difficult to generalize to other tasks. Additionally, training of the teacher neural network model can be computationally expensive (i.e. requires significant processing and memory resources that consume power). Moreover, many known KD techniques require access to the parameters (e.g., weights) of the teacher neural network model as well as the output of the teacher neural network model. Finally, for many known KD techniques, the difference between the accuracy of the teacher neural network model and the accuracy of the student neural model, can be large.
[0008] Improvements in methods of training a neural network model using knowledge distillation to reduce a difference between the accuracy of the teacher model and the accuracy of the student model are desirable.
SUMMARY
[0009] According to example aspects, the present disclosure describes a method of training a student neural network model using adversarial learning and knowledge distillation. The method includes training a generator to generate
a respective adversarial data sample for each of a plurality of input data samples by replacing selected parts of the input data samples, the training of the generator being based on an objective of maximizing divergence between output predictions generated by the student neural network and a teacher neural network model for the adversarial data samples. The method further includes training the student neural network model based on objectives of: (i) minimizing divergence between output predictions generated by the student neural network model and the teacher neural network model for the adversarial data samples generated by the generator, and (ii) minimizing divergence between output predictions generated by the student neural network model and the teacher neural network model for the input data samples.
[OO1O] In at least some applications, the introduction of perturbed input data samples that are specifically generated to cause a divergence of teacher neural network and student neural network model output predictions can improve the training and accuracy of the student neural network model.
[0011] In some example aspects, training the generator comprises: for each of the input data samples, randomly selecting and masking the selected parts of the input data sample and generating, using a generator neural network model, the respective adversarial data sample with replacement data for the selected parts of the masked input data; obtaining output predictions for the respective adversarial data samples from the teacher neural network model and the student teacher neural network model; computing a generator loss using a generator loss function that is minimized when divergence between the output predictions generated by the student neural network and a teacher neural network model for the respective adversarial data samples is maximized; and updating parameters of the generator neural network model using gradient decent based on the computed loss.
[0012] In some examples of one or more of the preceding aspects, randomly selecting the selected parts of the input data sample is performed by randomly determining, based on a defined probability, for each part of a input data sample, if the part is to be selected as one of the selected parts.
[0013] In some examples of one or more of the preceding aspects, generating the replacement data for the selected parts comprises sampling a Gumbel softmax distribution of logits generated by the generator neural network model.
[0014] In some examples of one or more of the preceding aspects, the student neural network model is trained to perform a natural language processing task and each of the input data samples is a text data sample comprising a set of tokens that each correspond to a discrete text element of the text data sample, wherein the selected parts of the input data samples correspond to individual tokens.
[0015] In some examples of one or more of the preceding aspects, the student neural network model is trained to perform an image processing and each of the input data samples is an image data sample comprising a set of pixels, wherein the selected parts of the input data samples correspond to pixels.
[0016] In some examples of one or more of the preceding aspects, divergence between the output predictions generated by the student neural network and a teacher neural network model for the respective adversarial data samples corresponds to a Kullback-Leibler (KL) divergence.
[0017] In some examples of one or more of the preceding aspects, the divergence between the output predictions generated by the student neural network and a teacher neural network model for the input data samples corresponds to a Kullback-Leibler (KL) divergence.
[0018] In some examples of one or more of the preceding aspects, training the student neural network model is also based on an objective of minimizing divergence between output predictions generated by the student neural network for the input data samples and ground truth labels for the input data samples.
[0019] In some examples of one or more of the preceding aspects, divergence between the output predictions generated by the student neural
network and the ground truth labels for the input data samples corresponds to a Cross-Entropy loss.
[0020] According to a further example aspect, a system for training a student neural network model is disclosed. The system comprises one or more processors and a non-transitory storage medium storing software instructions that, when executed by the one or more processors, configure the system to perform the method of one or more of the preceding aspects.
[0021] According to a further example aspect, a non-transitory computer readable medium storing software instructions is disclosed. The software instructions, when executed by the one or more processors, configure the one or more processors to perform the method of one or more of the preceding aspects.
[0022] According to further example aspects, the present disclosure provides methods and systems for training a student neural network using adversarial learning and knowledge distillation. A generator is trained to generate adversarial data samples for respective input data samples by masking parts of the input data samples with an objective of maximizing a divergence between output predictions generated by the student neural network and a teacher neural network model for the adversarial data samples. The student neural network is trained based on objectives of (i) minimizing a divergence between output predictions generated by the student neural network and the teacher neural network model for the adversarial data samples, and (ii) minimizing a divergence between output predictions generated by the student neural network and the teacher neural network model for the input data samples.
[0023] In some examples, training the student neural network is also based on an objective of minimizing a divergence between output predictions generated by the student neural network for the input data samples and ground truth labels for the training data samples.
BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Reference will now be made, by way of example, to the accompanying drawings which show example embodiments of the present application, and in which:
[0025] FIG. 1 illustrates a schematic block diagram of a system for training a neural network model for a NLP task using adversarial learning and knowledge distillation in accordance with an embodiment of the present disclosure;
[0026] FIG. 2 is a flow diagram of a process for training a text generator of the system of FIG. 1;
[0027] FIG. 3 is a flow diagram of a process for training a student neural network model of the system of FIG. 1;
[0028] FIG. 4 illustrates pseudo code of a method of training a neural network model (i.e. a student neural network model) using adversarial learning and knowledge distillation in accordance with an embodiment of the present disclosure;
[0029] FIG. 5 illustrates a schematic block diagram of a system for training a neural network model for an image processing task using adversarial learning and knowledge distillation in accordance with an embodiment of the present disclosure; and
[0030] FIG. 6 shows a block diagram of an example simplified processing system which may be used to implement embodiments disclosed herein.
DESCRIPTION OF THE INVENTION
[0031] The present disclosure relates to a method and system for training a neural network model using adversarial learning and knowledge distillation in order to reduce a difference between the accuracy of a teacher neural network model and the accuracy of a student neural network model.
[0032] According to aspects of the disclosure, a set of adversarial data samples are generated by perturbing a set of original training data samples. This can be performed by masking parts of the original training data samples and
generating respective adversarial data samples in which the masked parts are replaced with generated content. A generator is trained to generate the adversarial data samples so that the output predictions of the teacher and student neural network models diverge for the adversarial data samples. The student neural network model is then trained using KD with both the original training data samples and the adversarial data samples. The method and system of the present disclosure can improve knowledge distillation between the teacher neural network model and the student neural network for both discrete data processing tasks, such as embedding vectors representative of text, and continuous data processing tasks, such as image data.
[0033] FIG. 1 illustrates a schematic block diagram of a machine learning system 100 (hereinafter referred to as system 100) for training a neural network model for a NLP task using adversarial learning and knowledge distillation in accordance with an embodiment of the present disclosure. The system 100 includes a text generator 102 (illustrated as generator), a teacher neural network model 104 (illustrated as teacher), and a student neural network model 106 (illustrated as student). The teacher neural network model 104 is a large trained neural network model. In an NLP example, the teacher neural network model 104 has been trained to perform an NLP task to generate an output prediction for an input text data sample (x), which in example embodiments can be a vector of tokens, with each token representing a discrete element within the input text data sample (x). For example, in various processing tasks, the discrete elements that are used as tokens can be words, terms, sentences, symbols, or some other meaningful elements. In this regard, each token can be considered a discrete part of input text data sample (x). Aspects of the present disclosure can be applied to a variety a different NLP tasks, including for example NLP tasks that support chatbot applications, search engine autocomplete applications, voice assistant applications, language translator applications, sentiment analysis applications, grammar check applications, and email classification and filtering applications, among other things. In all such cases, the teacher neural network model 104 has been trained to map an input text data sample (x) to an output prediction (y) that is appropriate for the application. For example, in the case of sentiment analysis, an input text data
sample (x) could be mapped to an output prediction (y) that is selected from the candidate prediction set of "Good", "Bad", and "Neutral".
[0034] The student neural network model 106 is smaller than the teacher neural network model 104 (i.e., has fewer and/or compressed parameters, and/or fewer hidden layers, and/or requires fewer computations to generate a prediction). Student neural network model 106 is to be trained to imitate the behavior of the teacher neural network model for an NLP task.
[0035] The text generator 102 is configured to receive input text data samples (x) of an original training dataset (X) in which each input text data sample (x) includes a vector of discrete parts, namely tokens. The text generator 102 is configured to generate adversarial text data samples (x') by randomly replacing tokens in the input text data samples (x) with generated tokens to provide an augmented training dataset (X U X') that is used for KD training of the student neural network model 106.
[0036] The system 100 performs a method of the present disclosure that includes two stages. In a first stage, the text generator 102 is trained and in the second stage the student neural network model 106 is trained using knowledge distilled from the teacher neural network model 104. The first stage of the method of the present disclosure will now be described with reference to FIG. 1 and the flowchart of FIG. 2, which illustrates a process 200 for training the text generator 102 to generate adversarial data samples x' (also referred to as adversarial examples).
[0037] As indicated in block 202, training text data samples x from original training dataset X are provided as input text data samples x to text generator 102. As illustrated in FIG. 1, each input text data sample x comprises a vector of tokens 108. In the illustrated example, each token 108 corresponds to an individual word. As indicated in block 204, text generator 102 then randomly selects tokens from the vector of tokens 108 included in each text data sample x for masking. In one example, text generator 102 includes masking operation 109 that is configured to randomly determine for each token, based on a defined probability p (for example 30%), if the token should be masked or not. By way
of example, as shown in FIG. 1 in the case of the input text data sample "The food was delicious and the price was not very expensive", the mask operation 109 of text generator 102 randomly masks the tokens "delicious" and "expensive", and outputs a masked text data sample xm.
[0038] As indicated in block 206, text generator 102 is configured to generate replacement tokens to replace the masked tokens included in masked text data samples xm and thereby generate respective adversarial text data samples x'. In this regard, the text generator 102 includes a generator neural network model 110 that is configured to receive a masked text data sample xm as input and generate replacement tokens 112 for the masked tokens included in the masked text data sample xm. In example embodiments, the text generator 102 generates the replacement tokens 112 for the masked tokens by sampling a Gumbel softmax distribution of the logits generated by generator neural network model 110. This can enable end-to-end differentiation to support backpropagation based training. For each input text data sample x, the replacement tokens 112 are inserted at their respective locations into the masked text data sample xm (i.e. the masked version of the input data sample x ) to replace the masked tokens, thereby generating an adversarial data sample x'.
[0039] As indicated at block 208, the adversarial text data samples x' are provided to the teacher neural network model 104 and the student neural network model 106 to obtain their respective output predictions. In example embodiments, the output predictions of the teacher neural network model 104 and the student neural network model 106 that are obtained are groups of logits that are generated by each of the teacher neural network model 104 and the student neural network model 106 for each of the adversarial text data samples x'. A generator loss is computed using a generator loss function Lc = -DKL(T(x')\\S(x'y) that is designed to minimize a negative Kullback-Leibler (KL) objective, or in other words, to maximize the KL divergence between the teacher and student logits (outputs) on the adversarial text data samples x'. The KL divergence between two discrete distributions P and Q over a probability space is defined as:
P(x) DL(P|(?) = ) p(x)log(- ) 4- >xex Qw
[0040] As indicated at block 210, the parameters of the text generator 102 (including neural network model 110) are then updated using gradient decent with the objective of maximizing the KL divergence in future training iterations.
[0041] The second stage of the method of the present disclosure will now be described with reference to FIG. 1 and the flowchart of FIG. 3, which illustrates a process 300 for KD training of the student neural network model 106 using original training dataset X and the adversarial training dataset X'.
[0042] As indicated at block 302, the adversarial text data samples x' are provided to the teacher neural network model 104 and the student neural network model 106 to obtain their respective output predictions, and an adversarial loss LADV = DKL(T(x')\\S(x’')) is computed. The adversarial loss LADV is the opposite of the generator loss LG and in this regard is a measure of the KL divergence between the output prediction (i.e. the inferred or predicted output) of the teacher neural network model 104 and the output prediction (i.e. the inferred or predicted output) of the student neural network model 106 on the adversarial text data sample x'. In example embodiments logits (i.e., the output prediction of the final layer of a neural network before the softmax layer of the neural network) generated by the teacher neural network model 104 and the student neural network model 106 are used as the respective predicted outputs for determining the adversarial loss.
[0043] As indicated at block 304, the original input text data samples x are provided to the teacher neural network model 104 and the student neural network model 106 to obtain their respective output predictions, and an KD loss LKD = DKL(T(.X) \ I'S’W) is computed that is based on the KL divergence between the output predictions of the teacher neural network model 104 and the output predictions of the student neural network model 106 on the input text data samples x. In example embodiments logits (i.e., the output of the final layer of the neural network before the softmax layer of the neural network) generated by the teacher neural network model 104 and the student neural network model 106 are used as the respective output predictions for determining the KD loss.
[0044] As indicated at block 306, in the case where true labels y (e.g., labels that are known ground truth labels) are known for at least some of the samples included in the original training dataset x, a Cross Entropy (CE) Loss Lcs (.y \ \sM) is calculated based on a comparison of the true labels and output predictions from the student network model for the original training samples x. The Cross Entropy loss between the student neural network model output predictions can be defined as:
where II is an indicator function, C is the number of output prediction classes, and P is the probability of the student neural network model and fi is the ith student output.
[0045] In some examples, teacher neural network output predictions may be used as the true labels for determining CE loss.
[0046] As indicated at block 308, the adversarial loss LADV, KD loss LKD , and CE loss LCE are combined to generate a total loss Ltotai- In some examples, the total loss can be an average of the three losses as indicated below, however in other examples the relative weighting of the losses can be adjusted as hyperparameters:
[0047] As indicated at block 810, the parameters of the student neural network model 106 are then updated using gradient decent with the objective of minimizing the total loss Ltotai-
[0048] The stage 1 process 200 and stage 2 process 300 can be iteratively repeated until predefined training criteria are reached.
[0049] FIG. 4 illustrates an example of pseudo code 400 that corresponds
to the stage 1 process 200 and stage 2 process 300.
[0050] Advantageously, the method and system of the present disclosure can in some applications reduce a difference between the accuracy of a teacher neural network model 104 and the accuracy of a student neural network model 106. Further, the method and system of the present disclosure only require access to only the output predictions or logits of the teacher neural network model 104. Moreover, the teacher and student neural network models 104, 106 may have different architectures and the method and system of the present disclosure may be used for training teacher and student neural network models for any particular NLP task.
[0051] FIG. 5 illustrates a schematic block diagram of a machine learning system 500 for training a neural network model for an image processing task using adversarial learning and knowledge distillation in accordance with another embodiment of the present disclosure. The image processing based machine learning system 500 is similar to the NPL processing based machine learning system 100 of FIG. 1 with the exception of differences that will be apparent from the following description and FIG. 5. The machine learning system 500 includes an image generator 502, a teacher neural network model 504, and a student neural network model 506.
[0052] Teacher neural network 504 is trained to map input image data samples x to respective output predictions y for an image processing task, where the input data sample is an input image. For example, teacher neural network 504 may be configured to assign a classification label to an input image data sample to identify the subject matter of the input image data sample x (e.g., image is of a "cat" or "dog").
[0053] Image generator 502 receives, as inputs, image data samples x from an original training dataset X. Each image data sample is an array of pixels, each of which define a respective light intensity value as one or more variables. Whereas the text generator 102 of the NLP based machine learning system 100 of FIG. 1 is configured to perturb data samples by masking and replacing discrete elements or parts of a text data sample that correspond to tokens, the
image generator 502 is configured to perturb data samples by masking and replacing discrete elements or parts of an image data sample that correspond to pixels 508. Image generator 502 incudes a mask operation 509 that randomly selects pixels 508 each image data sample x for masking and outputs a masked image data sample xm. In one example, image generator 502 is configured to randomly determine for each pixel 508, based on a defined probability p (for example 30%), if the pixel 508 should be masked or not. Image generator 502 is configured to generate replacement pixels 508' to replace the masked pixels 508 included in masked image data samples xm and thereby generate respective adversarial image data samples x'. In this regard, the image generator 502 includes a neural network model 506 that is configured to receive a masked image data sample xm as input and generate replacement pixels 508' for the masked pixels 508. For each input image data sample x, the replacement pixels are inserted at their respective locations into the masked version (i.e., masked image data sample xm) of the input image data sample x to output adversarial image data samples x'.
[0054] The adversarial image data samples x' are provided to the teacher neural network model 504 and the student neural network model 506 to obtain their respective output predictions. A loss function Lc = -DKL(T(x')\\S(x’')) is computed to minimize a negative KL objective, or in other words, to maximize the KL divergence between the teacher and student output predictions on the adversarial image data samples x'.
[0055] The parameters of the image generator 502 (including neural network model 510) are then updated using gradient decent with the objective of maximizing the KL divergence in future training iterations.
[0056] The student neural network model 506 is then trained using original training dataset X and the adversarial training dataset X'. The adversarial image data samples x' are provided to the teacher neural network model 504 and the student neural network model 506 to obtain their respective output predictions, and an adversarial loss LADV = DKL(T(x')\\S(x’')) is computed.
[0057] The original training image data samples x are provided to the
teacher neural network model 504 and the student neural network model 506 to obtain their respective output predictions, and an KD loss LKD = DKL(T(x) | |S(x)) is computed that is based on the KL divergence between the predicted output of the teacher neural network model 504 and the predicted output of the student neural network model 506 on the original image data samples x.
[0058] In the case where true labels y are known for at least some of the image samples included in the original training dataset x, a Cross Entropy (CE) Loss LCE(y\ |S(x)) is computed based on a comparison of the true labels and output predictions from the student network model 506 for the original training samples x.
[0059] In some examples, the output predictions output by teacher neural network 504 may be used as the true labels for determining CE loss.
[0060] The adversarial loss LADV, KD loss LKD, and CE loss LCE are combined to generate a total loss Ltotai- The parameters of the student neural network model 506 are then updated using gradient decent with the objective of minimizing the total loss Ltotai- The image generator training process and student neural network training process can be iteratively repeated until predefined training criteria are reached.
[0061] Accordingly, the image generator 502 of the system of FIG. 5 is configured to mask pixels 508 of an image data sample and generate adversarial image data samples by maximizing a divergence between the teacher and student neural network models 504, 506. The student neural network model 506 is then trained using KD using an augmented training dataset comprising both the original images in the training dataset and the adversarial images generated by the image generator 502. The differences between the machine learning system 500 shown in FIG. 5 and the machine learning system 100 shown in FIG. 1 is that pixels of an image are masked rather than tokens and the image generator 502 does not use Gumbel-softmax method for replacement pixel generation. Rather, the image generator 502 may use any non-linear function for pixel generation because pixels are continuous variables and are end-to-end differentiable.
[0062] In some examples, sets of neighboring pixels may be grouped and used as discrete parts of the input data samples for masking, rather than just individual pixels.
[0063] The methods and systems described above, including each of the teacher neural network model, the student neural network model, the generator, and the processes 200 and 300, can be implemented on one or more computing devices that includes a processing unit (for example a CPU or GPU or special purpose Al processing unit) and persistent storage storing suitable instructions of the methods and systems that can be executed by the processing unit to configure the computing device to perform the functions described above.
[0064] Referring to FIG. 6, a block diagram of an example simplified processing system 1200, which may be used to implement embodiments disclosed herein, and provides a higher level implementation example. One or more of the teacher neural network model 104,504, the student neural network model 106, 506 and adversarial sample generator 102,502 as well as other functions included in the machine learning systems 100 and 500 may be implemented on the example processing system 1200, or variations of the processing system 1200. The processing system 1200 could be a terminal, for example, a desktop terminal, a tablet computer, a notebook computer, AR/VR, or an in-vehicle terminal, or may be a server, a cloud end, a smart phone or any suitable processing system. Other processing systems suitable for implementing embodiments described in the present disclosure may be used, which may include components different from those discussed below. Although Figure 12 shows a single instance of each component, there may be multiple instances of each component in the processing system 1200.
[OOO1] The processing system 1200 may include one or more processing devices 1202, such as a graphics processing unit, a processor, a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a dedicated logic circuitry, accelerator, a tensor processing unit (TPU), a neural processing unit (NPU), or combinations thereof. The processing system 1200 may also include one or more input/output (I/O) interfaces 1204, which may enable interfacing with one or more appropriate input devices 1214 and/or
output devices 1216. The processing system 1200 may include one or more network interfaces 1206 for wired or wireless communication with a network.
[0002] The processing system 1200 may also include one or more storage units 1208, which may include a mass storage unit such as a solid state drive, a hard disk drive, a magnetic disk drive and/or an optical disk drive. The processing system 1200 may include one or more memories 1210, which may include volatile or non-volatile memory (e.g., a flash memory, a random access memory (RAM), and/or a read-only memory (ROM)). The non-transitory memory of memory 1210 may store instructions for execution by the processing device(s) 1202, such as to carry out examples of the methods and systems described in the present disclosure, for example, CKD/CKD* instructions and data 1212, such as the training datasets, for machine learning system 100 or 500. The memory(ies) 1210 may include other software instructions, such as for implementing an operating system for the processing system 1200 and other applications/functions. In some examples, one or more data sets and/or modules may be provided by an external memory (e.g., an external drive in wired or wireless communication with the processing system 1200) or may be provided by a transitory or non-transitory computer-readable medium. Examples of non- transitory computer readable media include a RAM, a ROM, an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a CD-ROM, or other portable memory storage.
[0003] The processing system 1200 may also include a bus 1218 providing communication among components of the processing system 1200, including the processing device(s) 1202, I/O interface(s) 1204, network interface(s) 1206, storage unit(s) 1208 and/or memory(ies) 1210. The bus 1218 may be any suitable bus architecture including, for example, a memory bus, a peripheral bus or a video bus.
[0004] The computations of the teacher neural network model 104,504, student neural network model 106, 506 and adversarial sample generator 102,502 may be performed by any suitable processing device 1202 of the processing system 1200 or variant thereof. Further, teacher neural network model 104,504, student neural network model 106, 506 and adversarial sample generator 102,502 may be use suitable neural network model, including
variations such as recurrent neural network models, long short-term memory (LSTM) neural network models.
[0065] The present disclosure has been made with reference to the accompanying drawings, in which embodiments of technical solutions are shown. However, many different embodiments may be used, and thus the description should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete. Wherever possible, the same reference numbers are used in the drawings and the following description to refer to the same elements, and prime notation is used to indicate similar elements, operations or steps in alternative embodiments. Separate boxes or illustrated separation of functional elements of illustrated systems and devices does not necessarily require physical separation of such functions, as communication between such elements may occur by way of messaging, function calls, shared memory space, and so on, without any such physical separation. As such, functions need not be implemented in physically or logically separated platforms, although they are illustrated separately for ease of explanation herein. Different devices may have different designs, such that although some devices implement some functions in fixed function hardware, other devices may implement such functions in a programmable processor with code obtained from a machine-readable storage medium. Lastly, elements referred to in the singular may be plural and vice versa, except where indicated otherwise either explicitly or inherently by context.
[0066] The embodiments set forth herein represent information sufficient to practice the claimed subject matter and illustrate ways of practicing such subject matter. Upon reading the following description in light of the accompanying figures, those of skill in the art will understand the concepts of the claimed subject matter and will recognize applications of these concepts not particularly addressed herein. It should be understood that these concepts and applications fall within the scope of the disclosure and the accompanying claims.
[0067] Moreover, it will be appreciated that any module, component, or device disclosed herein that executes instructions may include or otherwise have access to a non-transitory computer/processor readable storage medium or
media for storage of information, such as computer/processor readable instructions, data structures, program modules, and/or other data. A non- exhaustive list of examples of non-transitory computer/processor readable storage media includes magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, optical disks such as compact disc read-only memory (CD-ROM), digital video discs or digital versatile discs (i.e. DVDs), Blu-ray Disc™, or other optical storage, volatile and non-volatile, removable and non-removable media implemented in any method or technology, random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology. Any such non-transitory computer/processor storage media may be part of a device or accessible or connectable thereto. Computer/processor readable/executable instructions to implement an application or module described herein may be stored or otherwise held by such non-transitory computer/processor readable storage media. Although the present disclosure may describe methods and processes with steps in a certain order, one or more steps of the methods and processes may be omitted or altered as appropriate. One or more steps may take place in an order other than that in which they are described, as appropriate.
[0068] Although the present disclosure may be described, at least in part, in terms of methods, a person of ordinary skill in the art will understand that the present disclosure is also directed to the various components for performing at least some of the aspects and features of the described methods, be it by way of hardware components, software or any combination of the two. Accordingly, the technical solution of the present disclosure may be embodied in the form of a software product. A suitable software product may be stored in a pre-recorded storage device or other similar non-volatile or non-transitory computer readable storage medium.
[0069] The present disclosure may be embodied in other specific forms without departing from the subject matter of the claims. The described example embodiments are to be considered in all respects as being only illustrative and not restrictive. Selected features from one or more of the above-described embodiments may be combined to create alternative embodiments not explicitly
described, features suitable for such combinations being understood within the scope of this disclosure.
[0070] All values and sub-ranges within disclosed ranges are also disclosed. Also, although the systems, devices and processes disclosed and shown herein may comprise a specific number of elements/components, the systems, devices and assemblies could be modified to include additional or fewer of such elements/components. For example, although any of the elements/components disclosed may be referenced as being singular, the embodiments disclosed herein could be modified to include a plurality of such elements/components. The subject matter described herein intends to cover and embrace all suitable changes in technology.
Claims
1. A method of training a student neural network model using adversarial learning and knowledge distillation comprising: training a generator to generate a respective adversarial data sample for each of a plurality of input data samples by replacing selected parts of the input data samples, the training of the generator being based on an objective of maximizing divergence between output predictions generated by the student neural network and a teacher neural network model for the adversarial data samples; and training the student neural network model based on objectives of: (i) minimizing divergence between output predictions generated by the student neural network model and the teacher neural network model for the adversarial data samples generated by the generator, and (ii) minimizing divergence between output predictions generated by the student neural network model and the teacher neural network model for the input data samples.
2. The method of claim 1 wherein training the generator comprises: for each of the input data samples, randomly selecting and masking the selected parts of the input data sample and generating, using a generator neural network model, the respective adversarial data sample with replacement data for the selected parts of the masked input data; obtaining output predictions for the respective adversarial data samples from the teacher neural network model and the student teacher neural network model; computing a generator loss using a generator loss function that is minimized when divergence between the output predictions generated by the student neural network and a teacher neural network model for the respective adversarial data samples is maximized; updating parameters of the generator neural network model using gradient decent based on the computed loss.
3. The method of claim 2 wherein randomly selecting the selected parts of the input data sample is performed by randomly determining, based on a defined probability, for each part of a input data sample, if the part is to be selected as one of the selected parts.
4. The method of claim 2 or 3 wherein generating the replacement data for the selected parts comprises sampling a Gumbel softmax distribution of logits generated by the generator neural network model.
5. The method of claim 4 wherein the student neural network model is trained to perform a natural language processing task and each of the input data samples is a text data sample comprising a set of tokens that each correspond to a discrete text element of the text data sample, wherein the selected parts of the input data samples correspond to individual tokens.
6. The method of any one of claims 1 to 3 wherein the student neural network model is trained to perform an image processing and each of the input data samples is an image data sample comprising a set of pixels, wherein the selected parts of the input data samples correspond to pixels.
7. The method of any one of claims 1 to 6 wherein divergence between the output predictions generated by the student neural network and a teacher neural network model for the respective adversarial data samples corresponds to a Kullback-Leibler (KL) divergence.
8. The method of any one of claims 1 to 7 wherein the divergence between the output predictions generated by the student neural network and a teacher neural network model for the input data samples corresponds to a Kullback- Leibler (KL) divergence.
9. The method of any one of claims 1 to 8 wherein training the student neural network model is also based on an objective of minimizing divergence between output predictions generated by the student neural network for the input data samples and ground truth labels for the input data samples.
10. The method of claim 9 the divergence between the output predictions generated by the student neural network and the ground truth labels for the input data samples corresponds to a Cross-Entropy loss.
11. A system for training a student neural network model, comprising one or more processors and a non-transitory storage medium storing software instructions that, when executed by the one or more processors, configure the system to perform the method of any one of claims 1 to 10.
12. A non-transitory computer readable medium storing software instructions that, when executed by the one or more processors, configure the one or more processors to perform the method of any one of claims 1 to 10.
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP21865432.5A EP4200763A4 (en) | 2020-09-09 | 2021-09-09 | Method and system for training a neural network model using adversarial learning and knowledge distillation |
CN202180054867.3A CN116171446A (en) | 2020-09-09 | 2021-09-09 | Method and system for training neural network model through countermeasure learning and knowledge distillation |
US18/119,211 US20230222353A1 (en) | 2020-09-09 | 2023-03-08 | Method and system for training a neural network model using adversarial learning and knowledge distillation |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US202063076374P | 2020-09-09 | 2020-09-09 | |
US63/076,374 | 2020-09-09 |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US18/119,211 Continuation US20230222353A1 (en) | 2020-09-09 | 2023-03-08 | Method and system for training a neural network model using adversarial learning and knowledge distillation |
Publications (2)
Publication Number | Publication Date |
---|---|
WO2022051856A1 true WO2022051856A1 (en) | 2022-03-17 |
WO2022051856A9 WO2022051856A9 (en) | 2022-06-16 |
Family
ID=80629703
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CA2021/051249 WO2022051856A1 (en) | 2020-09-09 | 2021-09-09 | Method and system for training a neural network model using adversarial learning and knowledge distillation |
Country Status (4)
Country | Link |
---|---|
US (1) | US20230222353A1 (en) |
EP (1) | EP4200763A4 (en) |
CN (1) | CN116171446A (en) |
WO (1) | WO2022051856A1 (en) |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114972904A (en) * | 2022-04-18 | 2022-08-30 | 北京理工大学 | Zero sample knowledge distillation method and system based on triple loss resistance |
CN114972839A (en) * | 2022-03-30 | 2022-08-30 | 天津大学 | Generalized continuous classification method based on online contrast distillation network |
CN115085805A (en) * | 2022-06-09 | 2022-09-20 | 南京信息工程大学 | Few-mode multi-core fiber optical performance monitoring method, system and device based on anti-distillation model and storage medium |
CN115223049A (en) * | 2022-09-20 | 2022-10-21 | 山东大学 | Knowledge distillation and quantification technology for large model compression of electric power scene edge calculation |
US11574168B1 (en) | 2021-10-20 | 2023-02-07 | Moffett International Co., Limited | System and method for pivot-sample-based generator training |
CN116091849A (en) * | 2023-04-11 | 2023-05-09 | 山东建筑大学 | Tire pattern classification method, system, medium and equipment based on grouping decoder |
CN116416212A (en) * | 2023-02-03 | 2023-07-11 | 中国公路工程咨询集团有限公司 | Training method of road surface damage detection neural network and road surface damage detection neural network |
Families Citing this family (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11977626B2 (en) * | 2021-03-09 | 2024-05-07 | Nec Corporation | Securing machine learning models against adversarial samples through backdoor misclassification |
US20230186903A1 (en) * | 2021-12-13 | 2023-06-15 | International Business Machines Corporation | Reducing Exposure Bias in Machine Learning Training of Sequence-to-Sequence Transducers |
CN116704196B (en) * | 2023-08-03 | 2023-11-10 | 珠海数字动力科技股份有限公司 | Method for training image semantic segmentation model |
CN117765484B (en) * | 2023-12-04 | 2024-06-11 | 淮阴工学院 | Intelligent vehicle line pressing detection method and device based on countermeasure distillation and real-time target detection |
CN117892841B (en) * | 2024-03-14 | 2024-05-31 | 青岛理工大学 | Self-distillation method and system based on progressive association learning |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20190244103A1 (en) * | 2018-02-07 | 2019-08-08 | Royal Bank Of Canada | Robust pruned neural networks via adversarial training |
US20190325269A1 (en) * | 2018-04-20 | 2019-10-24 | XNOR.ai, Inc. | Image Classification through Label Progression |
-
2021
- 2021-09-09 CN CN202180054867.3A patent/CN116171446A/en active Pending
- 2021-09-09 WO PCT/CA2021/051249 patent/WO2022051856A1/en unknown
- 2021-09-09 EP EP21865432.5A patent/EP4200763A4/en active Pending
-
2023
- 2023-03-08 US US18/119,211 patent/US20230222353A1/en active Pending
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20190244103A1 (en) * | 2018-02-07 | 2019-08-08 | Royal Bank Of Canada | Robust pruned neural networks via adversarial training |
US20190325269A1 (en) * | 2018-04-20 | 2019-10-24 | XNOR.ai, Inc. | Image Classification through Label Progression |
Non-Patent Citations (5)
Title |
---|
HAIDAR AKMAL, REZAGHOLIZADEH MEHDI: "TextKD-GAN: Text Generation Using Knowledge Distillation and Generative Adversarial Networks ", PATTERN RECOGNITION : 5TH ASIAN CONFERENCE, ACPR 2019, AUCKLAND, NEW ZEALAND, NOVEMBER 26–29, 2019, REVISED SELECTED PAPERS, PART II, SPRINGER INTERNATIONAL PUBLISHING, CHAM, 23 April 2019 (2019-04-23), Cham, pages 1 - 12, XP055915056, ISSN: 0302-9743, ISBN: 978-3-030-41298-2, DOI: 10.1007/978-3-030-18305-9_9 * |
JANG ERIC, GU SHIXIANG, POOLE BEN: "Categorical Reparameterization with Gumbel-Softmax", ARXIV.ORG, 5 August 2017 (2017-08-05), pages 1 - 13, XP055848996, Retrieved from the Internet <URL:https://arxiv.org/pdf/1611.01144v5.pdf> [retrieved on 20211007] * |
MATT J. KUSNER; JOS\'E MIGUEL HERN\'ANDEZ-LOBATO: "GANS for Sequences of Discrete Elements with the Gumbel-softmax Distribution", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 12 November 2016 (2016-11-12), 201 Olin Library Cornell University Ithaca, NY 14853 , XP080731322 * |
See also references of EP4200763A4 * |
WILLIAM FEDUS; IAN GOODFELLOW; ANDREW M. DAI: "MaskGAN: Better Text Generation via Filling in the______", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 23 January 2018 (2018-01-23), 201 Olin Library Cornell University Ithaca, NY 14853 , XP081222630 * |
Cited By (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11574168B1 (en) | 2021-10-20 | 2023-02-07 | Moffett International Co., Limited | System and method for pivot-sample-based generator training |
US11599794B1 (en) * | 2021-10-20 | 2023-03-07 | Moffett International Co., Limited | System and method for training sample generator with few-shot learning |
CN114972839A (en) * | 2022-03-30 | 2022-08-30 | 天津大学 | Generalized continuous classification method based on online contrast distillation network |
CN114972904A (en) * | 2022-04-18 | 2022-08-30 | 北京理工大学 | Zero sample knowledge distillation method and system based on triple loss resistance |
CN114972904B (en) * | 2022-04-18 | 2024-05-31 | 北京理工大学 | Zero sample knowledge distillation method and system based on fighting against triplet loss |
CN115085805A (en) * | 2022-06-09 | 2022-09-20 | 南京信息工程大学 | Few-mode multi-core fiber optical performance monitoring method, system and device based on anti-distillation model and storage medium |
CN115085805B (en) * | 2022-06-09 | 2024-03-19 | 南京信息工程大学 | Fiber optical performance monitoring method and system based on anti-distillation model |
CN115223049A (en) * | 2022-09-20 | 2022-10-21 | 山东大学 | Knowledge distillation and quantification technology for large model compression of electric power scene edge calculation |
CN115223049B (en) * | 2022-09-20 | 2022-12-13 | 山东大学 | Knowledge distillation and quantification method for large model compression of electric power scene edge calculation |
CN116416212A (en) * | 2023-02-03 | 2023-07-11 | 中国公路工程咨询集团有限公司 | Training method of road surface damage detection neural network and road surface damage detection neural network |
CN116416212B (en) * | 2023-02-03 | 2023-12-08 | 中国公路工程咨询集团有限公司 | Training method of road surface damage detection neural network and road surface damage detection neural network |
CN116091849A (en) * | 2023-04-11 | 2023-05-09 | 山东建筑大学 | Tire pattern classification method, system, medium and equipment based on grouping decoder |
Also Published As
Publication number | Publication date |
---|---|
WO2022051856A9 (en) | 2022-06-16 |
CN116171446A (en) | 2023-05-26 |
EP4200763A4 (en) | 2024-02-28 |
EP4200763A1 (en) | 2023-06-28 |
US20230222353A1 (en) | 2023-07-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20230222353A1 (en) | Method and system for training a neural network model using adversarial learning and knowledge distillation | |
US11631029B2 (en) | Generating combined feature embedding for minority class upsampling in training machine learning models with imbalanced samples | |
CN110163234B (en) | Model training method and device and storage medium | |
WO2022052997A1 (en) | Method and system for training neural network model using knowledge distillation | |
CN113837370B (en) | Method and apparatus for training a model based on contrast learning | |
US10217017B2 (en) | Systems and methods for containerizing multilayer image segmentation | |
EP4200762A1 (en) | Method and system for training a neural network model using gradual knowledge distillation | |
CN111882426B (en) | Method, device, equipment and storage medium for training business risk classifier | |
US20220004849A1 (en) | Image processing neural networks with dynamic filter activation | |
CN111709415B (en) | Target detection method, device, computer equipment and storage medium | |
CN113065459A (en) | Video instance segmentation method and system based on dynamic condition convolution | |
CN112819024A (en) | Model processing method, user data processing method and device and computer equipment | |
Dinov et al. | Black box machine-learning methods: Neural networks and support vector machines | |
CN114299304A (en) | Image processing method and related equipment | |
CN111198949A (en) | Text label determination method and system | |
US20220335274A1 (en) | Multi-stage computationally efficient neural network inference | |
Boursinos et al. | Improving prediction confidence in learning-enabled autonomous systems | |
Jain et al. | Flynet–neural network model for automatic building detection from satellite images | |
CN112507912A (en) | Method and device for identifying illegal picture | |
Burciu et al. | Sensing forest for pattern recognition | |
Fuengfusin et al. | Mixed precision weight networks: Training neural networks with varied precision weights | |
Ortiz et al. | Job offers classifier using neural networks and oversampling methods | |
CN117808095B (en) | Method and device for generating attack-resistant sample and electronic equipment | |
US20230342672A1 (en) | Method and system for classification and/or prediction on unbalanced datasets | |
Himthani et al. | Ordered Ensemble Classifier Chain for Image and Emotion Classification |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21865432 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2021865432 Country of ref document: EP Effective date: 20230323 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |