CN109858506A - A kind of visualized algorithm towards convolutional neural networks classification results - Google Patents
A kind of visualized algorithm towards convolutional neural networks classification results Download PDFInfo
- Publication number
- CN109858506A CN109858506A CN201810519569.7A CN201810519569A CN109858506A CN 109858506 A CN109858506 A CN 109858506A CN 201810519569 A CN201810519569 A CN 201810519569A CN 109858506 A CN109858506 A CN 109858506A
- Authority
- CN
- China
- Prior art keywords
- layer
- neuron
- correlation
- output
- layers
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000004422 calculation algorithm Methods 0.000 title claims abstract description 89
- 238000013527 convolutional neural network Methods 0.000 title claims abstract description 57
- 210000002569 neuron Anatomy 0.000 claims abstract description 114
- 230000004913 activation Effects 0.000 claims abstract description 39
- 238000013507 mapping Methods 0.000 claims abstract description 37
- 238000000034 method Methods 0.000 claims abstract description 29
- 238000004364 calculation method Methods 0.000 claims abstract description 11
- 230000001537 neural effect Effects 0.000 claims description 37
- 238000006243 chemical reaction Methods 0.000 claims description 7
- 239000011159 matrix material Substances 0.000 claims description 6
- 238000012549 training Methods 0.000 claims description 4
- 230000005540 biological transmission Effects 0.000 claims description 3
- 230000006854 communication Effects 0.000 claims description 3
- 238000012546 transfer Methods 0.000 claims description 3
- 235000013399 edible fruits Nutrition 0.000 claims description 2
- 230000008555 neuronal activation Effects 0.000 claims description 2
- 239000004744 fabric Substances 0.000 claims 1
- 230000004438 eyesight Effects 0.000 abstract description 2
- 238000012545 processing Methods 0.000 abstract description 2
- 230000000007 visual effect Effects 0.000 abstract description 2
- 230000007423 decrease Effects 0.000 description 14
- 238000013528 artificial neural network Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 238000002474 experimental method Methods 0.000 description 4
- 210000005036 nerve Anatomy 0.000 description 4
- 230000000644 propagated effect Effects 0.000 description 3
- 238000010206 sensitivity analysis Methods 0.000 description 3
- 241000282376 Panthera tigris Species 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 241000282326 Felis catus Species 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000003475 lamination Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000004451 qualitative analysis Methods 0.000 description 1
- 238000004445 quantitative analysis Methods 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Landscapes
- Image Analysis (AREA)
Abstract
The invention discloses a kind of visualized algorithms towards convolutional neural networks classification results, belong to computer vision and digital image processing techniques field.The present invention uses relevance propagation algorithm in full articulamentum, obtains each neuron in the last layer convolutional layer and goes out the Class Activation mapping graph of the convolutional neural networks according to its contribution calculation to the size of the contribution of final output.After obtaining Class Activation mapping graph, also it just obtains in the last layer convolutional layer to the position of the contributive neuron of classification results, according to the propagation algorithm based on location information of proposition, the neuron position of classification will be supported successively to redirect forward in convolutional layer, until input layer, to obtain in input picture to the output contributive location of pixels set of result, finally obtain the visual image that can explain convolutional neural networks classification foundation, the present invention has higher accuracy in terms of the classification for explaining convolutional neural networks, and the feature between classification can be distinguished in interpretive classification decision.
Description
Technical field
The invention belongs to computer visions and digital image processing techniques field, and in particular to one kind is towards convolutional Neural net
The visualized algorithm of network classification results.
Background technique
Currently, convolutional neural networks have been widely applied to many fields as the most commonly used deep learning model,
Such as image classification, speech recognition and natural language processing etc., and all achieve good effect, certain fields
Reach or even surmounted the performance of the mankind.As it is using more and more extensive, the understanding of network model is become more and more important,
More particularly to when the fields such as automatic driving and medical diagnosis, understanding and verifying to Decision of Neural Network are very heavy
The ring wanted, people are not intended to help all decisions of do-it-yourself by a uncomprehending black box model, thus to convolutional Neural
The research that network class result explains has very important significance.
Explain that convolutional neural networks classification results visualized algorithm mainly has Class Activation mapping algorithm and sensitivity analysis
Algorithm.The former is only used in the full convolutional neural networks without containing full articulamentum, the advantage is that the product that can stop reading with Exact Solutions
The classification foundation of neural network；And the latter's usability is wider, can be used in most of convolutional neural networks, but solving
When releasing classification foundation, for that can not be applicable in well containing the case where input for being more than a kind of target, accuracy need to be mentioned
It is high.
General structure convolutional neural networks can not be applied to for above-mentioned Class Activation mapping algorithm and sensitivity analysis is calculated
The accuracy deficiency both of these problems of method interpretation classification results are calculated using different propagation respectively in full articulamentum and convolutional layer
Method proposes the Class Activation mapping graph visualized algorithm based on correlation.This method is passed in full articulamentum using correlation first
Algorithm is broadcast, obtains each neuron in the last layer convolutional layer to the size of the contribution of final output, then according to its tribute
It offers advice and calculates the Class Activation mapping graph of the convolutional neural networks.After obtaining Class Activation mapping graph, last is also just obtained
It will according to the propagation algorithm based on location information of proposition to the position of the contributive neuron of classification results in layer convolutional layer
The neuron position of classification is supported successively to redirect forward in convolutional layer, until input layer, to obtain in input picture to defeated
The contributive location of pixels set of result out, finally obtains the visual image that can explain convolutional neural networks classification foundation.
Summary of the invention
The purpose of the present invention is to provide a kind of solution prior arts to pay no attention in the effect for explaining convolutional neural networks classification
The visualized algorithm towards convolutional neural networks classification results thought.
The object of the present invention is achieved like this:
The invention discloses a kind of visualized algorithms towards convolutional neural networks classification results, especially by following steps
It realizes:
(1) data set for extracting input picture, is trained convolutional neural networks using data set as training set, obtains
Trained model parameter；
(2) calculation method according to Rel-CAM algorithm in full articulamentum is successively counted using output result and model parameter
Each neural unit in full articulamentum is calculated to the contribution of output, until convolutional layer；
(3) according to contribution of all neural units to output in the last layer convolutional layer obtained in step (2), calculating should
The respective weights in each channel of layer and output result, to obtain the Class Activation mapping graph of the network model；
(4) neural unit that is positive of record Class Activation mapping graph intermediate value, the position of the neural unit as in the layer to defeated
These neural units are added to the layer in the contributive neuronal ensemble of output by the position of the contributive pixel of result out；
(5) each neuron in set is successively taken out, all neurons and corresponding are calculated in its preceding layer in receptive field
The Hadamard product of weight, and sum to the Hadamard in each channel product, take it to neutralize maximum channel as contribution channel, and take
The neuron that its intermediate value is positive is added to the layer to exporting in contributive neuronal ensemble, and removes wherein duplicate nerve
Member；
(6) communication process of step (5) is repeated, until the neuronal ensemble in input layer is obtained, the nerve in the set
Unit shows the pixel under the position and contributes to output result.
For a kind of visualized algorithm towards convolutional neural networks classification results, the step (2) passes through following step
It is rapid to realize:
(2.1) assume that a trained CNN model and a given input picture, model have been divided into c class,
Assuming that C node is such output node in output layer, and S is scored at the nodec, Softmax layers are selected in algorithm
Output before is mapped to the position of only feature relevant to c class as class score, output,
Wherein,Indicate the correlation that c class neuron is predicted as in output layer, i.e. prediction result correlation is on output layer
Distribution；
(2.2) assume that output layer preceding layer is l, then contribution degree of this layer of each neural unit to last output,
It is exactly each neuron with prediction result correlation is defined as:
Wherein,Indicate the activation value of i-th of neuron in l layers, andIndicate the neural unit and next layer, i.e., it is defeated
Weight connection between layer neuron out；
(2.3) since the last layer only has c class output node to have correlation, only consider each neuron to node C
Correlation；If the transmitting between middle layer, consider each neuron of preceding layer to the phase between all neurons of later layer
At this moment Guan Xing has:
Wherein,Indicate the correlation in l layers between j-th of neuron and the prediction output of c class,Indicate l-1
Correlation in layer between i-th of neuron and j-th of neuron in next layer of l；
(2.4) according to law of conservation, the sum of l layers correlation of all neurons is equal to the correlation of output layer, institute
With i neuron and next layer of correlation, it is equal to its correlation with prediction result, has:
Wherein,Indicate the correlation of i-th of neuron and prediction result in l-1 layers；
Meanwhile being had according to the law of conservation of transmitting:
For a kind of visualized algorithm towards convolutional neural networks classification results, in the step (3), in order to obtain
The CAM of classification schemes, and needs first by the correlation elder generation back transfer of prediction result to last one layer of convolutional layer, because in convolutional layer
The spatial information in input picture is saved, therefore correlation is first successively transmitted to the last one layer of convolutional layer, in next step
It calculates CAM figure to prepare, in general CNN structure, the output of the last layer convolutional layer can be converted into one-dimensional by three-dimensional tensor
Vector, in order to connect subsequent full articulamentum, concrete implementation step includes:
(3.1) it is assumed that the output of the last layer convolutional layer is located at m layers of network, according to the law of conservation of correlation:
The sum of each neuron correlation for having the last layer convolutional layer to export according to the law of conservation of correlation is equal to last
Class score:
(3.2) when forward-propagating carries out classification prediction, the Feature Mapping figure of m layers of output in convolutional layer, that is, phase
The three-dimensional tensor answered has been converted to one-dimensional vector, and the spatial information extracted in feature has been given up in this conversion, so carrying out
When reversed correlation backpropagation calculates CAM figure, need first to indicate the one-dimensional vector of m layers of neuron correlation, it is corresponding
The space structure of three-dimensional tensor namely this layer of Feature Mapping figure when ground is converted into forward-propagating；
Algorithm is converted into the correlation with Feature Mapping map space structure first by m layers of one-dimensional correlation vector
Three-dimensional tensor, and since its value is one-to-one, so its summation remains unchanged, to the correlation tensor after conversion,
Wherein,Indicate in network in m layer correlation tensor neuron that coordinate in k-th of channel is (i, j) and
Predict the correlation between classification results；
(3.3) if carrying out global average pond, obtained result to the output feature in each channel are as follows:
Wherein, fkCoordinate is (i, j) in k-th of channel in the Feature Mapping figure of (i, j) expression the last layer convolutional layer
The activation value of neuron, thus:
It can be obtained compared with the calculation formula of CAM figure:
I.e. each Feature Mapping figure carry out global average Chi Huahou with the weight that finally exportsAs shown in above formula, this
Sample can obtain the CAM figure of the CNN model comprising full articulamentum after weighted sum, are as follows:
For a kind of visualized algorithm towards convolutional neural networks classification results, in the step (4), if used
CNN model have N layers of convolutional layer, each layer of index is 1,2 ..., and N uses matrix A in l layerslIndicate this layer of all nerve
First activation value, WlIndicate the weight matrix of connection this layer and preceding layer,Indicate k-th of neuron in l layers, XlIndicate l
Position in layer to the contributive neuron of last decision in Feature Mapping figure, that is, it is related to last output result
Property for positive value neuron position, m indicates the number of wherein neuron；Hereinafter, by being schemed based on the CAM that is previously obtained, in conjunction with mentioning
New transmission method out, inputted in support the CNN decision pixel position.
For a kind of visualized algorithm towards convolutional neural networks classification results, the step (5) specifically include with
Lower step:
(5.1) in XlEach of neuron index, extract activation in l-1 layers in corresponding receptive field
Value
(5.2) the Hadamard product (Hadamard product) of these activation values and corresponding convolution kernel weight is calculated
(5.3) available maximum logical to next layer of neuron contribution by summing to Hadamard product in each channel
Road, the neuron that Hadamard product is positive in the channel are just recorded by algorithm in contributive neuronal ensemble of classifying；
(5.4) wherein duplicate neuron is removed.
The beneficial effects of the present invention are: Rel-CAM algorithm of the present invention is in the classification for explaining convolutional neural networks
Aspect has higher accuracy, and the feature between classification can be distinguished in interpretive classification decision, to help people more
The classification foundation of good geography deconvolution neural network solves the prior art and pays no attention in the effect for explaining convolutional neural networks classification
The problem of thinking.
Detailed description of the invention
Fig. 1 is the flow diagram of the visualized algorithm towards convolutional neural networks classification results in the present invention；
Fig. 2 is Rel-CAM algorithm and the qualitative relatively figure of Backprop and LRP algorithm in the present invention；
Fig. 3 is that Rel-CAM algorithm and Backprop and LRP algorithm decline in classification confidence, classification confidence rise and
Decline the result figure of minimum scale.
Fig. 4 is the structural schematic diagram of image procossing of the present invention.
Specific embodiment
The present invention is described further with reference to the accompanying drawing.
In conjunction with Fig. 1, the invention discloses a kind of visualized algorithms towards convolutional neural networks classification results, by following
Step is realized:
Step 1: the data set comprising the input picture to be explained is used to instruct as training set to convolutional neural networks
Practice, obtains trained model parameter；
Step 2: according to calculation method of the Rel-CAM algorithm in full articulamentum, using output result and model parameter by
Layer calculates each neural unit in full articulamentum to the contribution of output, until convolutional layer；
Step 3: according to contribution of all neural units to output in the last layer convolutional layer obtained in step 2, meter
It calculates this layer of each channel and exports the respective weights of result, to obtain the Class Activation mapping graph of the network model；
Step 4: the neural unit that record Class Activation mapping graph intermediate value is positive, the position of the neural unit is as in the layer
To the position of the output contributive pixel of result, these neural units are added to the layer to the contributive neuronal ensemble of output
In；
Step 5: successively taking out each neuron in set, calculate in its preceding layer in receptive field all neurons with
The Hadamard product of respective weights, and sum to the Hadamard in each channel product, it takes it to neutralize maximum channel and is used as contribution channel,
And the neuron for taking its intermediate value to be positive is added to the layer to exporting in contributive neuronal ensemble, and removes wherein duplicate mind
Through member；
Step 6: repeating the communication process of step 5, until the neuronal ensemble in input layer is obtained, the mind in the set
The pixel under the position is shown through unit to contribute output result.
Currently, explaining that the method for visualizing of convolutional neural networks classification results is a hot topic of current machine learning research
Direction, domestic and foreign scholars propose a variety of model methods and corresponding algorithm, they are for different network models and specifically
Practical problem, respectively have a feature, the present invention exists on the basis of forefathers study for existing sensitivity analysis visualized algorithm
The accuracy of interpretive classification result is insufficient and efficiency of algorithm it is beneath, in conjunction with the advantages of Class Activation mapping algorithm and innovation,
The Class Activation mapping graph visualized algorithm based on correlation is proposed, main points of view and content are as follows:
(1) Rel-CAM algorithm calculation method in full articulamentum.Relevance propagation algorithm is to explain convolutional neural networks point
One of algorithms most in use of class, the overall thought of relevance propagation algorithm are the contributions for understanding each pixel to last prediction result
Size, it carries out backpropagation to correlation using the structure of network.Algorithm is reversed along network since the output layer of network
The direction of propagation redistributes the score of prediction classification at every layer, until input layer.And the process redistributed is kept
Constant rule, i.e., every layer of correlation summation remain unchanged.Herein correlation with R (x) indicate, wherein x indicate single pixel or
One neuron of person's middle layer.Certain a kind of CAM figure in order to obtain, it is necessary first to be transmitted to the prediction result of the last layer
On the last layer convolutional layer.
First, it is assumed that there is a trained CNN model, and there is a given input picture, model is divided into
C class, it is assumed that C node is such output node in output layer, and be scored at S at the nodec, select in algorithm
Output before Softmax layers is as class score, because in this case, output can be mapped to the position of only feature relevant to c class
It sets；If selecting Softmax layers of output, the output after normalization can be mapped to the position of the feature comprising other class, produce in this way
Raw visualization result will be inaccurate, because it contains the feature for being classified as other classes, although the probability meeting of only very little
Assign to other classes.So taking into account the above, algorithm uses the output valve before Softmax as the beginning of relevance propagation.In this way,
Have:
Wherein,Indicate the correlation for being predicted as c class neuron in output layer namely prediction result correlation in output layer
On distribution because only one node is related to c class in output layer, only one valueIn this way, preceding layers
The sum of the correlation of each neuron is alsoAssuming that output layer preceding layer is l, then this layer of each neural unit is to last
Output contribution degree, that is, the correlation with prediction result of each neuron is defined as:
In formula,Indicate the activation value of i-th of neuron in l layers, andIndicate the neural unit and the (output of next layer
Layer) weight connection between neuron, because the last layer only has c, class output node has correlation, need to only consider each
Correlation of the neuron to node C.But if it is the transmitting between middle layer, each nerve of preceding layer just must be taken into consideration
At this moment member has to the correlation between all neurons of later layer:
WhereinIndicate the correlation in l layers between j-th of neuron and the prediction output of c class,Indicate l-1 layers
In correlation between i-th of neuron and j-th of neuron in next layer of l, that is, contribution of the i neuron to j neuron
Size, then i neuron is exactly its contribution to next layer to the sum of the contribution of all neurons of next layer, and fixed according to conservation
Rule, the sum of l layers correlation of all neurons is equal to the correlation of output layer, so i neuron is related to next layer
Property, it is equal to its correlation with prediction result, has:
Wherein,Indicate the correlation of i-th of neuron and prediction result in l-1 layers.Simultaneously according to the conservation of transmitting
Law has:
The CAM figure of classification in order to obtain, needs the correlation elder generation back transfer of prediction result to last one layer of convolution first
Correlation is first successively transmitted to the last one because saving the spatial information in input picture in convolutional layer by layer
Layer convolutional layer is prepared to calculate CAM figure in next step.In general CNN structure, the output of the last layer convolutional layer can be by three
Dimension tensor is converted into one-dimensional vector, in order to connect subsequent full articulamentum.It is assumed that the output of the last layer convolutional layer is located at net
M layers of network, according to the law of conservation of correlation:
The sum of each neuron correlation for having the last layer convolutional layer to export according to the law of conservation of correlation is equal to last
Class score:
Because when forward-propagating carries out classification prediction, the Feature Mapping figure of m layers of output in convolutional layer, that is, accordingly
Three-dimensional tensor be converted to one-dimensional vector, in order in the propagated forward of full articulamentum.It is special that extraction has been given up in this conversion
Spatial information in sign, so needing first indicate m layers of mind when carrying out reversed correlation backpropagation calculating CAM figure
One-dimensional vector through first correlation, the sky of the three-dimensional tensor being accordingly converted into when forward-propagating namely this layer of Feature Mapping figure
Between structure.
Algorithm is converted into the correlation with Feature Mapping map space structure first by m layers of one-dimensional correlation vector
Three-dimensional tensor, and due to its value be it is one-to-one, so its summation remains unchanged.Therefore to the correlation after conversion
Amount, equally has:
Wherein,Indicate in network in m layer correlation tensor neuron that coordinate in k-th of channel is (i, j) and
Predict the correlation between classification results.If carrying out global average pond, obtained result to the output feature in each channel
Are as follows:
Wherein, fkCoordinate is (i, j) in k-th of channel in the Feature Mapping figure of (i, j) expression the last layer convolutional layer
The activation value of neuron, thus:
It can be obtained compared with the calculation formula of CAM figure:
I.e. each Feature Mapping figure carry out global average Chi Huahou with the weight that finally exportsAs shown in above formula, this
Sample can obtain the CAM figure of the CNN model comprising full articulamentum after weighted sum, are as follows:
(2) calculation method of the Rel-CAM algorithm in convolutional layer.In convolutional layer, Rel-CAM algorithm, which uses, is based on position
The algorithm that information is propagated.The core concept of the algorithm is: assuming that in current layer, if a neuron supports last classification knot
Fruit, that is to say, that it with last output the result is that positively related, then in preceding layer with the positively related mind of the neuron
It all can be considered the evidence for supporting current layer neuron through member, should also be deemed to be the evidence for supporting last classification results.Correlation
It is positive, that is, the product of the neuron in preceding layer and the weight between them is positive value.Here it is Rel-CAM algorithms to roll up
The core concept that lamination is successively propagated.
Firstly, if the CNN model used has N layers of convolutional layer, each layer of index is 1,2 ... N.In l layers, use
Matrix AlIndicate this layer of all neuronal activation value, WlIndicate the weight matrix of connection this layer and preceding layer,Indicate l layers
In k-th of neuron, XlIndicate the position in l layers to the contributive neuron of last decision in Feature Mapping figure, also
It is the position with the neuron that the correlation of last output result is positive value, m indicates the number of wherein neuron.Hereinafter, by base
In be previously obtained CAM figure, in conjunction with the new transmission method of proposition, inputted in support the CNN decision pixel position.
The figure of CAM obtained in a upper section is located at m layers of network, and the neuron that CAM figure intermediate value is positive is in the layer to most
Judgement result afterwards plays the neuron of contribution, therefore XlAsThe set of element position of the intermediate value greater than 0.It wants below
These location informations are successively transmitted until input layer.
After reaching convolutional layer, XlIt is the set of a three-dimensional index, it is right that each index identifies one, the layer
The position of the contributive neuron of last categorised decision.Being explained below methods herein and how reverse-locating in preceding layer has
The neuron of identification.It should be noted that the receptive field of the progress pondization operation of typical pond layer is a two-dimensional surface,
And receptive field when convolutional layer progress convolution operation is three-dimensional space, therefore, in XlEach of neuron rope
Draw, needs to extract the activation value in l-1 layers in corresponding receptive fieldThen these activation values and corresponding volume are calculated
The Hadamard product (Hadamard product) of product core weightBy summing to Hadamard product in each channel,
Available to contribute maximum channel to next layer of neuron, the neuron that Hadamard product is positive in the channel is just recorded by algorithm
To in contributive neuronal ensemble of classifying.
Following algorithm 1 explains the process for obtaining in convolutional layer and supporting classification neuron position.In algorithm 1,Indicate the activation value in the receptive field of preceding layer, therefore it is a three-dimensional tensor.When the weight of it and corresponding neuronWhen carrying out Hadamard product, obtained result is also the three-dimensional tensor of a same size.Algorithm is first along x-axis and y-axis
Direction sums to output, so that positioning most has the Feature Mapping figure of identification.If convolutional layer be not carried out it is any under adopt
Sample operation, then the spatial position of deterministic neuron is will not be changed during this conversion, that is,
Say, the position (x, y) in succeeding layer, which can be transferred in current layer, contributes on maximum channel, this completes layer and layer it
Between location information propagation.Algorithm can also further select the maximum neuron of activation value in contributing maximum channel, but
It is that result both in an experiment is almost the same, therefore algorithm has still selected the element in the maximum channel of contribution refreshing as determining
Through member.
The algorithm steps of location updating are as follows:
Algorithm 1: the neuron position propagation algorithm of categorised decision is supported in convolutional layer
Input: Xl, to contributive neuron position of classifying from high level obtained in CAM: Xl[1]...Xl[m]
Wl, l layers of weight
Al-1, the activation value of l-1 layers of neuron
Output: Xl-1, there is in l-1 layers the position for the neuron for supporting classification
1 enables Xl-1=φ
2for i=1:m do
3 neuronsCorresponding convolution kernel weight
4 neuronsActivation value in corresponding receptive field
5 pairs of activation values and weight seek Hadamard product
6 find out the contribution margin in each channel according to Hadamard product, that is, sum to channel each in long-pending tensor, and assign
It is worth to C, i.e.,Wherein S (x) is summed to planar element
7 are saved in the neuron position that Hadamard product is positive in channel the position concentration that neuron is determined in current layer,
I.e. in Xl-1Middle increase (Xl-1,argmax(C))
8end for
9 for Xl-1The identical position of intermediate value retains one of those.
In the case where containing pond layer, neuron in two-dimentional receptive field range in algorithm extraction preceding layer, and therefrom look for
The position of maximum activation value is provided, this is because most of CNN structure reflects feature usually using the method in maximum pond
It penetrates figure and carries out down-sampling.In this way, the activation in later layer corresponds to the maximum activation occurred in receptive field in preceding layer.Cause
This can select the maximum that possesses in the receptive field of corresponding neuron to swash when activation of the algorithm in down-sampling layer backtracking preceding layer
The neuron of value living.
In this way, Rel-CAM algorithm can be first since the prediction result of the last layer when training CNN is to be identified
The algorithm based on relevance propagation first is used in full articulamentum, the Class Activation that characteristic of division is positioned can be mapped by generating
Figure.Then the set comprising position collection is converted by mapping graph, the propagation algorithm using another kind based on location information,
Convolutional layer recalls the position with deterministic neuron, until input layer.Finally, it obtains over an input image certainly
Surely the positioning for the feature classified.Although input picture usually contains tri- channels RGB, algorithm only considers that the plane of x and y is empty
Between, that is to say, that pixel is only concerned in the positioning of two-dimensional space.
The method of the present invention has used for reference the concept of Class Activation mapping graph when explaining the classification results of convolutional neural networks, because
It is shown in an input picture for Class Activation mapping graph and determines that it is divided into the region of generic, thus it is of the invention
Method also has the advantages that this, by the qualitative experiment result figure of Fig. 2 can be seen that come, when image is classified as cat or dog, Rel-
CAM algorithm all only identifies the pixel region of respective class in image, and not to the pixel region of other classes or environment in image into
Line identifier；And Backprop method in both cases all identifies the feature of all classes, illustrates that this method is being explained
It cannot distinguish between the feature between classification when categorised decision.LRP method is close with Rel-CAM algorithm proposed in this paper, compares
The region of Backprop method positioning has class distinction, but the algorithm is labelled with more not crucial characteristic areas and ring
Pixel in border, and it is bigger than Rel-CAM algorithm in calculation amount, therefore the Rel-CAM algorithm in the present invention is explaining CNN
Categorised decision on possess better effect, especially when in image containing more than in the presence of a type objects.
Additionally by quantitative experiment compare three kinds of methods under same data set (a) classification confidence decline drop and
(b) classification confidence increases increase, it was demonstrated that the Rel-CAM algorithm in the present invention can more accurately explain convolutional Neural net
The classification of network.Interpretational criteria therein has used the concept of explanation figure, define image explain E as generate thermodynamic chart H with it is defeated
Enter picture I and carry out the corresponding multiplication of element:
In formula,Indicate the corresponding Hadamard product being multiplied of element, I is input picture, and H is the thermodynamic chart that can determine that classification.?
In the experiment of each picture, c indicates the classification being divided by model.In simple terms, explanation figure is exactly to obtain each pixel according to algorithm
The importance to model decision, covered a part of input picture.
The average decline of classification confidence: a good explanation figure should be marked to most important part of classifying.Depth
CNN model can make final decision, therefore a part of shielded image according to all features of input picture in prediction
Confidence level of the model in decision will necessarily be reduced.On the other hand, it is influenced due to being remained in explanation figure in entire input picture
The most important part of the categorised decision, so this decline should be smaller.Therefore, this measurement compares a picture and exists
To the decline of certain kinds model confidence after covering.For example, if an image prediction is by known models with 0.8 confidence level
Tiger, when being predicted using explanation figure, this image classification is dropped to 0.4 at the confidence level of tiger by model.So mould
Type confidence level falls to 50%.Every a kind of highest 50 images of prediction have carried out Experimental comparison in experimental selection data set,
Compare the average decline of several algorithms.
Classification confidence increases: however, the portion that all features that sometimes possible CNN is found are identified all in explanation figure
Point, and the feature of other parts is all unnecessary feature, does not play help to categorised decision.In this case, mould
Type has increase to the confidence level of certain kinds instead.The entire data of this metric calculation are concentrated use in explanation figure when being predicted,
The increased number of model confidence, this number are expressed as a percentage.
Classification confidence decline is minimum: the explanation figure that the first two criterion can evaluate the generation of a method for visualizing is correctly marked
Know the ability that the region of classification is influenced in image, and this criterion is explicitly compare explanation figure that different method generates excellent
It is bad.It refers in given data set, the explanation figure that more each method for visualizing generates classification confidence on every image
The size of decline, which kind of method declines minimum, then its decline minimum number adds 1.Confidence level declines minimum, illustrates this
Such prior characteristic of division is identified in the explanation figure that method generates, that is to say, that it explanatory more preferable.Finally
Output is a percentage, i.e., the decline minimum number of every kind method accounts for the ratio of all algorithms.
By experimental analysis, as shown in figure 3, Rel-CAM algorithm wants low in terms of the average decline decline of classification confidence
In other two kinds of existing algorithms, in terms of confidence level increase, Rel-CAM algorithm is equally matched with other algorithms, only slightly has
A little advantages.On the other hand the experimental results showed that can help the positioning of feature to be promoted the performance of classifier, this may be depth
The researcher of study provides a new angle and goes to promote the performance of neural network, and as model increases specific identification component,
Then it instructs to train further according to another characteristic is known, and then promotes network performance.
Decline minimum aspect in confidence level, it can be seen that ratio shared by Rel-CAM algorithm is maximum, that is to say, that entire
In data set, in many cases, Rel-CAM algorithm, which can be identified, influences maximum feature to classification, illustrates that Rel-CAM is wanted
Better than other two methods.
To sum up, by qualitative and quantitative analysis, Rel-CAM algorithm of the present invention is in point for explaining convolutional neural networks
There is higher accuracy in terms of class, and the feature between classification can be distinguished in interpretive classification decision, to help people
More fully understand the classification foundation of convolutional neural networks.
The foregoing is only a preferred embodiment of the present invention, is not intended to restrict the invention, for the skill of this field
For art personnel, the invention may be variously modified and varied.All within the spirits and principles of the present invention, made any to repair
Change, equivalent replacement, improvement etc., should all be included in the protection scope of the present invention.
Claims (5)
1. a kind of visualized algorithm towards convolutional neural networks classification results, which is characterized in that especially by following steps reality
It is existing:
(1) data set for extracting input picture, is trained convolutional neural networks using data set as training set, is trained
Good model parameter；
(2) calculation method according to Rel-CAM algorithm in full articulamentum is successively calculated entirely using output result and model parameter
Each neural unit is to the contribution of output in articulamentum, until convolutional layer；
(3) according to contribution of all neural units to output in the last layer convolutional layer obtained in step (2), it is every to calculate the layer
The respective weights in a channel and output result, to obtain the Class Activation mapping graph of the network model；
(4) output is tied as in the layer position of the neural unit that record Class Activation mapping graph intermediate value is positive, the neural unit
These neural units are added to the layer in the contributive neuronal ensemble of output by the position of the contributive pixel of fruit；
(5) each neuron in set is successively taken out, all neurons and respective weights in receptive field are calculated in its preceding layer
Hadamard product, and sum to the Hadamard in each channel product, take it to neutralize maximum channel as contribution channel, and take wherein
The neuron that value is positive is added to the layer to exporting in contributive neuronal ensemble, and removes wherein duplicate neuron；
(6) communication process of step (5) is repeated, until the neuronal ensemble in input layer is obtained, the neural unit in the set
The pixel shown under the position contributes output result.
2. a kind of visualized algorithm towards convolutional neural networks classification results according to claim 1, which is characterized in that
The step (2) is realized by following steps:
(2.1) assume that a trained CNN model and a given input picture, model have been divided into c class, it is assumed that
C node is such output node in output layer, and is scored at S at the nodec, in algorithm before Softmax layers of selection
Output as class score, output is mapped to the position of only feature relevant to c class,
Wherein,Indicate the correlation that c class neuron is predicted as in output layer, i.e. point of the prediction result correlation on output layer
Cloth；
(2.2) assume that output layer preceding layer is l, then contribution degree of this layer of each neural unit to last output, that is,
The correlation with prediction result of each neuron is defined as:
Wherein,Indicate the activation value of i-th of neuron in l layers, andIndicate the neural unit and next layer, i.e. output layer
Weight connection between neuron；
(2.3) since the last layer only has c class output node to have correlation, only consider each neuron to node C phase
Guan Xing；If the transmitting between middle layer, consider each neuron of preceding layer to the correlation between all neurons of later layer,
At this moment have:
Wherein,Indicate the correlation in l layers between j-th of neuron and the prediction output of c class,It indicates in l-1 layers
Correlation between i-th of neuron and j-th of neuron in next layer of l；
(2.4) according to law of conservation, the sum of l layers correlation of all neurons is equal to the correlation of output layer, so i is refreshing
Correlation through member with next layer, is equal to its correlation with prediction result, has:
Wherein,Indicate the correlation of i-th of neuron and prediction result in l-1 layers；
Meanwhile being had according to the law of conservation of transmitting:
3. a kind of visualized algorithm towards convolutional neural networks classification results according to claim 1, which is characterized in that
In the step (3), in order to obtain classification CAM figure, need first by the correlation elder generation back transfer of prediction result to the end
One layer of convolutional layer first successively transmits correlation straight because saving the spatial information in input picture in convolutional layer
To last one layer of convolutional layer, prepare to calculate CAM figure in next step, in general CNN structure, the last layer convolutional layer it is defeated
One-dimensional vector can be converted by three-dimensional tensor out, in order to connect subsequent full articulamentum, concrete implementation step includes:
(3.1) it is assumed that the output of the last layer convolutional layer is located at m layers of network, according to the law of conservation of correlation:
The sum of each neuron correlation for having the last layer convolutional layer to export according to the law of conservation of correlation is equal to last class
Score:
(3.2) when forward-propagating carries out classification prediction, the Feature Mapping figure of m layers of output in convolutional layer, that is, accordingly
Three-dimensional tensor has been converted to one-dimensional vector, and the spatial information extracted in feature has been given up in this conversion, so carrying out reversely
Correlation backpropagation calculate CAM figure when, need first will indicate m layers of neuron correlation one-dimensional vector, accordingly turn
Turn to the space structure of the three-dimensional tensor namely this layer of Feature Mapping figure when forward-propagating；
Algorithm is converted into three of the correlation with Feature Mapping map space structure first by m layers of one-dimensional correlation vector
Tensor is tieed up, and since its value is one-to-one, so its summation remains unchanged, to the correlation tensor after conversion,
Wherein,Indicate that coordinate is neuron and the prediction point of (i, j) in k-th of channel in m layers of correlation tensor in network
Correlation between class result；
(3.3) if carrying out global average pond, obtained result to the output feature in each channel are as follows:
Wherein, fkCoordinate is the neuron of (i, j) in k-th of channel in the Feature Mapping figure of (i, j) expression the last layer convolutional layer
Activation value, thus:
It can be obtained compared with the calculation formula of CAM figure:
I.e. each Feature Mapping figure carry out global average Chi Huahou with the weight that finally exportsAs shown in above formula, in this way, plus
The CAM figure of the CNN model comprising full articulamentum can be obtained after power summation, are as follows:
4. a kind of visualized algorithm towards convolutional neural networks classification results according to claim 1, it is characterised in that:
In the step (4), if the CNN model used has N layers of convolutional layer, each layer of index is 1,2 ... N, in l layers,
Use matrix AlIndicate this layer of all neuronal activation value, WlIndicate the weight matrix of connection this layer and preceding layer,Indicate l
K-th of neuron in layer, XlIndicate the position in l layers to the contributive neuron of last decision in Feature Mapping figure,
It is exactly the position with the neuron that the correlation of last output result is positive value, m indicates the number of wherein neuron；Hereinafter, will
Based on be previously obtained CAM figure, in conjunction with the new transmission method of proposition, inputted in support the CNN decision pixel position
It sets.
5. a kind of visualized algorithm towards convolutional neural networks classification results according to claim 1, which is characterized in that
The step (5) specifically includes the following steps:
(5.1) in XlEach of neuron index, extract activation value in l-1 layers in corresponding receptive field
(5.2) the Hadamard product (Hadamard product) of these activation values and corresponding convolution kernel weight is calculated
(5.3) available that maximum channel is contributed to next layer of neuron by summing to Hadamard product in each channel, it should
The neuron that Hadamard product is positive in channel is just recorded by algorithm in contributive neuronal ensemble of classifying；
(5.4) wherein duplicate neuron is removed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810519569.7A CN109858506B (en) | 2018-05-28 | 2018-05-28 | Visualization algorithm for classification result of convolutional neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810519569.7A CN109858506B (en) | 2018-05-28 | 2018-05-28 | Visualization algorithm for classification result of convolutional neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109858506A true CN109858506A (en) | 2019-06-07 |
| CN109858506B CN109858506B (en) | 2022-11-18 |
Family
ID=66889621
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810519569.7A Active CN109858506B (en) | 2018-05-28 | 2018-05-28 | Visualization algorithm for classification result of convolutional neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109858506B (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110533093A (en) * | 2019-08-24 | 2019-12-03 | 大连理工大学 | A kind of automobile front face brand family analysis method |
| CN110769258A (en) * | 2019-11-05 | 2020-02-07 | 山东浪潮人工智能研究院有限公司 | Image compression method and system for multi-semantic region of specific scene |
| CN110852394A (en) * | 2019-11-13 | 2020-02-28 | 联想(北京)有限公司 | Data processing method and device, computer system and readable storage medium |
| CN111046939A (en) * | 2019-12-06 | 2020-04-21 | 中国人民解放军战略支援部队信息工程大学 | CNN (CNN) class activation graph generation method based on attention |
| CN111401472A (en) * | 2020-04-09 | 2020-07-10 | 中国人民解放军国防科技大学 | Infrared target classification method and device based on deep convolutional neural network |
| CN111553462A (en) * | 2020-04-08 | 2020-08-18 | 哈尔滨工程大学 | Class activation mapping method |
| CN111582376A (en) * | 2020-05-09 | 2020-08-25 | 北京字节跳动网络技术有限公司 | Neural network visualization method and device, electronic equipment and medium |
| CN111666861A (en) * | 2020-06-01 | 2020-09-15 | 浙江工业大学 | Wireless signal modulation classifier visualization method based on convolutional neural network |
| CN112347252A (en) * | 2020-11-04 | 2021-02-09 | 吉林大学 | Interpretability analysis method based on CNN text classification model |
| CN112651407A (en) * | 2020-12-31 | 2021-04-13 | 中国人民解放军战略支援部队信息工程大学 | CNN visualization method based on discriminative deconvolution |
| CN112735514A (en) * | 2021-01-18 | 2021-04-30 | 清华大学 | Training and visualization method and system for neural network extraction regulation and control DNA combination mode |
| CN115829005A (en) * | 2022-12-09 | 2023-03-21 | 之江实验室 | Automatic defect diagnosis and repair method and device for convolutional neural classification network |
| CN116561752A (en) * | 2023-07-07 | 2023-08-08 | 华测国软技术服务南京有限公司 | Safety testing method for application software |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106909945A (en) * | 2017-03-01 | 2017-06-30 | 中国科学院电子学研究所 | The feature visualization and model evaluation method of deep learning |
| US20170200078A1 (en) * | 2014-08-28 | 2017-07-13 | Commissariat A L'energie Atomique Et Aux Energies Alternatives | Convolutional neural network |
| CN107392085A (en) * | 2017-05-26 | 2017-11-24 | 上海精密计量测试研究所 | The method for visualizing convolutional neural networks |
| CN107766933A (en) * | 2017-10-24 | 2018-03-06 | 天津大学 | A kind of method for visualizing for explaining convolutional neural networks |
-
2018
- 2018-05-28 CN CN201810519569.7A patent/CN109858506B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170200078A1 (en) * | 2014-08-28 | 2017-07-13 | Commissariat A L'energie Atomique Et Aux Energies Alternatives | Convolutional neural network |
| CN106909945A (en) * | 2017-03-01 | 2017-06-30 | 中国科学院电子学研究所 | The feature visualization and model evaluation method of deep learning |
| CN107392085A (en) * | 2017-05-26 | 2017-11-24 | 上海精密计量测试研究所 | The method for visualizing convolutional neural networks |
| CN107766933A (en) * | 2017-10-24 | 2018-03-06 | 天津大学 | A kind of method for visualizing for explaining convolutional neural networks |
Non-Patent Citations (2)
| Title |
|---|
| RAMPRASAATH R. SELVARAJU ET AL.: "《Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization》", 《IEEE XPLORE》 * |
| 张美多等: "基于最低冗余度策略的医学图像可视化技术研究", 《光电子.激光》 * |
Cited By (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110533093A (en) * | 2019-08-24 | 2019-12-03 | 大连理工大学 | A kind of automobile front face brand family analysis method |
| CN110769258A (en) * | 2019-11-05 | 2020-02-07 | 山东浪潮人工智能研究院有限公司 | Image compression method and system for multi-semantic region of specific scene |
| CN110852394A (en) * | 2019-11-13 | 2020-02-28 | 联想(北京)有限公司 | Data processing method and device, computer system and readable storage medium |
| CN110852394B (en) * | 2019-11-13 | 2022-03-25 | 联想(北京)有限公司 | Data processing method and device, computer system and readable storage medium |
| CN111046939A (en) * | 2019-12-06 | 2020-04-21 | 中国人民解放军战略支援部队信息工程大学 | CNN (CNN) class activation graph generation method based on attention |
| CN111046939B (en) * | 2019-12-06 | 2023-08-04 | 中国人民解放军战略支援部队信息工程大学 | Attention-based CNN class activation graph generation method |
| CN111553462A (en) * | 2020-04-08 | 2020-08-18 | 哈尔滨工程大学 | Class activation mapping method |
| CN111401472B (en) * | 2020-04-09 | 2023-11-24 | 中国人民解放军国防科技大学 | Infrared target classification method and device based on deep convolutional neural network |
| CN111401472A (en) * | 2020-04-09 | 2020-07-10 | 中国人民解放军国防科技大学 | Infrared target classification method and device based on deep convolutional neural network |
| CN111582376A (en) * | 2020-05-09 | 2020-08-25 | 北京字节跳动网络技术有限公司 | Neural network visualization method and device, electronic equipment and medium |
| CN111582376B (en) * | 2020-05-09 | 2023-08-15 | 抖音视界有限公司 | Visualization method and device for neural network, electronic equipment and medium |
| CN111666861A (en) * | 2020-06-01 | 2020-09-15 | 浙江工业大学 | Wireless signal modulation classifier visualization method based on convolutional neural network |
| CN112347252A (en) * | 2020-11-04 | 2021-02-09 | 吉林大学 | Interpretability analysis method based on CNN text classification model |
| CN112347252B (en) * | 2020-11-04 | 2024-02-27 | 吉林大学 | Interpretability analysis method based on CNN text classification model |
| CN112651407A (en) * | 2020-12-31 | 2021-04-13 | 中国人民解放军战略支援部队信息工程大学 | CNN visualization method based on discriminative deconvolution |
| CN112651407B (en) * | 2020-12-31 | 2023-10-20 | 中国人民解放军战略支援部队信息工程大学 | CNN visualization method based on discriminative deconvolution |
| CN112735514A (en) * | 2021-01-18 | 2021-04-30 | 清华大学 | Training and visualization method and system for neural network extraction regulation and control DNA combination mode |
| CN112735514B (en) * | 2021-01-18 | 2022-09-16 | 清华大学 | Training and visualization method and system for neural network extraction regulation and control DNA combination mode |
| CN115829005A (en) * | 2022-12-09 | 2023-03-21 | 之江实验室 | Automatic defect diagnosis and repair method and device for convolutional neural classification network |
| CN116561752B (en) * | 2023-07-07 | 2023-09-15 | 华测国软技术服务南京有限公司 | Safety testing method for application software |
| CN116561752A (en) * | 2023-07-07 | 2023-08-08 | 华测国软技术服务南京有限公司 | Safety testing method for application software |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109858506B (en) | 2022-11-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109858506A (en) | A kind of visualized algorithm towards convolutional neural networks classification results | |
| Gao et al. | The deep features and attention mechanism-based method to dish healthcare under social IoT systems: An empirical study with a hand-deep local–global net | |
| CN111368896B (en) | Hyperspectral remote sensing image classification method based on dense residual three-dimensional convolutional neural network | |
| CN108537742A (en) | A kind of panchromatic sharpening method of remote sensing images based on generation confrontation network | |
| CN106649487A (en) | Image retrieval method based on interest target | |
| CN110059741A (en) | Image-recognizing method based on semantic capsule converged network | |
| CN109543602A (en) | A kind of recognition methods again of the pedestrian based on multi-view image feature decomposition | |
| CN116051574A (en) | Semi-supervised segmentation model construction and image analysis method, device and system | |
| CN109272487A (en) | The quantity statistics method of crowd in a kind of public domain based on video | |
| CN109657582A (en) | Recognition methods, device, computer equipment and the storage medium of face mood | |
| CN109903339B (en) | Video group figure positioning detection method based on multi-dimensional fusion features | |
| CN109711401A (en) | A kind of Method for text detection in natural scene image based on Faster Rcnn | |
| CN114092697B (en) | Building facade semantic segmentation method with attention fused with global and local depth features | |
| CN109377441A (en) | Tongue with privacy protection function is as acquisition method and system | |
| CN108764018A (en) | A kind of multitask vehicle based on convolutional neural networks recognition methods and device again | |
| CN110211127A (en) | Image partition method based on bicoherence network | |
| CN110490189A (en) | A kind of detection method of the conspicuousness object based on two-way news link convolutional network | |
| CN111126155B (en) | Pedestrian re-identification method for generating countermeasure network based on semantic constraint | |
| Hu et al. | Supervised multi-scale attention-guided ship detection in optical remote sensing images | |
| CN114612709A (en) | Multi-scale target detection method guided by image pyramid characteristics | |
| Shen et al. | Multi-dimensional, multi-functional and multi-level attention in YOLO for underwater object detection | |
| CN116843952A (en) | Small sample learning classification method for fruit and vegetable disease identification | |
| CN111369124A (en) | Image aesthetic prediction method based on self-generation global features and attention | |
| Sun et al. | Automatic building age prediction from street view images | |
| CN114332107A (en) | Improved tunnel lining water leakage image segmentation method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20231225 Address after: Room 2106-501, Building 4, Innovation and Entrepreneurship Square, Science and Technology Innovation City, Harbin High tech Industrial Development Zone, Heilongjiang Province, 150001 (No. 689 Shize Road, Songbei District) Patentee after: Harbin Three Cup Tea Technology Co.,Ltd. Address before: 150001 Intellectual Property Office, Harbin Engineering University science and technology office, 145 Nantong Avenue, Nangang District, Harbin, Heilongjiang Patentee before: HARBIN ENGINEERING University |