CN114625657A - Model interpretation method and device, electronic equipment and storage medium - Google Patents
Model interpretation method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN114625657A CN114625657A CN202210282784.6A CN202210282784A CN114625657A CN 114625657 A CN114625657 A CN 114625657A CN 202210282784 A CN202210282784 A CN 202210282784A CN 114625657 A CN114625657 A CN 114625657A
- Authority
- CN
- China
- Prior art keywords
- model
- feature
- dimension
- sample
- importance degree
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 71
- 238000004458 analytical method Methods 0.000 claims abstract description 19
- 238000004891 communication Methods 0.000 claims description 10
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000012935 Averaging Methods 0.000 claims description 4
- 238000013145 classification model Methods 0.000 claims description 4
- 235000000332 black box Nutrition 0.000 claims description 3
- 230000006399 behavior Effects 0.000 description 19
- 238000005516 engineering process Methods 0.000 description 15
- 230000006870 function Effects 0.000 description 15
- 230000000694 effects Effects 0.000 description 11
- 238000013473 artificial intelligence Methods 0.000 description 8
- 238000013135 deep learning Methods 0.000 description 7
- 238000012545 processing Methods 0.000 description 7
- 238000010801 machine learning Methods 0.000 description 6
- 230000008859 change Effects 0.000 description 4
- 238000003066 decision tree Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000001514 detection method Methods 0.000 description 3
- 238000010295 mobile communication Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 238000007477 logistic regression Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000003058 natural language processing Methods 0.000 description 2
- 238000003062 neural network model Methods 0.000 description 2
- 238000012954 risk control Methods 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 230000005236 sound signal Effects 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 238000007599 discharging Methods 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000010079 rubber tapping Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000010897 surface acoustic wave method Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 230000007723 transport mechanism Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3604—Software analysis for verifying properties of programs
- G06F11/3608—Software analysis for verifying properties of programs using formal methods, e.g. model checking, abstract interpretation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3604—Software analysis for verifying properties of programs
- G06F11/3616—Software analysis for verifying properties of programs using software metrics
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the application provides a model interpretation method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a pre-trained black box model, a sample set of a target task scene and a target sample to be interpreted; inputting the sample set into the black box model to obtain characteristic distribution information; inputting the target sample into the black box model for model prediction to obtain a model prediction score; traversing features of each dimension in the target sample; calculating a weighted average score corresponding to the feature of each dimension according to the feature distribution information; determining the importance degree of the feature of each dimension in the target sample according to the difference value between the weighted average and the model prediction score; and interpreting the black box model based on the model of the target sample according to the importance degree. The method and the device can help engineers explain the behavior of the model in a specific sample to perform feature significance analysis.
Description
Technical Field
The present application relates to the field of artificial intelligence technologies, and in particular, to a model interpretation method and apparatus, an electronic device, and a storage medium.
Background
Under the wind control scene, especially when the model prediction result is used for functions of customer behavior interception, warning and the like, when the model sends out wrong interception and warning signals, the risk control function cannot be achieved, the customer experience is greatly influenced, and therefore the wind control scene has high requirements on model interpretability. Therefore, engineers in the wind control scene often need to explain the model, and if the models cannot be explained, the effect of the models is easily questioned by front-line business, so that the engineers can select the models with high interpretability.
In the related art, there is often a conflict between the model interpretability and performance. Models with high interpretability are often poor in effect, such as logistic regression and single decision tree, have good interpretability, and are poor in effect. And some models with better effect, such as neural network and gbdt (multi-decision tree integration), are difficult to explain the behavior of a single sample. In the face of conflicts between interpretability and performance, wind control engineers often move back to find next, selecting models with high interpretability and poor results. Therefore, how to help engineers explain the behavior of the model in a specific sample to perform feature significance analysis becomes a technical problem to be solved urgently in the field.
Disclosure of Invention
The embodiment of the application mainly aims to provide a model interpretation method, a model interpretation device, electronic equipment and a storage medium, which can help engineers interpret the behavior of a model in a specific sample to perform feature significance analysis.
To achieve the above object, a first aspect of an embodiment of the present application provides a model interpretation method, including:
acquiring a pre-trained black box model, a sample set of a target task scene and a target sample to be interpreted;
inputting the sample set into the black box model to obtain characteristic distribution information;
inputting the target sample into the black box model for model prediction to obtain a model prediction score;
traversing the feature of each dimension in the target sample, and calculating a weighted average score corresponding to the feature of each dimension according to the feature distribution information;
determining the importance degree of the feature of each dimension in the target sample according to the difference value between the weighted average and the model prediction score;
and interpreting the black box model based on the model of the target sample according to the importance degree of the characteristics.
In some embodiments, the feature distribution information includes feature values corresponding to the features of each dimension in the sample set and a proportion of the feature values in the sample set.
In some embodiments, the calculating a weighted average score corresponding to the feature for each dimension according to the feature distribution information includes:
replacing the features of each dimensionality in the target sample with feature values corresponding to the features of the same dimensionality in the sample set to obtain a new sample;
carrying out model scoring on the new sample through the black box model to obtain a model scoring score;
and obtaining a weighted average score corresponding to the feature of each dimension in the target sample according to the model score and the proportion of the feature value in the sample set.
In some embodiments, determining the importance of the feature in the target sample for each dimension according to the difference between the weighted average and the model prediction score comprises:
calculating the difference between the weighted average corresponding to the features of each dimensionality and the model prediction fraction to obtain a first difference value;
calculating the difference between the weighted average corresponding to the features in all dimensions and the model prediction score to obtain a second difference value;
and dividing the first difference value by the second difference value to obtain a first feature importance degree value, wherein the first feature importance degree value is used for representing the importance degree of the feature in each dimension in the target sample, and the size of the first feature importance degree value is in direct proportion to the importance degree.
In some embodiments, interpreting the black box model based on the model of the target sample according to the degree of importance includes:
sorting the features according to the magnitude of the first feature importance degree value;
determining the first N characteristics with the highest importance degree value of the first characteristics;
locally interpretable of the black box model based on the model of the target sample using the first N of the features, where N is a positive integer greater than zero.
In some embodiments, said interpreting the black box model based on a model of the target sample according to the importance level comprises:
calculating the first feature importance degree value corresponding to the feature of each dimension of each sample in the sample set;
averaging the sum of the first feature importance degree values corresponding to the features of each dimension of each sample to obtain a second feature importance degree value, wherein the second feature importance degree value is used for representing the importance degree of the features of each dimension in the sample set, and the magnitude of the second feature importance degree value is in direct proportion to the importance degree;
sorting the features of each dimension in the sample set according to the magnitude of the second feature importance degree value;
determining the characteristics of the first N dimensions with the highest second characteristic importance degree values;
globally interpretable of the black-box model based on the model of the target sample using the features of the first N dimensions, where N is a positive integer greater than zero.
In some embodiments, the black box model is a classification model.
To achieve the above object, a second aspect of embodiments of the present application proposes a model interpretation apparatus, including:
the acquisition module is used for acquiring a pre-trained black box model, a sample set of a target task scene and a target sample to be interpreted;
the characteristic analysis module is used for inputting the sample set into the black box model to obtain characteristic distribution information;
the prediction module is used for inputting the target sample into the black box model to perform model prediction to obtain a model prediction score;
the calculation module is used for traversing the characteristics of each dimension in the target sample and calculating the weighted average score corresponding to the characteristics of each dimension according to the characteristic distribution information;
the analysis module is used for determining the importance degree of the feature of each dimension in the target sample according to the difference value between the weighted average and the model prediction score;
and the interpretation module is used for interpreting the black box model based on the model of the target sample according to the importance degree of the characteristics.
In order to achieve the above object, a third aspect of the embodiments of the present application provides an electronic device, which includes a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for implementing connection communication between the processor and the memory, wherein the program, when executed by the processor, implements the method of the first aspect.
To achieve the above object, a fourth aspect of the embodiments of the present application proposes a storage medium, which is a computer-readable storage medium for computer-readable storage, and stores one or more programs, which are executable by one or more processors to implement the method of the first aspect.
According to the model interpretation method, the model interpretation device, the electronic equipment and the storage medium, a pre-trained black box model, a sample set of a target task scene and a target sample to be interpreted are obtained; inputting the sample set into a black box model to obtain characteristic distribution information; inputting a target sample into a black box model for model prediction to obtain a model prediction score; traversing the characteristics of each dimension in the target sample, and calculating a weighted average score corresponding to each dimension characteristic according to the characteristic distribution information; determining the importance degree of each dimension characteristic in the target sample according to the difference value of the weighted average and the model prediction score; according to the model interpretation method, the model is regarded as an invisible black box without considering model implementation details, and the importance degree of a certain characteristic on the model is analyzed by adjusting the change of model input and model output, so that an engineer is helped to interpret the behavior of the model on a specific sample to perform characteristic significance analysis, and the behavior of the model on a single case and the overall behavior of the model can be interpreted.
Drawings
FIG. 1 is a schematic diagram of a system architecture platform for performing a model interpretation method provided by one embodiment of the present application;
FIG. 2 is a flow chart of a model interpretation method provided by one embodiment of the present application;
FIG. 3 is a sub-flowchart of a particular method of step S400 in FIG. 2;
FIG. 4 is a sub-flowchart of a particular method of step S500 in FIG. 2;
FIG. 5 is a sub-flowchart of a particular method of step S600 in FIG. 2;
FIG. 6 is another sub-flowchart of a specific method of step S600 in FIG. 2;
FIG. 7 is a schematic diagram of a model interpretation apparatus provided in one embodiment of the present application;
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
It is noted that while functional block divisions are provided in device diagrams and logical sequences are shown in flowcharts, in some cases, steps shown or described may be performed in sequences other than block divisions within devices or flowcharts. The terms first, second and the like in the description and in the claims, and the drawings described above, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the present application only and is not intended to be limiting of the application.
The embodiment of the application can acquire and process related data based on an artificial intelligence technology. Among them, Artificial Intelligence (AI) is a theory, method, technique and application system that simulates, extends and expands human Intelligence using a digital computer or a machine controlled by a digital computer, senses the environment, acquires knowledge and uses the knowledge to obtain the best result.
The artificial intelligence infrastructure generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a robot technology, a biological recognition technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.
Deep Learning (DL) is a new research direction in the field of Machine Learning (ML), which is introduced into Machine Learning to make it closer to the original target, Artificial Intelligence (AI). Deep learning is the intrinsic law and expression level of the learning sample data, and the information obtained in the learning process is very helpful for the interpretation of data such as characters, images and sounds. The final aim of the method is to enable the machine to have the analysis and learning capability like a human, and to recognize data such as characters, images and sounds. Deep learning is a complex machine learning algorithm, and achieves the effect in speech and image recognition far exceeding the prior related art. Deep learning has achieved a number of achievements in search technology, data mining, machine learning, machine translation, natural language processing, multimedia learning, speech, recommendation and personalization technologies, and other related fields. The deep learning enables the machine to imitate human activities such as audio-visual and thinking, solves a plurality of complex pattern recognition problems, and makes great progress on the artificial intelligence related technology.
Under the wind control scene, especially when the model prediction result is used for functions such as customer behavior interception and warning, when the model sends wrong interception and warning signals, the risk control function cannot be achieved, and customer experience is greatly influenced, so that the wind control scene has high requirements on model interpretability. Therefore, engineers in the wind control scene often need to explain the model, and if the models cannot be explained, the effect of the models is easily questioned by front-line business, so that the engineers can select the models with high interpretability.
In the related art, there is often a conflict between the model interpretability and performance. Models with high interpretability are often poor in effect, such as logistic regression and single decision tree, have good interpretability, and are poor in effect. And some models with better effect, such as neural networks and gbdt (multi-decision tree integration), are difficult to explain the behavior of a single sample. In the face of conflicts between interpretability and performance, wind control engineers often move back to find next, selecting models with high interpretability and poor results. Therefore, how to help engineers explain the behavior of the model in a specific sample to perform feature significance analysis becomes a technical problem to be solved urgently in the field.
In order to solve the above technical problems, embodiments of the present invention provide a model interpretation method, apparatus, electronic device, and storage medium, in which a pre-trained black box model, a sample set of a target task scene, and a target sample to be interpreted are obtained; inputting the sample set into a black box model to obtain characteristic distribution information; inputting a target sample into a black box model for model prediction to obtain a model prediction score; traversing the features of each dimension in the target sample, and calculating the weighted average score corresponding to each dimension feature according to feature distribution information; determining the importance degree of each dimension characteristic in the target sample according to the difference value of the weighted average and the model prediction score; according to the model interpretation method, the model is regarded as an invisible black box without paying attention to the implementation details of the model, the importance degree of a certain characteristic to the model is analyzed by adjusting the input of the model and observing the change of the output of the model, so that an engineer is helped to interpret the behavior of the model on a specific sample to perform characteristic significance analysis, and the behavior of the model on a single case and the overall behavior of the model can be interpreted.
First, several terms referred to in the present application are resolved:
1. black box model: for a model, its internal structure is not clear, but the black-box model returns output results given the inputs.
Inputting: the common data type of the wind control scene is table data, which is different from data such as images and texts. One row in the tabular data is a sample and one column is a feature. The features are divided into numerical features and discrete features.
And (3) outputting: taking the classification model as an example, the model will output a probability distribution for each classification label:
P＝[P0,P1,……,PC-1]
a total of B classification classes, PiProbability distribution of ith class label predicted for model, where PiLess than or equal to 1 and
2. interpretability:
model interpretability is generally divided into two categories: local interpretability, global interpretability.
Locally interpretable: finger model explains the specific case. For example, the model makes risk interception for a certain case, and it needs to explain why the model intercepts the case, and makes an interception action according to the values of which features.
Globally interpretable: and generally refers to the behavior rules of the model for all samples. E.g. a model of the task scenario, mainly learn which features.
The technical solutions of the embodiments of the present application are further described below with reference to the accompanying drawings.
As shown in fig. 1, fig. 1 is a schematic diagram of a system architecture platform 100 for executing a model interpretation method according to an embodiment of the present invention.
In the example of fig. 1, the system architecture platform 100 is provided with a processor 110 and a memory 120, wherein the processor 110 and the memory 120 may be connected by a bus or other means, and fig. 1 illustrates the connection by the bus as an example.
The memory 120, which is a non-transitory computer readable storage medium, may be used to store non-transitory software programs as well as non-transitory computer executable programs. Further, the memory 120 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 120 optionally includes memory located remotely from processor 110, which may be connected to the system architecture platform via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
It will be understood by those skilled in the art that the system architecture platform may be applied to existing communication network systems, mobile communication network systems evolved later, and the like, and the embodiment is not limited thereto.
Those skilled in the art will appreciate that the system architecture platform illustrated in FIG. 1 does not constitute a limitation on embodiments of the invention, and may include more or fewer components than those illustrated, or some components may be combined, or a different arrangement of components.
The system architecture platform 100 may be an independent system architecture platform, or may be a cloud system architecture platform 100 that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, web services, cloud communications, middleware services, domain name services, security services, Content Delivery Networks (CDNs), and big data and artificial intelligence platforms.
Based on the above system architecture platform, various embodiments of the model interpretation method of the present invention are presented below.
As shown in fig. 2, fig. 2 is a flowchart of a model interpretation method according to an embodiment of the present invention, which can be applied to the above-mentioned architecture platform, and includes, but is not limited to, step S100, step S200, step S300, step S400, step S500, and step S600.
And S100, acquiring a pre-trained black box model, a sample set of a target task scene and a target sample to be interpreted.
It will be appreciated that the black box model that has been trained may be a classification model that, given inputs, is capable of outputting a probability distribution under each classification label. The sample set of the target task scene refers to that each sample can be used as a model input to enable a model to predict a result, has a certain data volume, can represent the data and feature distribution of the scene, and is used for helping people to know the distribution rule of the data and the feature in the scene. While the format of the particular sample to be interpreted is consistent with the format of the bulk of the data for the task scenario.
And S200, inputting the sample set into the black box model to obtain characteristic distribution information.
It can be understood that the feature distribution information includes a feature value corresponding to each dimension feature in the sample set and a proportion of the feature value in the sample set. According to a sample set of a target task scene, counting to obtain a distribution rule of each feature, wherein the obtained feature distribution is as follows:
D＝[D0,D1,……,DK-1]
wherein D isiIs the distribution of the ith dimension feature.
The ith dimension feature has LiA different value, Vi,jIs the j value of the i-dimension feature, Ci,jFor the j-th value of the ith dimension feature in the sample set,
it should be noted that if there is no specific data, but the distribution rule is known to be directly usable, the characteristic distribution format is the same as that described above.
And step S300, inputting the target sample into the black box model for model prediction to obtain a model prediction score.
It can be understood that, assuming that the target sample has K-dimensional features, the target sample F ═ F0,F1,……,FK-1]Wherein F isiAnd taking the value of the ith dimension characteristic of the sample F. Suppose that the current black box model predicts F as class b to obtain the model score S of the current samplenow，Snow＝Modelb(F)，Modelb(F) The score predicted on the b-th class for sample F for the model.
Step S400, traversing the features of each dimension in the target sample, and calculating the weighted average score corresponding to the features of each dimension according to the feature distribution information.
It will be appreciated that traversing each dimension feature FiFor each dimension feature, calculating the weighted average score of the feature value traversal under the dimension feature
Wherein,
is on a sample FiIs replaced by Vi,jAnd (4) scoring the model of the new sample, namely replacing the i-dimensional feature of the target sample with the jth value of the i-dimensional feature of the sample set, and Ci,jThe weight is the ratio of the jth value of the ith dimension characteristic of the sample set in the sample set.
Wherein, [ F ]0,……,Fi-1,Vi,j,Fi+1,……,FK-1]In F, F isiIs replaced by Vi,jThe latter new sample.
And step S500, determining the importance degree of each dimension characteristic in the target sample according to the difference value between the weighted average and the model prediction score.
It will be appreciated that the importance of each dimension feature in the target sample may be characterized by the magnitude of the feature's importance value. For example, F for each dimension featureiCalculating the importance value of the feature
Current value FiThe more important the target sample F, the SnowAnd
the greater the difference, i.e.
The larger, among others, there are a total of K-dimensional features.
And S600, interpreting the model of the black box model based on the target sample according to the importance degree of the features.
It is understood that the interpretation of the model includes local interpretations, which are characteristic interpretations of the target sample, and global interpretations, which are characteristic analyses of the black box model. For example, for local interpretability, several features with the highest important feature value of the target sample can be taken as the features for deciding that the target sample is classified into the category by the black box model. For global interpretability, the feature of the several dimensions with the highest importance eigenvalue is taken as the salient feature for the black box model.
It can be understood that the model interpretation method of the embodiment of the present application can be generally applied to model schemes such as a mainstream deep neural network model, an integrated tree model, and the like. The scheme has the following characteristics/advantages:
1. the universality is strong. The method is generally applied to mainstream model schemes such as a deep neural network model and an integrated tree model. The method does not need to pay attention to the implementation details of the model, and the importance degree of certain characteristics to the model is mainly analyzed by adjusting the input of the model and observing the change of the output of the model.
2. The method is locally interpretable. The method can explain the behavior of the model on a single case.
3. The method is fully interpretable. The method may account for the overall behavior of the model, such as analyzing salient features of the model.
Based on this, the model interpretation method in the embodiment of the application can regard the model as an invisible black box without paying attention to the implementation details of the model, and analyze the importance degree of a certain characteristic to the model by adjusting the input of the model and observing the change of the output of the model, thereby helping engineers interpret the behavior of the model in a specific sample to perform characteristic significance analysis, and being capable of interpreting the behavior of the model on a single case and the overall behavior of the model.
Referring to fig. 3, in an embodiment, step S400 includes, but is not limited to, step S410, step S420, and step S430.
Step S410, replacing each dimension feature in the target sample with a feature value corresponding to the feature of the same dimension in the sample set to obtain a new sample;
step S420, carrying out model scoring on the new sample through a black box model to obtain a model scoring score;
and step S430, obtaining a weighted average score corresponding to each dimension characteristic in the target sample according to the model score and the ratio of the characteristic value in the sample set.
It will be appreciated that traversing each dimension feature FiFor each dimension feature, calculating the weighted average score of the feature value traversal under the dimension feature
Wherein,
is on a sample FiIs replaced by Vi,jAnd (4) scoring the model of the new sample, namely replacing the i-dimensional feature of the target sample with the jth value of the i-dimensional feature of the sample set, and Ci,jThe weight is the ratio of the jth value of the ith dimension characteristic of the sample set in the sample set.
Wherein, [ F ]0,……,Fi-1,Vi,j,Fi+1,……,FK-1]In F, F isiIs replaced by Vi,jThe latter new sample.
Referring to FIG. 4, in an embodiment, step S500 includes, but is not limited to, step S510, step S520, and step S530.
Step S510, calculating the difference between the weighted average corresponding to each dimension characteristic and the model prediction score to obtain a first difference value;
step S520, calculating the difference between the weighted average corresponding to all the dimensional characteristics and the model prediction score to obtain a second difference value;
in step S530, a first feature importance level value is obtained by dividing the first difference value by the second difference value, wherein the first feature importance level value is used for representing the importance level of each dimension feature in the target sample, and the magnitude of the first feature importance level value is in direct proportion to the importance level.
It can be understood that the difference between the weighted average corresponding to each dimension feature and the model prediction score is calculated to obtain a first difference value
Calculating the difference between the weighted average corresponding to all the dimensional characteristics and the model prediction fraction to obtain a second difference value
The importance degree of each dimension feature in the target sample can be characterized by the importance degree value of the feature. For example, F for each dimension featureiCalculating the importance value of the feature
Current value FiThe more important the target sample F, the SnowAnd
the greater the difference, i.e.
The larger, among others, there are a total of K-dimensional features.
Referring to FIG. 5, in an embodiment, step S600 includes, but is not limited to, step S610, step S620, and step S630.
Step S610, sorting the features according to the magnitude of the first feature importance degree value;
step S620, determining the first N characteristics with the highest first characteristic importance degree value;
step S630, the black box model is locally interpretable on the model based on the target sample by using the first N features, where N is a positive integer greater than zero.
It is to be understood that the interpretation of the model includes local interpretations, which are characteristic interpretations of the target specimen. For example, for local interpretability, the N features with the highest significant feature value of the target sample may be taken. Determining characteristics of the target sample classified into the category by the black box modelIs composed of
And get
The N features with the highest value. That is, the important characteristics of the target sample are
The highest number of features.
Referring to fig. 6, in an embodiment, step S600 further includes, but is not limited to, step S640, step S650, step S660, step S670, and step S680.
Step S640, calculating a first feature importance degree value corresponding to the feature of each dimension of each sample in the sample set;
step S650, averaging the sum of the first feature importance degree values corresponding to the features of each dimension of each sample to obtain a second feature importance degree value, wherein the second feature importance degree value is used for representing the importance degree of the features of each dimension in the sample set, and the size of the second feature importance degree value is in direct proportion to the importance degree;
step S660, sorting each dimension feature in the sample set according to the size of the second feature importance degree value;
step S670, determining the characteristics of the first N dimensions with the highest second characteristic importance degree value;
step S680, globally interpretable of the black box model based on the model of the target sample using the features of the first N dimensions, where N is a positive integer greater than zero.
It is to be understood that the interpretation of the model includes global interpretations, wherein global interpretations are characteristic analyses of the black box model. For example, for global interpretability, the feature of the several dimensions with the highest importance feature value is taken as the salient feature for the black box model.
For example, for sample set E, E ═ { E ═ E0,E1,……,EM-1}，EiIs one sample in the set of samples.
Feature importance of jth dimension features of a model
Comprises the following steps:
wherein,
is the feature importance of the j-th dimension of the sample E _ i.
The model has the remarkable characteristics of
The highest few-dimensional features.
It can be understood that, in the premise of the sample set and the black box model, the feature distribution information of the sample set is firstly obtained through the sample set, wherein the feature distribution information includes each value of each dimension feature and the ratio of each value in the sample set, and then, aiming at the characteristics of a certain dimensionality of a target sample, replacing the characteristics of the dimensionality with each value of the characteristics of the same dimensionality of the characteristic distribution information of a sample set to obtain each replaced target sample, scoring each replaced target sample by using an existing model, and combining the occupation ratio of each value in the sample set to obtain the weighted average score of the characteristics of one dimensionality of the original target sample (the calculation method can be that the score and the occupation ratio are multiplied and then superposed), wherein the weighted average scores of the characteristics of other dimensionalities of the original target sample can also be obtained by the method; then, the original target sample is scored by using the existing model, the important index of one dimension characteristic of the original target sample can be obtained according to the obtained weighted average score and the model score of the original target sample (the calculation method can be that the absolute value of the difference between the weighted average score and the model score of the dimension characteristic is compared with the sum of the absolute values of the differences between the weighted average score and the model score of all the dimension characteristics), and so on, the important indexes aiming at other dimension characteristics of the original target sample can also be completed by using the method to obtain the important indexes of all the dimension characteristics of the original target sample, and the important characteristics of the target sample can be several dimension characteristics with the highest important indexes, so that the local interpretability of the model is realized.
In addition, by using the method for obtaining the important indexes of the one-dimensional features of the target samples, the important indexes of the one-dimensional features of all the samples in the sample set can be obtained, the important indexes of the one-dimensional features of the sample set can be obtained by summarizing the important indexes (the calculation method can be the sum of the important indexes of the one-dimensional features of all the samples is compared with the number of all the samples), and so on, the important indexes of the other dimensions of the sample set can be obtained by the method, the important indexes of the all-dimensional features of the original sample set can be obtained, the significant features of the sample set can be the several dimensional features with the highest important indexes, and the global interpretability of the model is realized.
In summary, according to the method, starting from the feature of one dimension, according to various values of the feature of one dimension of a sample set, feature replacement is performed on a target sample, then the target sample after feature replacement is scored, a weighted average score is obtained according to the score and the proportion of various values of the feature of one dimension of the sample set, and then an important index of the feature of the target sample is obtained according to the weighted average score and the model score of an original target sample (not subjected to feature replacement), so that local interpretability and global interpretability of the model are realized, and the problem of interpretability of the model, particularly the black box model, is solved.
Referring to fig. 7, an embodiment of the present invention also provides a model interpretation apparatus including:
an obtaining module 710, configured to obtain a pre-trained black box model, a sample set of a target task scene, and a target sample to be interpreted;
the characteristic analysis module 720 is used for inputting the sample set into the black box model to obtain characteristic distribution information;
the prediction module 730 is used for inputting the target sample into the black box model to perform model prediction to obtain a model prediction score;
the calculating module 740 is configured to traverse the features of each dimension in the target sample, and calculate a weighted average score corresponding to each dimension feature according to the feature distribution information;
the analysis module 750 is used for determining the importance degree of each dimension characteristic in the target sample according to the difference value between the weighted average and the model prediction score;
and the interpretation module 760 is used for interpreting the black box model based on the model of the target sample according to the importance degree of the features.
In an embodiment, the calculating module 740 is further configured to replace each dimension feature in the target sample with a feature value corresponding to a feature of the same dimension in the sample set, so as to obtain a new sample; carrying out model scoring on the new sample through a black box model to obtain a model scoring score; and obtaining a weighted average score corresponding to each dimension characteristic in the target sample according to the model scoring score and the ratio of the characteristic value in the sample set.
In an embodiment, the analysis module 750 is further configured to calculate a difference between the weighted average corresponding to each dimension feature and the model prediction score to obtain a first difference value; calculating the difference between the weighted average corresponding to all the dimensional characteristics and the model prediction fraction to obtain a second difference value; and dividing the first difference value by the second difference value to obtain a first feature importance degree value, wherein the first feature importance degree value is used for representing the importance degree of each dimension feature in the target sample, and the size of the first feature importance degree value is in direct proportion to the importance degree.
In one embodiment, the analysis module 760 is further configured to sort the features according to the magnitude of the first feature importance degree value; determining the first N characteristics with the highest first characteristic importance degree value; the black box model is locally interpretable on a target sample-based model using the first N features, where N is a positive integer greater than zero.
In an embodiment, the analysis module 760 is further configured to calculate a first feature importance degree value corresponding to the feature of each dimension of each sample in the sample set; averaging the sum of the first feature importance degree values corresponding to the features of each dimension of each sample to obtain a second feature importance degree value, wherein the second feature importance degree value is used for representing the importance degree of each dimension feature in the sample set, and the size of the second feature importance degree value is in direct proportion to the importance degree; sorting each dimension feature in the sample set according to the magnitude of the second feature importance degree value; determining the characteristics of the first N dimensions with the highest second characteristic importance degree value; globally interpretable of the black box model based on the model of the target sample using features of the first N dimensions, where N is a positive integer greater than zero.
It should be noted that, the technical means, the technical problems solved and the technical effects achieved in the embodiments of the model interpretation apparatus and the embodiments of the model interpretation method are the same, and detailed descriptions are not repeated here, and refer to the embodiments of the model interpretation method.
In addition, an embodiment of the present invention provides an electronic apparatus including: a memory, a processor, and an electronic device program stored on the memory and executable on the processor.
The processor and memory may be connected by a bus or other means.
It should be noted that the electronic device in this embodiment may be configured to include a memory and a processor as in the embodiment shown in fig. 1, and can form a part of the system architecture platform in the embodiment shown in fig. 1, and both are within the same inventive concept, so that both have the same implementation principle and beneficial effects, and are not described in detail herein.
The non-transitory software programs and instructions required to implement the model interpretation methods of the above embodiments are stored in the memory and, when executed by the processor, perform the model interpretation methods of the above embodiments, e.g., performing the above-described method steps S100 to S600 in fig. 2, method steps S410 to S430 in fig. 3, method steps S510 to S530 in fig. 4, method steps S610 to S630 in fig. 5, and method steps S640 to S680 in fig. 6.
The electronic device includes: radio Frequency (RF) circuit, memory, input unit, display unit, sensor, audio circuit, wireless fidelity (WiFi) module, processor, and power supply. Those skilled in the art will appreciate that the present embodiment is not limited solely to the structure of the electronic device, and may include more or fewer components than the present embodiment, or some components in combination, or a different arrangement of components.
The RF circuit can be used for receiving and transmitting signals in the process of information receiving and transmitting or conversation, and particularly, the downlink information of the base station is received and then is processed by the processor; in addition, the data for designing uplink is transmitted to the base station. In general, the RF circuit includes, but is not limited to, an antenna, at least one Amplifier, a transceiver, a coupler, a Low Noise Amplifier (LNA), a duplexer, and the like. In addition, the RF circuitry may communicate with networks and other devices via wireless communications. The wireless communication may use any communication standard or protocol, including but not limited to Global System for Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Message Service (SMS), and the like.
The memory may be used to store software programs and modules, and the processor may execute various functional applications of the electronic device and data processing by operating the software programs and modules stored in the memory. The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the electronic device, and the like. Further, the memory may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.
The input unit may be used to receive input numeric or character information and generate key signal inputs related to settings and function control of the electronic device. Specifically, the input unit may include a touch panel and other input devices. The touch panel, also called a touch screen, may collect touch operations thereon or nearby (such as operations on or near the touch panel using any suitable object or accessory, such as a finger, a stylus, etc.) and drive the corresponding connection device according to a preset program. Alternatively, the touch panel may include two parts, a touch detection device and a touch controller. The touch detection device detects a touch direction, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts the touch information into touch point coordinates, sends the touch point coordinates to the processor, and can receive and execute commands sent by the processor. In addition, the touch panel may be implemented by various types such as resistive, capacitive, infrared, and surface acoustic wave. The input unit may include other input devices in addition to the touch panel. In particular, other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.
The display unit may be used to display input information or provided information and various menus of the electronic device. The Display unit may include a Display panel, and optionally, the Display panel may be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like. Further, the touch panel may cover the display panel, and when the touch panel detects a touch operation thereon or nearby, the touch panel transmits the touch operation to the processor to determine a category of the touch event, and then the processor provides a corresponding visual output on the display panel according to the category of the touch event. Although the touch panel and the display panel are two separate components to implement the input and output functions of the electronic device, in some embodiments, the touch panel and the display panel may be integrated to implement the input and output functions of the electronic device.
The electronic device may also include at least one sensor, such as a light sensor, a motion sensor, and other sensors. In particular, the light sensor may include an ambient light sensor that may adjust the brightness of the display panel according to the brightness of ambient light, and a proximity sensor that may turn off the display panel and/or the backlight when the electronic device is moved to the ear. As one of the motion sensors, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally, three axes), detect the magnitude and direction of gravity when stationary, and can be used for applications of segmenting the posture of the electronic device (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration segmentation related functions (such as pedometer and tapping), and the like; as for other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, and an infrared sensor, which may be further configured to the electronic device, detailed descriptions thereof are omitted.
Audio circuitry, speakers, microphones may provide an audio interface. The audio circuit can transmit the electric signal converted from the received audio data to the loudspeaker, and the electric signal is converted into a sound signal by the loudspeaker to be output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit and converted into audio data, which is then output to the processor for processing, and then transmitted to, for example, another electronic device via the RF circuit, or the audio data is output to the memory for further processing.
WiFi belongs to short-distance wireless transmission technology, electronic equipment can receive and send electronic mails, browse webpages, access streaming media and the like through a WiFi module, and wireless broadband internet access is provided. The WiFi module does not belong to the essential constitution of the electronic device and can be omitted entirely as needed within the scope not changing the essence of the invention.
The processor is a control center of the electronic equipment, connects various parts of the whole electronic equipment by various interfaces and lines, and executes various functions and processes data of the electronic equipment by operating or executing software programs and/or modules stored in the memory and calling the data stored in the memory, thereby integrally monitoring the electronic equipment. Alternatively, the processor may include one or more processing units; preferably, the processor may integrate an application processor and a modem processor, wherein the application processor mainly handles operating systems, operating interfaces, application programs, and the like, and the modem processor mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor.
The electronic device also includes a power supply (e.g., a battery) for powering the various components, and preferably, the power supply may be logically coupled to the processor via a power management system, such that functions of managing charging, discharging, and power consumption are performed via the power management system.
Although not shown, the electronic device may further include a camera, a bluetooth module, and the like, which are not described in detail herein.
In the present embodiment, the terminal device includes a processor capable of executing the pedestrian re-segmentation method of the previous embodiment.
Furthermore, an embodiment of the present invention also provides a computer-readable storage medium storing computer-executable instructions for performing the above-described terminal-side model interpretation method, for example, performing the above-described method steps S100 to S600 in fig. 2, method steps S410 to S430 in fig. 3, method steps S510 to S530 in fig. 4, method steps S610 to S630 in fig. 5, and method steps S640 to S680 in fig. 6.
It will be understood by those of ordinary skill in the art that all or some of the steps, systems, and methods disclosed above may be implemented as software, firmware, hardware, or suitable combinations thereof. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as is well known to those of ordinary skill in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by a computer. In addition, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media as known to those skilled in the art.
While the preferred embodiments of the present invention have been described in detail, it will be understood by those skilled in the art that the foregoing and various other changes, omissions and deviations in the form and detail thereof may be made without departing from the scope of this invention.
Claims (10)
1. A method of model interpretation, the method comprising:
acquiring a pre-trained black box model, a sample set of a target task scene and a target sample to be interpreted;
inputting the sample set into the black box model to obtain characteristic distribution information;
inputting the target sample into the black box model for model prediction to obtain a model prediction score;
traversing the feature of each dimension in the target sample, and calculating a weighted average score corresponding to the feature of each dimension according to the feature distribution information;
determining the importance degree of the feature of each dimension in the target sample according to the difference value between the weighted average and the model prediction score;
and interpreting the black box model based on the model of the target sample according to the importance degree of the characteristics.
2. The method according to claim 1, wherein the feature distribution information includes feature values corresponding to the features of each dimension in the sample set and a ratio of the feature values in the sample set.
3. The method according to claim 2, wherein the calculating a weighted average score corresponding to the feature in each dimension according to the feature distribution information comprises:
replacing the features of each dimension in the target sample with feature values corresponding to the features of the same dimension in the sample set to obtain a new sample;
carrying out model scoring on the new sample through the black box model to obtain a model scoring score;
and obtaining a weighted average score corresponding to the feature of each dimension in the target sample according to the model score and the proportion of the feature value in the sample set.
4. The method of claim 1, wherein determining the importance of the feature in each dimension in the target sample according to the difference between the weighted average and the model prediction score comprises:
calculating the difference between the weighted average corresponding to the features of each dimension and the model prediction score to obtain a first difference value;
calculating the difference between the weighted average corresponding to the features in all dimensions and the model prediction score to obtain a second difference value;
and dividing the first difference value by the second difference value to obtain a first feature importance degree value, wherein the first feature importance degree value is used for representing the importance degree of the feature in each dimension in the target sample, and the size of the first feature importance degree value is in direct proportion to the importance degree.
5. The method according to claim 4, wherein said interpreting the black box model based on the model of the target sample according to the degree of importance comprises:
sorting the features according to the magnitude of the first feature importance degree value;
determining the first N characteristics with the highest importance degree value of the first characteristics;
locally interpretable of the black box model based on the model of the target sample using the first N of the features, where N is a positive integer greater than zero.
6. The method according to claim 4, wherein said interpreting the black box model based on the model of the target sample according to the degree of importance comprises:
calculating the first feature importance degree value corresponding to the feature of each dimension of each sample in the sample set;
averaging the sum of the first feature importance degree values corresponding to the features of each dimension of each sample to obtain a second feature importance degree value, wherein the second feature importance degree value is used for representing the importance degree of the features of each dimension in the sample set, and the size of the second feature importance degree value is in direct proportion to the importance degree;
sorting the features of each dimension in the sample set according to the magnitude of the second feature importance degree value;
determining the characteristics of the first N dimensions with the highest second characteristic importance degree values;
globally interpretable of the black-box model based on the model of the target sample using the features of the first N dimensions, where N is a positive integer greater than zero.
7. The method according to any one of claims 1 to 6, wherein the black box model is a classification model.
8. A model interpretation apparatus, characterized in that the apparatus comprises:
the acquisition module is used for acquiring a pre-trained black box model, a sample set of a target task scene and a target sample to be explained;
the characteristic analysis module is used for inputting the sample set into the black box model to obtain characteristic distribution information;
the prediction module is used for inputting the target sample into the black box model to perform model prediction to obtain a model prediction score;
the calculation module is used for traversing the characteristics of each dimension in the target sample and calculating the weighted average score corresponding to the characteristics of each dimension according to the characteristic distribution information;
the analysis module is used for determining the importance degree of the feature of each dimension in the target sample according to the difference value between the weighted average and the model prediction score;
and the interpretation module is used for interpreting the black box model based on the model of the target sample according to the importance degree of the characteristics.
9. An electronic device, characterized in that the electronic device comprises a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for enabling connection communication between the processor and the memory, the program, when executed by the processor, implementing the steps of the model interpretation method of any of claims 1 to 7.
10. A storage medium, which is a computer-readable storage medium for computer-readable storage, characterized in that the storage medium stores one or more programs, which are executable by one or more processors to implement the steps of the model interpretation method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210282784.6A CN114625657A (en) | 2022-03-22 | 2022-03-22 | Model interpretation method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210282784.6A CN114625657A (en) | 2022-03-22 | 2022-03-22 | Model interpretation method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114625657A true CN114625657A (en) | 2022-06-14 |
Family
ID=81903763
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210282784.6A Pending CN114625657A (en) | 2022-03-22 | 2022-03-22 | Model interpretation method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114625657A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024119865A1 (en) * | 2022-12-06 | 2024-06-13 | 北京京东尚科信息技术有限公司 | Prediction information generation method and apparatus, and device, medium and program product |
-
2022
- 2022-03-22 CN CN202210282784.6A patent/CN114625657A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024119865A1 (en) * | 2022-12-06 | 2024-06-13 | 北京京东尚科信息技术有限公司 | Prediction information generation method and apparatus, and device, medium and program product |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111553162B (en) | Intention recognition method and related device | |
| CN109033156B (en) | Information processing method and device and terminal | |
| TW201543239A (en) | Method and apparatus for grouping contacts | |
| CN110852109A (en) | Corpus generating method, corpus generating device, and storage medium | |
| CN111597804B (en) | Method and related device for training entity recognition model | |
| CN113821589B (en) | Text label determining method and device, computer equipment and storage medium | |
| WO2021147421A1 (en) | Automatic question answering method and apparatus for man-machine interaction, and intelligent device | |
| CN113723378B (en) | Model training method and device, computer equipment and storage medium | |
| CN111612178A (en) | Model diagnosis method and related equipment | |
| CN113868427A (en) | Data processing method and device and electronic equipment | |
| CN112749252A (en) | Text matching method based on artificial intelligence and related device | |
| CN114117056B (en) | Training data processing method and device and storage medium | |
| CN114625657A (en) | Model interpretation method and device, electronic equipment and storage medium | |
| CN116975295B (en) | Text classification method and device and related products | |
| CN113269279A (en) | Multimedia content classification method and related device | |
| CN113704008A (en) | Anomaly detection method, problem diagnosis method and related products | |
| CN112232048B (en) | Form processing method based on neural network and related device | |
| CN115269895A (en) | Question answering method and related device | |
| CN114970562A (en) | Semantic understanding method, device, medium and equipment | |
| CN114722234A (en) | Music recommendation method, device and storage medium based on artificial intelligence | |
| CN116259083A (en) | Image quality recognition model determining method and related device | |
| CN113822435A (en) | User conversion rate prediction method and related equipment | |
| CN114743081B (en) | Model training method, related device and storage medium | |
| CN116580268B (en) | Training method of image target positioning model, image processing method and related products | |
| CN118427350A (en) | Text emotion analysis method and device, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |