CN113205176B

CN113205176B - Method, device and equipment for training defect classification detection model and storage medium

Info

Publication number: CN113205176B
Application number: CN202110420016.8A
Authority: CN
Inventors: 高航; 杜松
Original assignee: Thundercomm Technology Co ltd
Current assignee: Thundercomm Technology Co ltd
Priority date: 2021-04-19
Filing date: 2021-04-19
Publication date: 2022-09-06
Anticipated expiration: 2041-04-19
Also published as: CN113205176A

Abstract

The embodiment of the invention provides a method, a device, equipment and a storage medium for training a defect classification detection model. The method comprises the following steps: acquiring a sample image; inputting a sample image into a preset defect grading detection model; carrying out defect classification detection on the sample image by utilizing a classification branch network and a classification branch network in the defect classification detection model to obtain predicted values of the category, the position and the grade of the defect in the sample image; processing the sample image by utilizing a clustering branch network in the defect grading detection model to obtain a pseudo label of the defect level in the sample image, and taking the pseudo label as a true value of the defect level in the sample image; and training a preset defect grading detection model according to the predicted values and the true values of the defect types, positions and levels to obtain a target defect grading detection model. According to the embodiment of the invention, a model capable of detecting the defect level can be obtained, and the defect level does not need to be marked in the training process.

Description

Method, device and equipment for training defect classification detection model and storage medium

Technical Field

The invention belongs to the technical field of industrial surface defect detection, and particularly relates to a method and a device for training a defect grading detection model, a defect grading detection device and method, computing equipment and a computer storage medium.

Background

With the development and application of neural network technology, industrial surface defect detection has started to perform defect localization and classification by using a trained detection model, and a good effect is achieved. However, for industrial surface defect detection, attention is paid not only to the type and location of the defect, but also to the grade of the defect. The current detection model cannot well complete the detection of the defect level.

Disclosure of Invention

The embodiment of the invention provides a method and a device for training a defect grading detection model, a defect grading detection method and a device, a computing device and a computer storage medium, which can obtain a model capable of detecting defect levels, do not need to label the defect levels in the training process, and overcome the defect points with large workload and low accuracy of manual labeling.

In a first aspect, an embodiment of the present invention provides a method for training a defect classification detection model, where the method includes:

acquiring a sample image, wherein the sample image comprises marking information of defect positions and types;

inputting the sample image into a preset defect grading detection model, wherein the defect grading detection model comprises a classification branch network for acquiring the category and the position of the defect, a classification branch network for acquiring the grade of the defect, and a clustering branch network for generating a defect grade pseudo label;

carrying out defect grading detection on the sample image by utilizing a classification branch network and a classification branch network in the defect grading detection model to obtain a prediction type, a prediction position and a prediction grade of the defect in the sample image;

processing the sample image by utilizing a clustering branch network in the defect grading detection model to obtain a pseudo label of the defect level in the sample image, and taking the pseudo label as a true value of the defect level in the sample image;

and training the preset defect grading detection model according to the prediction type, the prediction position and the prediction grade of the defect of the sample image, the labeling information of the defect position and type in the sample image and the pseudo label of the defect grade in the sample image to obtain a target defect grading detection model.

In a second aspect, an embodiment of the present invention provides a method for detecting a trapping stage, where the method includes:

acquiring a target image to be subjected to defect detection;

carrying out defect grading detection on the target image by using a target defect grading detection model trained by the method of the first aspect to obtain a prediction type, a prediction position and a prediction grade of a defect in the target image;

and determining the defect information of the target image according to the prediction type and the prediction position of the defect in the target image and the prediction level.

In a third aspect, an embodiment of the present invention provides an apparatus for training a defect classification detection model, where the apparatus includes:

the system comprises a sample acquisition module, a defect detection module and a defect detection module, wherein the sample acquisition module is used for acquiring a sample image which comprises marking information of defect positions and types;

the input module is used for inputting the sample image into a preset defect grading detection model, and the defect grading detection model comprises a classification branch network for acquiring the category and the position of the defect, a grading branch network for acquiring the grade of the defect and a clustering branch network for generating a defect grade pseudo label;

the first prediction module is used for carrying out defect grading detection on the sample image by utilizing a classification branch network and a grading branch network in the defect grading detection model to obtain the prediction category, the prediction position and the prediction grade of the defect in the sample image;

the second prediction module is used for processing the sample image by utilizing a clustering branch network in the defect hierarchical detection model to obtain a pseudo label of the defect level in the sample image, and taking the pseudo label as a true value of the defect level in the sample image;

and the training module is used for training the preset defect grading detection model according to the prediction type, the prediction position and the prediction grade of the defect of the sample image, the labeling information of the defect position and type in the sample image and the pseudo label of the defect grade in the sample image to obtain a target defect grading detection model.

In a fourth aspect, an embodiment of the present invention provides a defect classification detection apparatus, including:

the image acquisition module is used for acquiring a target image to be subjected to defect detection;

the prediction module is used for carrying out defect grading detection on the target image by using a target defect grading detection model trained by the device in the third aspect to obtain the prediction type, the prediction position and the prediction grade of the defects in the target image;

and the determining module is used for determining the defect information of the target image according to the prediction type and the prediction position of the defect in the target image and the prediction level.

In a fifth aspect, an embodiment of the present invention provides a computing device, including: a processor, and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the method for training a defect classification detection model according to the first aspect, or the detection method according to the second aspect.

In a sixth aspect, an embodiment of the present invention provides a computer storage medium, on which computer program instructions are stored, and when executed by a processor, the computer program instructions implement the method for training a defect classification detection model according to the first aspect, or the detection method according to the second aspect.

According to the method and the device for training the defect classification detection model, the defect classification detection module and the device, the computing equipment and the computer storage medium, the defect classification detection branch network is added in the original model capable of realizing defect classification and position detection, so that for a detection image, after a feature map is obtained, the feature map can be directly used for predicting the classification, the position and the grade of the defect, and the defect grade is obtained without performing additional analysis after the classification and the position of the defect are predicted. In the training process, a thermodynamic diagram of the defect is obtained according to the feature diagram and the predicted defect type and position through the clustering branch network, so that feature vectors related to the defect grade are obtained, a clustering method is adopted to generate a defect grade label (pseudo label) as a true value of the defect grade, and then the newly-added defect grading branch network is trained by using the pseudo label of the defect grade, so that the grade of the defect is not required to be labeled in a sample image, the defects of high manual labeling cost and great main effect influence are effectively overcome, and the problem of grading information sources is solved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the embodiments of the present invention will be briefly described below, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

FIG. 1 is a schematic diagram of a structure of an inspection model used in a defect classification inspection method;

FIG. 2 is a schematic diagram of another inspection model used in a defect classification inspection method;

FIG. 3 is a flowchart illustrating a method for training a defect classification detection model according to an embodiment of the present invention;

FIG. 4 is a schematic structural diagram of a default defect classification detection model according to an embodiment of the present invention;

FIG. 5 is a block diagram of an exemplary defect classification detection model shown in FIG. 4;

FIG. 6 is a schematic structural diagram of a clustering branch network according to an embodiment of the present invention;

FIG. 7 is a flowchart illustrating a method for generating a pseudo tag at a defect level according to an embodiment of the present invention;

FIG. 8 is a flowchart illustrating a defect classification detection method according to an embodiment of the present invention;

FIG. 9 is a schematic structural diagram of an apparatus for training a defect classification detection model according to an embodiment of the present invention;

FIG. 10 is a schematic structural diagram of a defect classification detection apparatus provided in an embodiment of the present invention;

fig. 11 is a schematic structural diagram of a computing device according to an embodiment of the present invention.

Detailed Description

Features and exemplary embodiments of various aspects of the present invention will be described in detail below, and in order to make objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting. It will be apparent to one skilled in the art that the present invention may be practiced without some of these specific details. The following description of the embodiments is merely intended to provide a better understanding of the present invention by illustrating examples of the present invention.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

Description of related terms:

Faster R-CNN

a common two-stage (two-stage) target detection model. The method is an R-CNN (Region CNN) family third generation model, and integrates four basic steps (candidate Region generation, feature extraction, classification and position refinement) of target detection into a deep network framework. The model can be divided into 4 main parts: background (CNN backbone Networks, such as VGG, ResNet, DenseNet, etc., for extracting feature maps), RPN (Region pro-temporal Networks, for generating Region suggestions (also referred to as candidate boxes), ROI posing (Region of interest Pooling for synthesizing feature maps and Region suggestion Networks to extract the feature maps), and Classification (Classification, head of model, calculating suggested classes, refining location).

CNN (Convolutional Neural Networks)

One class of feed-forward neural networks, which includes convolution calculations and has a deep structure, is one of the representative algorithms for deep learning. On one hand, the method reduces the number of parameters needing training of the neural network through the sharing of receptive fields and weights, and on the other hand, the method obtains the mapping of different levels of features of the image through layer-by-layer convolution. The output of each convolutional layer is called a feature map (feature maps), and typically deeper convolutional layers contain richer spatial and semantic information.

Class Activation Mapping (CAM)

A web interpretable problem-related technique by which a generated thermodynamic map (heatmap) well visualizes the basis of model classification decisions. The general CAM utilizes the feature map output by the last convolutional layer, connects with the fully connected layer of softmax through GAP (Global Average Pooling), and trains the weight of the feature map and the category; and weighting and summing the feature maps to generate a thermodynamic map, and upsampling and adding the thermodynamic map to the original map when visualization is needed. The later-appearing Grad-CAM (Gradient-weighted Class Activation Mapping) calculates the weight by using the global average of the Gradient, and the training process is omitted; and performing the ReLU operation on the weighted sum solves the problem that the deconvolution and directed back propagation classes are not sensitive.

For better understanding of the present invention, the present defect classification detection method will be described with reference to fig. 1 and 2.

FIG. 1 is a schematic structural diagram of an inspection model used in a defect classification inspection method.

As shown in fig. 1, the defect detection model includes a feature extraction network 10, a region suggestion network 11, a region of interest pooling network 12, a classification network 13, and a positioning network 14, and can realize the positioning and classification of defects. After the positioning and classification are completed, the inferred result is input into other classifiers 20 to judge the defect level, the classification confidence level (confidence) is used for thresholding for judgment, and the complicated feature map which possibly traces the suggested frame (or candidate frame) where the defect is located is mapped back to the original image and is identified by using a CV method.

In the defect classification detection method shown in fig. 1, the defect detection model does not undertake the classification task of the defects, provides at most the classification confidence level, feature map and original image of the defects, and determines the grade of the defects by other classifiers outside the model. There are two main problems with this solution:

firstly, classification confidence degrees are simply used for classification, classification is unreliable, the confidence degree is possibly very high, but the classification is not high, overdetection is easy, the confidence degree of some defects is not high, the characteristics are very obvious, and omission is easy.

Secondly, the traditional characteristic engineering is used as a classifier, and firstly, the design burden is large; secondly, the selection of the characteristics depends on the data distribution of the initial sample, and the scheme has poor adaptability; sometimes the required features are in the feature map of the neural network, and there is also a problem of repeated feature extraction. Furthermore, the post-inference recognition thus performed naturally requires more running time.

FIG. 2 is a schematic diagram of another conventional defect classification detection method using an inspection model

As shown in fig. 2, a classification network 15 is added to the head of the defect detection model, and is trained by using manual labels (labels of defect grades are added in the training process, that is, labels contain three description information of the position, category and grade of the defect), and the training model learns the classification capability, and directly obtains the classification result of the defect during inference. The problem with this example of a scheme is mainly the need for manual annotation of rating information. The existing target detection marking data only mark the position and the category and have no additional grade information. Firstly, the characteristic differences of different types of defects are easily distinguished manually, and then the grade differences are difficult to distinguish from the same defect characteristics, errors caused by subjective factors are larger, and the influence cannot be ignored; secondly, more annotations mean more labor and time costs are paid, and even if new annotations are added to existing annotations, the cost may be as low as the cost of the whole re-annotation.

In summary, the current defect detection of industrial images mainly faces the following problems:

firstly, in the current multi-class defect detection, the positions and classes of defects are manually marked by using a training set, and the trained model can be used for positioning and classifying the defects but cannot be used for classifying the defects. Grading is a concern for industrial activities and one solution is to use whether the confidence of the defect classification (confidence) meets a threshold to determine whether it is a serious defect, but this is not reliable. Another approach is to use post-inference data to achieve ranking through feature engineering, but is time consuming and poorly adaptable.

Secondly, marking the defects is very labor-consuming, time-consuming and subjective. The conventional situation of marking only the position and the type has non-negligible subjective factors, and when the grade of the defect is additionally marked, the grade characteristic of the same defect needs to be manually identified, which is more difficult to grasp than the classification characteristic of different defects.

Thirdly, some quantization indexes are frequently used in industry to guide defect grading, for example, a foreground background gray level difference threshold value, in non-example segmented target detection, a relatively accurate foreground region is difficult to obtain, and the quantization indexes are difficult to utilize.

In order to at least partially solve the above technical problem, embodiments of the present invention provide a method and an apparatus for training a defect classification detection model, a defect classification detection apparatus, a computing device, and a computer storage medium, which obtain a thermodynamic diagram of a defect according to a feature map and predicted defect types and positions by adding a clustering branch network, further obtain a feature vector related to a defect level, generate a defect level label (pseudo label) by using a clustering method as a true value of the defect level, and train a newly added defect classification branch network by using the pseudo label of the defect level, so that the defect level does not need to be labeled in a sample image.

First, a method for training a defect classification detection model provided in the embodiment of the present invention is described below.

Fig. 3 is a flowchart illustrating a method for training a defect classification detection model according to an embodiment of the present invention.

An embodiment of the present invention provides a method 100 for training a defect classification detection model, where the defect classification detection model is used to detect the type, position, and level of a defect from an industrial defect recognition image, as shown in fig. 3, the method 100 may include the following steps:

s101, obtaining a sample image, wherein the sample image comprises marking information of defect positions and types.

The sample image can be used as a true value of the defect position and the defect type in training by manually marking the defect position and the defect type.

S102, inputting the sample image into a preset defect grading detection model, wherein the preset defect grading detection model comprises a classification branch network for acquiring the category and the position of the defect, a grading branch network for acquiring the grade of the defect, and a clustering branch network for generating a defect grade pseudo label.

After the sample image is acquired in S101, the sample image is input to a preset defect classification detection model. In the embodiment of the invention, the defect grading detection model comprises a classification branch network for acquiring the category and the position of the defect, a grading branch network for acquiring the grade of the defect, and a clustering branch network for generating the defect grade pseudo label. The structure of the preset defect classification detection model is shown with reference to fig. 4 and 5, and will be further described later.

S103, carrying out defect grading detection on the sample image by using the classification branch network and the classification branch network in the defect grading detection model to obtain the prediction type, the prediction position and the prediction grade of the defect in the sample image.

After the sample image is input into the preset defect grading detection model in S102, the classification branch network and the classification branch network in the defect grading detection model are used to perform defect grading detection on the sample image, so as to obtain the prediction category, the prediction position, and the prediction level of the defect in the sample image.

And S104, processing the sample image by using a clustering branch network in the defect classification detection model to obtain a pseudo label of the defect level in the sample image, and taking the pseudo label as a true value of the defect level in the sample image.

After the sample image is input into the preset defect classification detection model in S102, the sample image is processed by using the clustering branch network in the defect classification detection model to obtain the pseudo label of the defect level in the sample image, and the pseudo label is used as the true value of the defect level in the sample image.

It should be understood that steps S103 and S104 may be performed synchronously or sequentially, and it is not limited herein that S103 is performed first and then S104 is performed.

S105, training the preset defect grading detection model according to the prediction type, the prediction position and the prediction grade of the defect of the sample image, the labeling information of the defect position and type in the sample image and the pseudo label of the defect grade in the sample image to obtain a target defect grading detection model.

And after the prediction type, the prediction position, the prediction level and the pseudo label of the defect level of the defect are obtained, training the preset defect grading detection model according to the prediction type, the prediction position, the prediction level and the pseudo label of the defect level, and the labeling information of the defect type and the defect level of the defect, so as to obtain the target defect grading detection model.

According to the method for training the defect classification detection model, a defect classification detection branch network and a clustering branch network are added in an original model capable of realizing defect classification and position detection, in the training process, a thermodynamic diagram of the defect is obtained through the clustering branch network according to a feature diagram and the predicted defect classification and position, so that a feature vector related to the defect grade is obtained, a clustering method is adopted to generate a defect grade label (pseudo label) as a true value of the defect grade, and then the newly added defect classification branch network is trained by using the pseudo label of the defect grade, so that the defect grade does not need to be labeled in a sample image, the defects of high manual labeling cost and great influence of the sample image are effectively overcome, and the problem of classification information sources is solved.

The detailed process of the method 100 is described below.

In S101, a sample image is acquired, and the sample image includes annotation information of a defect position and a category.

The sample image can be obtained by collecting various defective industrial products, and after the sample image is obtained, the positions and the types of the defects in the sample image are labeled in a manual labeling or machine labeling mode, so that the sample image can comprise labeling information of the positions and the types of the defects. Therefore, in the training process, the defect type and the position of the defect predicted by the model can be compared, so that the model parameters are adjusted to obtain the target defect detection model.

It should be noted that in the embodiment of the present invention, only information of defect positions and categories need to be labeled in the sample image, and information of defect levels/levels does not need to be labeled, which is a problem that manual labeling of defect levels is laborious and accuracy is difficult to guarantee. As to the information on how the defect level/grade does not need to be labeled, it will be described later.

In S102, the sample image is input into a preset defect classification detection model, where the preset defect classification detection model includes a classification branch network for obtaining the category and the position of the defect, a classification branch network for obtaining the level of the defect, and a clustering branch network for generating a defect level pseudo tag.

In the embodiment of the invention, the preset defect detection model can predict the type and the position of the defect, can also predict the level of the defect and generates the pseudo label of the defect level. While the default defect classification detection model may employ various suitable network architectures, in the present embodiment, an exemplary architecture diagram is provided in fig. 4 and 5 to facilitate understanding of the present invention.

Fig. 4 is a schematic structural diagram of a preset defect classification detection model according to an embodiment of the present invention.

As shown in fig. 4, the predetermined defect classification detection model 200 provided by the embodiment of the present invention includes a classification branch network 110, a clustering branch network 120, and a classification branch network 130. The classification branching network 110 is used to obtain a classification branching network of the category and the location of the defect. The clustering branching network 120 is used to generate defect level pseudo labels. The hierarchical branching network 130 is used to obtain the level of the defect. The classification branch network 110 may employ various detection networks that may perform object localization and classification. As one example, the classification branching network 110 includes a master-RCNN network. The clustering branch network 120 may employ various feature cluster generators of suitable structure, and the embodiments of the present invention described hereinafter provide an exemplary feature cluster generator to explain the principles of the present invention.

Fig. 5 is a diagram showing an exemplary structure of a defect classification detection model shown in fig. 4.

As shown in fig. 5, the preset defect classification detection model 200 provided by the embodiment of the present invention includes a feature extraction network 10, a classification branch network 110, a clustering branch network 120, and a classification branch network 130. The classification branching network 110 includes an area proposal network 11, an area-of-interest pooling network 12, a classification network 13, and a positioning network 14. The feature extraction network 10 is used for extracting a feature map of the image, and the area suggestion network is used for obtaining suggested candidate frames according to the feature map. The region of interest pooling network 12 is used to derive a proposed feature map from the features and the candidate boxes. The classification network 13 is used to derive a prediction class of defects (or to derive a score for defects belonging to various classes) from the proposed feature map. The positioning network 14 is used for obtaining the predicted position of the defect according to the characteristic diagram. The hierarchical branching network 130 is used to derive a predicted level of a defect from the proposed feature map. The clustering branch network 120 is configured to obtain a classification feature set of the defect according to the feature map and the prediction category and the prediction position of the defect, and further obtain a pseudo label of the defect level.

The default defect classification detection model 200 shown in fig. 5 can be regarded as being obtained by adding a clustering branch network and a classification branch network in the master-RCNN network. Of course, the preset defect classification detection model 200 according to the embodiment of the present invention may also be obtained by adding a clustering branch network and a classification branch network on the basis of other target positioning and classification models.

In S103, the classification branch network and the classification branch network in the defect classification detection model are used to perform defect classification detection on the sample image, so as to obtain a prediction type, a prediction position, and a prediction level of a defect in the sample image.

Illustratively, in the embodiment of the present invention, the performing defect classification detection on the sample image by using the defect classification detection model to obtain the prediction category, the prediction position and the prediction level value of the defect in the sample image includes the following steps:

first, a feature map of the sample image is acquired, for example, the feature map of the sample image is extracted by the feature extraction network 10.

And then, processing the feature map by using the classification branch network to obtain the prediction type and the prediction position of the defect in the sample image. An example structure of the classification branching network may be as shown with reference to fig. 5.

And then, processing the characteristic graph by using the hierarchical branch network to obtain the predicted value of the defect level in the sample image.

It should be understood that the step of obtaining the predicted location and the predicted category of the defect and the step of obtaining the predicted level of the defect may be performed simultaneously or sequentially.

In S104, the sample image is processed by using the clustering branch network in the defect classification detection model to obtain a pseudo label of the defect level in the sample image, and the pseudo label is used as a true value of the defect level in the sample image.

In the embodiment of the invention, because the clustering branch network in the defect hierarchical detection model is used for processing the sample image to obtain the pseudo label of the defect level in the sample image, and the label is used as the true value of the defect level in the sample image, the defect level can be known through the pseudo label, so that the network can be adjusted according to the predicted level of the defect and the pseudo label in the training process to finally obtain the target defect detection network. And because the false label is used as the true value of the defect level in the sample image, the information of the defect level does not need to be marked in the sample image, so that the workload of marking the sample image is reduced, the problems that manual marking of the defect level is time-consuming and labor-consuming and is influenced by great observation and has lower accuracy are solved, and the problem of defect classification information source in the training of a defect classification detection model is solved.

The structure of the clustering branch network and the process of obtaining the pseudo label by the clustering branch network are described below with reference to fig. 6 and 7.

Fig. 6 is a schematic structural diagram of a clustering branch network according to an embodiment of the present invention.

As shown in fig. 6, clustering branch network 120 includes activation-like mapping subnetwork 121, feature extraction subnetwork 122, feature set subnetwork 123, and clustering subnetwork 124. The class activation mapping sub-network 121 is used to generate a thermodynamic diagram of the defect using a class activation mapping (e.g., CAM or Grad-CAM). Feature extraction subnetwork 122 is used to extract hierarchical feature vectors from the thermodynamic diagram. The feature set subnetwork 123 is used for classifying the obtained grading feature vectors according to defect categories, and classifying the grading feature vectors of the same defect into a group to obtain a grading feature set of each defect. The clustering subnetwork 124 is used for clustering the hierarchical feature set of each defect, and then marking a pseudo label of the defect level for each hierarchical feature vector group obtained by clustering to represent the grade of the defect.

Fig. 7 is a flowchart illustrating a method 200 for generating a pseudo tag at a defect level according to an embodiment of the present invention.

As shown in fig. 7, in the embodiment of the present invention, a method 200 for generating a pseudo tag at a defect level includes the steps of:

s201, utilizing the class activation mapping sub-network to obtain a thermodynamic diagram of each class of defects according to the feature diagram and the prediction classes and prediction positions of the defects in the sample image.

In the embodiment of the invention, the class activation mapping sub-network adopts Grad-CAM algorithm, and the thermodynamic diagram calculation method comprises the following steps:

the weight of the kth feature map for class c is

Can be calculated by the following formula:

wherein Z is the number of pixels of the feature map, y ^c Is the score for the corresponding category c (is the value before entering the softmax layer),

the pixel value at the (i, j) position in the kth feature map is represented.

After the weights of the class c to all the feature maps are obtained, the thermodynamic diagram can be obtained by obtaining the weighted sum of the class c and the feature maps:

and S202, utilizing the feature extraction sub-network to obtain a defect grading feature vector of each type of defect according to the thermodynamic diagram.

Since the result of clustering based directly on thermodynamic diagrams is unpredictable, it is likely that the distribution of the original confidence is reflected. Therefore, there is a need to enhance defect classification features and extract feature vectors using feature extraction sub-networks.

In the embodiment of the invention, different strengthening methods can be adopted to extract the grading characteristic vector of the defect according to different service scenes. Exemplary, e.g., product defects:

i) if the severity is related to the area of the foreground region, performing threshold segmentation (for example, using Otsu method) on the thermodynamic diagram during extracting the hierarchical feature vectors, and counting the area of the foreground;

ii) calculating the filling degree of the foreground binary image when extracting the hierarchical feature vector if the filling degree is related to the concentration degree of the foreground;

iii) if the defect severity is uniformly related to the characteristic region, counting the uniformity of the foreground region when extracting the grading characteristic vector;

iv) if the contrast differences with the foreground background are relevant, calculating their contrast constast when extracting the hierarchical feature vectors;

v) if it is related to the intensity of the defect core area, counting the proportion of the highlight area when extracting the hierarchical feature vector.

Illustratively, in this embodiment of the present invention, the obtaining a defect-level feature vector of each type of defect according to the thermodynamic diagram by using the feature extraction sub-network includes:

obtaining the thermodynamic diagram foreground area size and thermodynamic diagram foreground two of the thermodynamic diagramFilling degree of the value map, uniformity of a foreground region of the thermodynamic diagram, contrast of the foreground and the background of the thermodynamic diagram, and occupation ratio of a highlight region of the thermodynamic diagram. The extracted information together form a hierarchical feature vector v ═ (v ═ v) ₁ ,v ₂ ,...,v _m ). It should be understood that the feature vector is not limited to the features in the examples, but may include other features as well.

And S203, storing the defect grading feature vector into the grading feature set of the corresponding category of defects according to the prediction of the defects by using the feature set sub-network.

It should be noted that, in the feature set, the dimensions of the features are different from one feature to another, and normalization is required to eliminate the dimension difference. In addition, each feature contributes differently to the defect level, and a weight vector may be used

Weighted combination, in which a negative weighting coefficient indicates that the feature is negatively correlated with rank, the resulting feature vector being

In the embodiment of the present invention, the weighting coefficients may be preset empirically.

It should be further noted that, in the embodiment of the present invention, a maximum feature number of the feature set may be set, and if the hierarchical feature vectors in the feature set reach the maximum number, the hierarchical feature vectors that are added at the beginning are automatically discarded.

And S204, clustering the classification feature set of each category of defects by using the clustering subnetwork to obtain the pseudo labels of the defect levels of each category of defects, wherein the pseudo labels are used for representing the defect levels.

Exemplarily, in the embodiment of the present invention, the clustering sub-network is used to cluster the classification feature set of each category of defects to obtain the pseudo label of the defect level of each category of defects, where the pseudo label is used to indicate the level of the defect, and the method includes the following steps:

firstly, clustering the classification feature set of each category of defects by using a clustering subnetwork to obtain a classification feature vector group corresponding to the defect grade number and the feature size of the classification feature vector group.

Illustratively, for example, for a certain category of defects, the level is 2, so when the hierarchical feature sets are clustered, the hierarchical feature sets can be clustered into two hierarchical feature vector groups, and the feature vectors in each hierarchical feature vector group are similar in size. And the feature size of the hierarchical feature vector group is represented by the average value vector of each hierarchical feature vector group.

It should be understood that the number of defect levels may be determined by a preset. For example, for a certain category of defects, the category of defects is usually divided into two levels, so that the number of levels of the category of defects can be preset to be 2, and thus when a clustering subnetwork is used for clustering the classification feature set of the category of defects, the classification feature set of the defects can be divided into 2 groups of classification feature vector sets by a clustering method.

And then, identifying the pseudo label for each classification characteristic vector group according to a preset pseudo label and the characteristic size of each classification characteristic vector group to obtain the pseudo label of the defect level of each category of defects.

In the implementation of the present invention, a preset dummy tag is used to indicate the level of the defect. After the hierarchical feature vector groups are obtained, the pseudo label can be identified for each hierarchical feature vector group according to the feature size of each hierarchical feature vector group, so as to obtain the pseudo labels of the defect levels of the defects of each category.

Illustratively, where a defect includes two levels, for example, we can preset the pseudo label to be 0 and 1, where 1 indicates severe and 0 indicates not severe. (of course, 0 may mean serious, and 1 may mean not serious). Then, since the hierarchical feature vectors are weighted, the high value of the feature in the two hierarchical feature vector groups of the cluster shows a higher degree of severity, so that the average vector x of the two clusters is counted ₁ And x ₀ Check x ₁ And x ₀ Corresponding to the size relationship of the features, if more features exist, the serious defect is found, and then the set of the group level feature vectors is markedUpper dummy tag 1.

For example, in the embodiment of the present invention, a kmean algorithm or other clustering algorithms may be adopted for clustering the hierarchical feature set used for each category of defects by using a clustering subnetwork.

As an example, a kmeans algorithm is adopted in the embodiment of the present invention, and the principle is to divide a sample set into k clusters according to the distance between samples. The points within the clusters are held together as closely as possible while the distance between clusters is made as large as possible. The process is as follows:

assume that input sample set D ═ x ₁ ,x ₂ ,...x _m H, the cluster tree of the cluster is k, the maximum iteration number is N, and the output is the cluster division C ═ C ₁ ,C ₂ ,...C _k }，

i) Randomly select k samples from the data set D as the initial k centroid vectors: { mu. } ₁ ,μ ₂ ,...,μ _k }

ii) for N ═ 1,2

1) Initializing cluster partition C to

2) For i 1, 2.. times.m, sample x is calculated _i And each centroid vector mu _j Distance of (j ═ 1, 2.. k):

x is to be _i Minimum mark is d _ij Corresponding class λ _i . At this time, update is performed

3) For j 1, 2.. k, pair C _j Recalculate new centroid for all sample points in the image

4) If all k centroid vectors have not changed, go to step iii).

iii) output cluster partitioning C ═ C ₁ ,C ₂ ,...C _k }。

Further, in the embodiment of the present invention, the clustering operation is started after a certain number of hierarchical feature vectors are accumulated in the hierarchical feature vector set. Namely, after executing S201-S203 for a certain number of times, then executing S204.

Furthermore, because the clustering analysis is time-consuming, the speed is very low and unnecessary when the defect is added once, in order to improve the training efficiency, the clustering can be performed once after all the defects of each mini-batch are enqueued, and then the nearest neighbor method is adopted to calculate the grade of the defect. Therefore, in the embodiment of the present invention, the sample image may be divided into a plurality of sample image groups, each of the sample image groups includes a plurality of sample images, and before the classification feature sets of each category of defects are clustered by using the clustering subnetwork, all the classification feature sets of the defects in each of the sample image groups are obtained by using the class activation mapping subnetwork, the feature extraction subnetwork and the feature set subnetwork.

In S105, the preset defect classification detection model is trained according to the prediction type, the prediction position, the prediction level of the defect in the sample image, the labeling information of the defect position and type in the sample image, and the pseudo label of the defect level in the sample image, so as to obtain a target defect classification detection model.

Illustratively, in the embodiment of the present invention, the loss function of the preset defect classification detection model is composed of two parts, namely a loss function of the classification branch network and a loss function of the classification branch network. Therefore, the preset defect classification detection model is trained according to the prediction type, the prediction position and the prediction level of the defect of the sample image, the labeling information of the defect position and type in the sample image and the pseudo label of the defect level in the sample image to obtain a target defect classification detection model, and the method comprises the following steps:

determining a loss function value of the classification branch network according to the prediction type and the prediction position of the defect of the sample image and the marking information of the defect position and type in the sample image;

determining a loss function value of the hierarchical branch network according to the prediction level of the defect of the sample image and the pseudo label;

adjusting parameters of the classification branch network according to the loss function value of the classification branch network;

and adjusting the parameters of the hierarchical branch network according to the loss function values of the hierarchical branch network.

Further, in the embodiment of the present invention, the loss function of the preset defect classification detection model is determined according to the loss function and the first weight of the classification branch network, and the loss function and the second weight of the classification branch network, where the first weight is the weight of the loss function of the classification branch network, the second weight is the weight of the loss function of the classification branch network, and the sum of the first weight and the second weight is 1. Therefore, in S105, the classification branching network and the hierarchical branching network may be alternately trained by adjusting the first weight and the second weight. And, because a hierarchical feature vector set needs to be accumulated, the classification branch network is trained first, and then the hierarchical branch network is trained.

As an example, the alternating training process may be: adjusting the second weight of the loss function of the hierarchical branching network to 0; training the classification branch network by using the sample image; adjusting the second weight of the loss function of the hierarchical branching network to 1; training the hierarchical branching network using the sample images.

As an example, the alternating training process of the defect classification detection model of the present invention may be divided into processes of M1N1M2N2 and so on, where M1, M2 and so on are the training processes of the classification branching network, and N1, N2 and so on are the training processes of the classification branching network. And due to two reasons (firstly, the clustering gen subnetwork needs to accumulate feature sets, and secondly, the positioning and classification of the defects in the initial stage are not accurate), a cold start scheme is adopted during training, the stage of M1 is longer, so that the original network is trained and promoted more, and the quantity and the quality of the feature sets are promoted.

In some embodiments of the present invention, in order to avoid mutual interference of parameter updates of the hierarchical branching network and the classification branching network, only the parameters of the classification branching network are updated when the classification branching network is trained, and only the parameters of the classification branching network are updated when the classification branching network is trained. That is, the parameter updates of the hierarchical branching network and the classification branching network are not performed simultaneously at the time of the alternate training.

It should be understood that, in the alternating training process, when the classification branch network is trained by using the sample images, a thermodynamic diagram of each type of defect and a defect classification feature set of each type of defect are obtained by using the clustering branch network according to the feature diagram of the sample images and the prediction categories and prediction positions of the defects in the sample images.

Further, in some embodiments of the present invention, after the alternating training, the defect classification detection model may also be trained using the sample images, and the classification branch network are cooperatively trained during the training process.

Fig. 8 is a flowchart illustrating a defect classification detection method according to an embodiment of the present invention.

As shown in fig. 8, a defect classification detection method 400 provided by the embodiment of the present invention includes:

s301, acquiring a target image to be subjected to defect detection.

Illustratively, the target image may be acquired by an image capture device such as a camera.

S302, the target image is subjected to defect grading detection by using the target defect grading detection model trained by the method for training the defect grading detection model according to the embodiment of the invention, so that the prediction type, the prediction position and the prediction grade of the defects in the target image are obtained.

The structure of the target defect classification detection model trained by the method for training the defect classification detection model according to the embodiment of the invention can refer to the model shown in fig. 2, the head part of the target defect classification detection model comprises a classification network, a positioning network and a classification network, and after the feature map of the target image is obtained, the prediction category, the prediction position and the prediction grade of the defect in the target image can be respectively obtained through the classification network, the positioning network and the classification network according to the feature map.

S303, determining the defect information of the target image according to the prediction type and the prediction position of the defect in the target image and the prediction level.

When the predicted category, predicted position, and predicted level of the defect in the target image are obtained in S302, the defect information of the target image, i.e., what types of defects the defect includes, at what positions, and what levels the defect belongs to, respectively, is determined according to these information.

According to the defect classification detection method provided by the embodiment of the invention, as the defect classification detection model trained by the method for training the defect classification detection model provided by the embodiment of the invention is additionally provided with the head for defect classification, the target image can be directly used for predicting the grade of the defect after the characteristic diagram is extracted, and no additional analysis is needed after reasoning.

Fig. 9 is a schematic structural diagram of an apparatus for training a defect classification detection model according to an embodiment of the present invention.

As shown in fig. 9, the apparatus for training a defect classification detection model according to the embodiment of the present invention includes a sample obtaining module 210, an input module 220, a first prediction module 230, a second prediction module 240, and a training module 250.

The sample acquiring module 210 is configured to acquire a sample image, where the sample image includes annotation information of defect locations and categories. The sample acquiring module 210 is configured to execute S101 in the method for training the defect classification detection model shown in fig. 3.

The input module 220 is configured to input the sample image into a preset defect classification detection model, where the defect classification detection model includes a classification branch network for obtaining the category and the position of the defect, a classification branch network for obtaining the level of the defect, and a clustering branch network for generating a defect level pseudo tag. The input module 220 is used for executing S102 in the method for training the defect classification detection model shown in fig. 3.

The first prediction module 230 is configured to perform defect classification detection on the sample image by using a classification branch network and a classification branch network in the defect classification detection model, so as to obtain a prediction category, a prediction position, and a prediction level of a defect in the sample image. The first prediction module 230 is used for executing S103 of the method for training the defect classification detection model shown in fig. 3.

The second prediction module 240 is configured to process the sample image by using a clustering branch network in the defect classification detection model to obtain a pseudo label of a defect level in the sample image, and use the pseudo label as a true value of the defect level in the sample image. The second prediction module 240 is used to perform S104 in the method of training the defect classification detection model shown in fig. 3, and S201 to S204 in the method of generating the pseudo labels of the defect levels shown in fig. 7.

The training module 250 is configured to train the preset defect classification detection model according to the prediction type, the prediction position, the prediction level of the defect of the sample image, the labeling information of the defect position and type in the sample image, and the pseudo label of the defect level in the sample image, so as to obtain a target defect classification detection model. The training module 250 is used to execute step S105 of the method for training the defect classification detection model shown in fig. 3.

Further, in the embodiment of the present invention, the training module 250 further includes a weight adjuster, configured to adjust weights of the loss functions of the classification branching network and the classification branching network in the preset defect detection model, so as to train the classification branching network and the classification branching network alternately. The loss weights of the original prediction network (the separation branch network) and the new branch (the hierarchical branch network) can be automatically coordinated through the weight regulator to carry out collaborative training, and the defect positioning and classification network does not need to be trained in advance.

Each module/unit of the apparatus 500 shown in fig. 9 has a function of implementing each step in fig. 3, and can achieve corresponding technical effects, and for brevity, no further description is provided here.

Fig. 10 is a schematic structural diagram of a defect classification detection apparatus according to an embodiment of the present invention.

As shown in fig. 10, the defect classification detecting apparatus 600 according to the embodiment of the present invention includes a picture obtaining module 310, a predicting module 320, and a determining module 330.

The image obtaining module 310 is configured to obtain a target image to be subjected to defect detection. The picture taking module 310 is configured to execute S301 in the defect classification detection method shown in fig. 8.

The prediction module 320 is configured to perform defect classification detection on the target image by using the target defect classification detection model trained by the device for training a defect classification detection model according to the embodiment of the present invention, so as to obtain a prediction type, a prediction position, and a prediction level of a defect in the target image. The prediction module 3200 is used for executing S302 in the defect classification detecting method shown in fig. 8.

The determining module 330 is configured to determine the defect information of the target image according to the predicted category and the predicted position of the defect in the target image and the prediction level. The determining module 330 is used for executing S303 in the defect classification detecting method shown in fig. 8.

Fig. 10 shows that each module/unit of the apparatus 600 has a function of implementing each step in fig. 7, and can achieve the corresponding technical effect, and for brevity, the description is not repeated here.

Fig. 11 is a schematic diagram illustrating a hardware structure of a computing device 700 according to an embodiment of the present invention.

The computing device 700 may include a processor 701 and a memory 702 storing computer program instructions.

Specifically, the processor 701 may include a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits configured to implement the embodiments of the present invention.

Memory 702 may include a mass storage for data or instructions. By way of example, and not limitation, memory 702 may include a Hard Disk Drive (HDD), a floppy Disk Drive, flash memory, an optical Disk, a magneto-optical Disk, tape, or a Universal Serial Bus (USB) Drive or a combination of two or more of these. In one example, memory 702 may include removable or non-removable (or fixed) media, or memory 302 is non-volatile solid-state memory. The memory 702 may be internal or external to the integrated gateway disaster recovery device.

In one example, the Memory 702 may be a Read Only Memory (ROM). In one example, the ROM may be mask programmed ROM, programmable ROM (prom), erasable prom (eprom), electrically erasable prom (eeprom), electrically rewritable ROM (earom), or flash memory, or a combination of two or more of these.

Memory 702 may include Read Only Memory (ROM), Random Access Memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical/tangible memory storage devices. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software comprising computer-executable instructions and when the software is executed (e.g., by one or more processors), it is operable to perform operations described with reference to the methods according to an aspect of the present disclosure.

The processor 701 reads and executes the computer program instructions stored in the memory 702 to implement the methods/steps S101 to SS105 in the embodiment shown in fig. 3, and achieve the corresponding technical effects achieved by the embodiments shown in fig. 3, fig. 7, and fig. 8 executing the methods/steps thereof, which are not described herein again for brevity.

In one example, computing device 700 may also include a communication interface 703 and a bus 710. As shown in fig. 11, the processor 701, the memory 702, and the communication interface 703 are connected via a bus 710 to perform communication with each other.

The communication interface 703 is mainly used for implementing communication between modules, apparatuses, units and/or devices in the embodiment of the present invention.

Bus 710 comprises hardware, software, or both to couple the components of the online data traffic billing device to each other. By way of example, and not limitation, a Bus may include an Accelerated Graphics Port (AGP) or other Graphics Bus, an Enhanced Industry Standard Architecture (EISA) Bus, a Front-Side Bus (Front Side Bus, FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) Bus, an infiniband interconnect, a Low Pin Count (LPC) Bus, a memory Bus, a Micro Channel Architecture (MCA) Bus, a Peripheral Component Interconnect (PCI) Bus, a PCI-Express (PCI-X) Bus, a Serial Advanced Technology Attachment (SATA) Bus, a video electronics standards association local (VLB) Bus, or other suitable Bus or a combination of two or more of these. Bus 710 may include one or more buses, where appropriate. Although specific buses have been described and shown in the embodiments of the invention, any suitable buses or interconnects are contemplated by the invention.

In addition, in combination with the method for training the defect classification detection model and the defect detection method in the above embodiments, the embodiments of the present invention may be implemented by providing a computer storage medium. The computer storage medium having computer program instructions stored thereon; the computer program instructions, when executed by a processor, implement any one of the above-described methods of training a defect classification detection model or a defect detection method.

It is to be understood that the invention is not limited to the specific arrangements and instrumentality described above and shown in the drawings. A detailed description of known methods is omitted herein for the sake of brevity. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present invention are not limited to the specific steps described and illustrated, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps, after comprehending the spirit of the present invention.

The functional blocks shown in the above-described structural block diagrams may be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it may be, for example, an electronic Circuit, an Application Specific Integrated Circuit (ASIC), suitable firmware, plug-in, function card, or the like. When implemented in software, the elements of the invention are the programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine-readable medium or transmitted by a data signal carried in a carrier wave over a transmission medium or a communication link. A "machine-readable medium" may include any medium that can store or transfer information. Examples of a machine-readable medium include electronic circuits, semiconductor memory devices, ROM, flash memory, Erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, Radio Frequency (RF) links, and so forth. The code segments may be downloaded via computer networks such as the internet, intranet, etc.

It should also be noted that the exemplary embodiments mentioned in this patent describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above steps, that is, the steps may be performed in the order mentioned in the embodiments, may be performed in an order different from the order in the embodiments, or may be performed at the same time.

Aspects of the present disclosure are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions/acts specified in the flowchart and/or block diagram block or blocks. Such a processor may be, but is not limited to, a general purpose processor, a special purpose processor, an application specific processor, or a field programmable logic circuit. It will also be understood that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based computer instructions which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

As described above, only the specific embodiments of the present invention are provided, and it can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the system, the module and the unit described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again. It should be understood that the scope of the present invention is not limited thereto, and any equivalent modifications or substitutions can be easily made by those skilled in the art within the technical scope of the present invention.

Claims

1. A method of training a defect classification detection model for detecting the type, location and level of defects from an industrial defect identification image, the method comprising:

2. The method of claim 1, wherein the performing defect classification detection on the sample image by using the defect classification detection model to obtain a prediction type, a prediction position and a prediction level of a defect in the sample image comprises:

acquiring a characteristic map of the sample image;

processing the feature map by using the classification branch network to obtain the prediction type and the prediction position of the defect in the sample image;

and processing the characteristic graph by utilizing the hierarchical branch network to obtain the prediction level of the defects in the sample image.

3. The method of claim 2, wherein the clustering branch network comprises a class activation mapping subnetwork, a feature extraction subnetwork, a feature set subnetwork, and a clustering subnetwork;

the processing the sample image by using the clustering branch network in the defect classification detection model to obtain the pseudo label of the defect level in the sample image comprises:

utilizing the class activation mapping sub-network to obtain a thermodynamic diagram of each class of defects according to the feature diagram and the prediction classes and prediction positions of the defects in the sample image;

obtaining a defect grading feature vector of each type of defects according to the thermodynamic diagram by using the feature extraction sub-network;

utilizing the feature set subnetwork, and storing the defect grading feature vectors into grading feature sets of corresponding categories of defects according to the prediction levels of the defects;

and clustering the classification feature set of each category of defects by using the clustering subnetwork to obtain the pseudo labels of the defect levels of each category of defects, wherein the pseudo labels are used for expressing the levels of the defects.

4. The method of claim 3, wherein said using the feature extraction sub-network to obtain a defect-ranking feature vector for each type of defect based on the thermodynamic diagram comprises:

and acquiring at least one of the information of the thermodynamic diagram foreground area size, the thermodynamic diagram foreground binary image filling degree, the thermodynamic diagram foreground region uniformity, the thermodynamic diagram foreground and background contrast and the thermodynamic diagram highlight area occupation ratio.

5. The method of claim 3, wherein the clustering the hierarchical feature set of each category of defects by the clustering subnetwork to obtain the pseudo label of the defect level of each category of defects, the pseudo label being used for indicating the defect level, comprises:

clustering the classification feature set of each category of defects by utilizing a clustering subnetwork to obtain a classification feature vector group corresponding to the defect grade number and the feature size of the classification feature vector group;

and identifying the pseudo label for each classification characteristic vector group according to a preset pseudo label and the characteristic size of each classification characteristic vector group to obtain the pseudo label of the defect grade of each category of defects.

6. The method according to claim 3, wherein the sample image is divided into a plurality of sample image groups, each of the sample image groups including a plurality of the sample images,

before clustering the hierarchical feature set of each category of defects by using the clustering subnetwork,

and acquiring all the classification feature sets of the defects of each sample image group by utilizing the class activation mapping sub-network, the feature extraction sub-network and the feature set sub-network.

7. The method according to any one of claims 1-6, wherein training the preset defect classification detection model according to the prediction category, prediction position and prediction level of the defect of the sample image and the labeling information of the defect position and category in the sample image and the pseudo label of the defect level in the sample image comprises:

determining a loss function value for the hierarchical branch network from the prediction level of the defect of the sample image and the pseudo label;

8. The method of claim 7, wherein the loss function of the predetermined defect classification detection model is determined according to the loss function and the first weight of the classification branch network, and the loss function and the second weight of the classification branch network, and the method further comprises:

adjusting the first weight and the second weight, alternately training the classification branch network and the classification branch network, and training the classification branch network first and then training the classification branch network;

wherein only the parameters of the classification branch network are updated when the classification branch network is trained, and only the parameters of the classification branch network are updated when the classification branch network is trained.

9. The method of claim 8, wherein the alternately training the classification branching network and the hierarchical branching network comprises:

adjusting the second weight of the loss function of the hierarchical branching network to 0;

training the classification branch network by using the sample image;

adjusting the second weight of the loss function of the hierarchical branching network to 1;

training the hierarchical branching network using the sample images.

10. The method according to claim 9, wherein when the classification branch network is trained by using the sample images, a thermodynamic diagram of each type of defect and a defect classification feature set of each type of defect are obtained by using the clustering branch network according to the feature diagram of the sample images and the prediction categories and prediction positions of the defects in the sample images.

11. A defect classification detection method is characterized by comprising the following steps:

acquiring a target image to be subjected to defect detection;

carrying out defect grading detection on the target image by using a target defect grading detection model trained by the method according to any one of claims 1-10 to obtain a prediction type, a prediction position and a prediction grade of a defect in the target image;

12. An apparatus for training a defect classification detection model, the apparatus comprising:

13. A defect classification detection apparatus, comprising:

a prediction module, configured to perform defect classification detection on the target image by using a target defect classification detection model trained by the apparatus according to claim 12, so as to obtain a prediction type, a prediction position, and a prediction level of a defect in the target image;

14. A computing device, wherein the device comprises: a processor, and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the method for training a defect classification detection model according to any one of claims 1 to 10, or the detection method according to claim 11.

15. A computer storage medium having computer program instructions stored thereon, which when executed by a processor, implement the method of training a defect classification detection model of any one of claims 1-10, or the detection method of claim 11.