Background
The Ream of Paper (real Paper) refers to a certain number of sheets with the same unit in the Paper industry, and 500 sheets with the same unit are generally taken as a Ream of Paper, namely Ream of Paper. For example, in office paper, a typical a4 sheet is typically 500 pages in one package. The reams need to be subjected to quality inspection screening on an automatic production line of a paper making enterprise, defective reams are removed from a production line queue, and qualified normal reams enter the next packaging procedure.
The level of mechanized and automated information production of large paper making enterprises is increasing day by day, but various defects occur due to the influence of a series of factors such as incoming materials, cutters, links of a production line and production environment in the process of producing ream paper, such as: dirt, wrinkles, bulges, irregularities, dead lines, burrs, etc. The existence of the defects reduces the quality of paper products leaving factories, and the quality management of the paper making enterprises is generally carried out in a manual inspection mode. However, the manual inspection mode has low sampling inspection rate, low accuracy, poor real-time performance, low efficiency, high labor intensity, large influence of manual experience and subjective factors, repeated work, noisy field and easy fatigue of personnel, the yield cannot be effectively counted, great inconvenience is brought to the information production management work, and the inspection effect is not ideal.
In order to solve the problem, some paper-making enterprises introduce a ream visual detection system, and a series of image processing operations (such as edge detection, projection, threshold segmentation, and equal texture ratio) are carried out on a real-time image of the ream on an assembly line so as to find out the defective ream. However, due to the numerous defect types and different appearances, the algorithm is difficult to adapt to various exceptions, so that the practical effect is difficult to achieve.
In addition to the above-mentioned traditional ream paper visual processing method, the deep learning classification method based on the neural network is also being popularized and used, such as a target detection classification method. The target detection classification method includes a YOLO series, an SSD (single Shot multiple boxdetector) algorithm, a Faster R-CNN, and the like, wherein the YOLO and the SSD perform classification judgment of the object while predicting whether the object to be detected exists in the specific region, and the Faster R-CNN detects a possible position of the object to be detected, and then captures image feature information of the corresponding position for classification processing. Because the size scale span of the defects in the ream defect detection process is large, the large defects can occupy the whole ream image, the small defects are only 10 pixels in size, meanwhile, the defect proportion is various, and a small number of candidate frames with different sizes and proportions are difficult to cover all the situations.
Aiming at the problems, the invention provides a ream paper defect segmentation and classification detection method, namely a ream paper defect detection method which comprises the steps of segmenting the complete defect position of a ream paper image, and then intercepting the image characteristics for classification.
Disclosure of Invention
The invention provides a method for detecting ream paper defect segmentation classification, which aims to solve the problems that the ream paper defect has large size and scale span and is difficult to cover all conditions by using a small number of candidate frames with different sizes and proportions.
In order to achieve the above object, the method for detecting ream paper defect segmentation and classification provided by the present invention specifically comprises the following implementation steps:
step S1, training, segmenting and classifying network module:
1) and (3) training a segmentation network module: performing pixel-level labeling on a ream defect area to obtain a certain number of labeled data sets, performing two classification on each pixel, adopting cross entropy as a basic Loss function, then properly adjusting function side emphasis, adopting the Loss function as a target to perform training, and training a segmentation network to learn to obtain the representation characteristics of the ream defect, so that the network can segment the ream defect area from an input ream image;
2) training a classification network module: the classification network comprises 4 submodules which are respectively connected with 4 levels of different network layers in the segmentation network in an abutting mode, the input of the classification network module is a characteristic tensor of H, W and C, the characteristic tensor is from the ROI (region of interest) characteristics of the corresponding network layer, the characteristic tensor is output as a ream defect type, and the training of the classification network submodules is completed by combining ream defect type labels and adopting a multi-classification cross entropy and random gradient descent method.
Step S2, inputting the ream image into a segmentation network module, adopting an Encoder-Decoder neural network structure, obtaining abstract ream defect characteristics through the Encoder, then gradually up-sampling the defect characteristics in the Decoder process, and simultaneously fusing the characteristic information of the corresponding network layer in the Encoder process, thereby obtaining the characteristics of global and local combination, so as to more effectively segment ream defect areas and obtain a pixel-level ream defect segmentation graph;
step S3, entering the ROI calculating module: processing a two-class segmentation graph of the ream defects by adopting an expansion method, merging defect points with short distances, finding out an outer wrapping curve of the ream defects from the two-class segmentation graph by using a findContours () method, obtaining a corresponding external rectangle capable of framing a complete ream defect area by adopting a bounngRect () method, and finally obtaining the external rectangle of the defects and a corresponding proper network layer number;
step S4, entering the ROI information extraction module: extracting corresponding features on a selected network layer according to the defect external rectangle, and mapping feature areas with different sizes to a feature tensor with the size of H, W and C, wherein H is the height of a feature plane, W is the width of the feature plane, and C is the number of feature channels;
and step S5, the classification network module classifies and judges the ream defects, sets the classification category number as K +1, comprises K defect categories and 1 normal ream category, 4 submodules in the module respectively correspond to different network layers, inputs a feature tensor of H W C, converts the feature tensor into a feature tensor of 1W 1 (K +1) after two-layer convolution processing, and outputs the ream defect categories through Softmax to obtain the detection result of the ream defects.
Detailed Description
In order to make the purpose and technical solution of the present invention clearer, the following will clearly and completely describe the technical solution of the present invention with reference to the accompanying drawings. It should be understood that the specific technical embodiments described herein are merely to explain the technical solutions of the present invention, and other embodiments obtained by those skilled in the art without making creative efforts should fall within the protective scope of the present invention.
Referring to fig. 1, a flow chart of ream defect detection according to the present invention is shown, and a method for detecting ream defect segmentation and classification according to the present invention specifically includes the following steps:
step S1, training, segmenting and classifying network module:
1) and (3) training a segmentation network module: performing pixel-level labeling on a ream defect area to obtain a certain number of labeled data sets, performing two classification on each pixel, adopting cross entropy as a basic Loss function, then properly adjusting function side emphasis, adopting the Loss function as a target to perform training, and training a segmentation network to learn to obtain the representation characteristics of the ream defect, so that the network can segment the ream defect area from an input ream image;
2) training a classification network module: the classification network comprises 4 submodules which are respectively connected with 4 levels of different network layers in the segmentation network in an abutting mode, the input of the classification network module is a characteristic tensor of H, W and C, the characteristic tensor is from the ROI (region of interest) characteristics of the corresponding network layer, the characteristic tensor is output as a ream defect type, and the training of the classification network submodules is completed by combining ream defect type labels and adopting a multi-classification cross entropy and random gradient descent method.
Preferably, in step S1, the basic Loss function in the training of the segmentation network module is expressed by the following formula:
wherein N is the total number of pixels in a graph, i is the serial number of the pixel point, and yiA label for this point, 0 is represented as a normal pixel, 1 is represented as a defective pixel, piRepresenting the probability of predicting the point as a defective pixel, α is a class balance coefficient for correcting the imbalance between the defective pixel and the normal pixel, γ is a suppression coefficient for suppressing the contribution of the easily classified pixels to the loss function to improve the segmentation effect of the difficultly classified pixels, and w is a weight for suppressing the contribution of the easily classified pixels to the loss functioniThe weight coefficient of the pixel point is used for adjusting the training weight of the small defect area so as to improve the segmentation effect of the small defect;
the w weight coefficient is calculated image by image, a wrapping curve of a single defect on the label map is obtained by a findContours () method, the area a of the wrapping region is obtained by a contourArea () method, and the weights of all pixels in the wrapping region are set by the following formula:
w=(At÷A)÷β
wherein At is the total area of the input ream image, A is the ream defect wrapping area, and beta is the proportional adjustment coefficient.
Preferably, in step S1, the training of the classification network module includes 4 sub-modules, which are independent from each other, and can be trained separately or collectively, and in the training process, the network layer numbers of the segmentation networks are obtained by a method of calculating ROI regions in the segmentation networks to determine corresponding classification networks, the input of the module is a feature tensor of H × W × C given by the "ROI information extraction module", the output is a defect class, the number of classification classes is set to K +1, and the classification classes include K defect classes and 1 normal ream class, and then the Loss function formula is:
where K is the number of types of ream defect classes, c is the number of classes, c-0 indicates a normal class, and y iscIs an indicator variable, is 0 or 1, that is if the class is c, then yc1, otherwise equal to 0; p is a radical ofcThe probability that the ROI area belongs to the class c is calculated by a multi-classification Softmax function. And (3) finishing the training of each classification network by adopting the loss function and combining the input ROI characteristics and the ream defect class labels by adopting a random gradient descent method.
And step S2, inputting the ream image into a segmentation network module, acquiring abstract ream defect characteristics through an Encoder by adopting an Encoder-Decoder neural network structure, gradually up-sampling the defect characteristics in the Decoder process, and simultaneously fusing the characteristic information of a corresponding network layer in the Encoder process, thereby obtaining the characteristics of global and local combination, so as to more effectively segment ream defect areas and obtain a pixel-level ream defect segmentation graph.
Referring to fig. 2, a ream defect detection network architecture diagram of the present invention is shown, wherein a network segmentation module inputs a ream image and outputs a ream defect segmentation diagram, which includes 9 layers, L1 is denoted as Layerl, and so on.
Step S3, entering the ROI calculating module: processing the two-class segmentation drawing of the ream defects by adopting an expansion method, merging defect points with short distances, finding out an outer wrapping curve of the ream defects from the two-class segmentation drawing by using a findContours () method, obtaining a corresponding external rectangle capable of framing a complete ream defect area by adopting a bounngRect () method, and finally obtaining the external rectangle of the defects and a corresponding proper network layer number.
Specifically, as the defect scale span of the ream is large, the unmatched condition may occur by directly adopting the external rectangle to intercept the features from a certain network layer for classification. Considering that network layer features of different stages in a Decoder part of a split network have different abstraction levels and detail levels, the network layer feature abstraction level of the front stage of the Decoder is high and is suitable for extracting large-size defect features, and the network layer feature detail level of the rear stage is high and is suitable for extracting small-size defect features, therefore, different network layers are selected according to the size of the long edge of the circumscribed rectangle to carry out defect feature interception.
Further, 4 network layers used for extracting features in the network are numbered and a corresponding length threshold table is established, wherein the length threshold represents that the network layer of the number is selected to extract the features when the long edge of the circumscribed rectangle is smaller than the threshold, the corresponding length threshold of the network layer at the front stage of the Decoder is large, and the corresponding length threshold of the network layer at the rear stage of the Decoder is small. Because only one length threshold upper limit is reserved in one network layer, when the judgment is carried out according to the size of the long edge of the external rectangle, the judgment needs to be carried out from the network layer at the later stage of the Decoder, if the threshold condition is not met, whether the threshold value of the network layer at the previous stage is met is judged, and until the network layer meeting the threshold condition is found.
In order to avoid the occurrence of too small or too slender rectangles in the subsequent processing, the minimum value Lmin of the rectangular sides is set for 4 network layers used for extracting features in the network respectively, if the external rectangular sides are smaller than the Lmin of the selected network layer, the Lmin is modified, and the two-class classification segmentation graph of the paper defects can be processed to obtain the external rectangles of the defects and the numbers of the corresponding network layers.
Step S4, entering the ROI information extraction module: extracting corresponding features on the selected network layer according to the defect external rectangle, and mapping the feature regions with different sizes to a feature tensor with the size of H, W and C, wherein H is the height of the feature plane, W is the width of the feature plane, and C is the number of feature channels.
Specifically, the ROI information extraction module described in step S4 includes 4 sub-modules, each of which corresponds to a different network layer, the mapping method of each sub-module uses the ROI Align in the Mask RCNN detection algorithm, a rectangle to be extracted is mapped to a region on the network layer, a decimal is reserved, the region is uniformly divided into grids of H × W size, as shown in fig. 3, each grid is subdivided into 2 × 2 uniform intervals, a bilinear interpolation method is used for each interval to calculate a value of an interval center point, 4 feature values are calculated for 2 × 2 intervals, and a maximum value among the 4 is taken as a feature value of the grid, so as to obtain ROI feature information.
And step S5, the classification network module classifies and judges the ream defects, sets the classification category number as K +1, comprises K defect categories and 1 normal ream category, 4 submodules in the module respectively correspond to different network layers, inputs a feature tensor of H W C, converts the feature tensor into a feature tensor of 1W 1 (K +1) after two-layer convolution processing, and outputs the ream defect categories through Softmax to obtain the detection result of the ream defects.
Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the present disclosure. It will be understood that the disclosure is not limited to the embodiments described and disclosed above, but is intended to cover all modifications and changes that may be made without departing from the scope of the disclosure, as defined in the appended claims.