CN114005081A

CN114005081A - Intelligent detection device and method for foreign matters in tobacco shreds

Info

Publication number: CN114005081A
Application number: CN202111122042.9A
Authority: CN
Inventors: 王显昆; 常永军
Original assignee: Changzhou Shinco Automotive Electronic Co ltd
Current assignee: Changzhou Shinco Automotive Electronic Co ltd
Priority date: 2021-09-24
Filing date: 2021-09-24
Publication date: 2022-02-01

Abstract

The invention discloses an intelligent detection device and method for foreign matters in tobacco shreds, and belongs to the technical field of image processing technology and deep learning. The method comprises the following steps: suppressing disordered dark edge textures based on progressive median filtering, segmenting foreign matters based on HSV color space, and positioning the foreign matters based on a boundary tracking algorithm; based on the fast R-CNN model, the model structure is optimized by combining a bottom-up two-way characteristic pyramid and a linear interpolation pooling method, and the detection precision is improved. The invention aims to solve the problems of foreign body segmentation under the background of complex tobacco materials and tobacco foreign body detection under a small target sample in the prior art, the foreign body inspection efficiency can be ensured by adopting a rapid coarse inspection algorithm based on an image processing algorithm with high inspection yield and high false detection rate, and the universality of the detection effect and the accuracy rate of the detection are improved by carrying out fine inspection detection on the basis of an improved FasterR-CNN model.

Description

Intelligent detection device and method for foreign matters in tobacco shreds

Technical Field

The invention relates to the technical field of cigarette production management, belongs to the technical field of information acquisition and processing, and collectively relates to an intelligent detection device and method for tobacco shred foreign matters.

Background

As early as the end of the twentieth century, tobacco enterprises advanced abroad began to study the automatic tobacco foreign matter detection and impurity removal technology and were gradually put into use in factories. With the development of science and technology, emerging technologies such as laser, photoelectric and machine vision are gradually applied to the impurity removal of tobacco foreign matter. In early research, various tobacco impurity removal devices have higher requirements on the specificity of targets, such as a metal detector for detecting metal component foreign matters in materials or a nylon barb for cleaning filiform foreign matters.

The foreign matter detection problem is divided from the aspect of a foreign matter state and mainly comprises two types of foreign matter existence detection and foreign matter invasion detection. The background state can be divided into foreign object detection in a single background and foreign object detection in a changed background. Foreign matter detection under a single background is mostly foreign matter intrusion detection, and the application scene of the foreign matter intrusion detection is mainly performed in places with higher requirements on security and isolation, such as airport runways, railway rails and the like. For foreign body intrusion detection under a single background, a background difference method is mostly adopted, modeling is firstly carried out on a conventional background without a foreign body state, and then a pixel region which is not matched with the background is extracted by using a frame difference method. Foreign matter detection under the background is mostly foreign matter existence detection, and the tobacco shred foreign matter detection studied in the text is taken as an example, and specific materials are taken as the background, so that foreign matters or impurities which are not consistent with the materials are detected. The background is also changed in real time as the material is disorderly and disorderly when placed in a heap. The problems are different from foreign object intrusion detection, cannot be solved based on methods such as a background difference method and the like, and are more similar to the surface flaw detection problem.

In order to solve the problem of industrial surface defect detection or foreign matter detection, algorithms based on image processing technology are mainly classified into two categories, namely a spatial domain method and a frequency domain method. Common spatial domain analysis methods are: gray level co-occurrence matrix method, morphological method, edge detection method, etc. Common frequency domain analysis methods are: fourier transform method, Gabor filter method, wavelet transform method, etc. Numerous studies have shown that the method of pre-processing an image using conventional image processing algorithms to enhance the useful information of the image while suppressing background and noise also has little effect and success on the foreign object/flaw detection problem.

In recent years, the deep learning technology is also gradually applied to industrial intelligent production, and a plurality of assembly line work which is finished by manual detection in the past, such as workpiece type identification, surface flaw detection, material foreign matter detection and the like, gradually depend on the deep learning technology to finish automatic or semi-automatic improvement. Compared with a common target, the foreign matters or flaws often appear in the image in a small-size form, and for the whole image, the target information is less, the image features are less obvious, the detection difficulty is higher, and the method is a key point for model optimization improvement.

The Chinese invention patent CN202110071378.0 discloses a self-learning type tobacco shred sundry visual image detection method, which relates to machine visual detection and deep learning, but the method is different from the method. In general, experts and scholars at home and abroad have a lot of researches around foreign body/flaw detection by using deep learning technology, and the results are quite abundant. Most current deep learning studies focus on workpiece or cloth fabric related flaw detection and foreign object intrusion detection in large backgrounds. Foreign matter detection research aiming at small backgrounds, disordered texture backgrounds and small targets of tobacco shreds is less.

Disclosure of Invention

The invention provides an intelligent detection device for tobacco foreign matters, which effectively solves the technical problems and comprises a spreader, a conveyor belt, an image acquisition hardware platform, a removing device and an image processing software platform, wherein the image processing software platform comprises an image acquisition reading module, a foreign matter detection module, an algorithm selection module, a rough detection module, a Faster R-CNN model and a result storage module; the spreading device is connected with the conveying belt, the materials to be detected are evenly spread on the conveying belt and moved to the image acquisition hardware platform, the image acquisition hardware platform is in network communication with the image processing software platform, the acquired images are transmitted to the image acquisition reading module to be fixedly cut and corrected and then transmitted to the foreign matter detection module, the algorithm selection module is used for selecting the rough detection module or the fast R-CNN model to correspondingly detect the acquired images, the detection results are stored in the result storage module, and meanwhile, the foreign matter detection module transmits the region numbers of the foreign matters to the removing device.

Further, pipe tobacco foreign matter intellectual detection system device still includes retraining module, and the data of waiting to examine that image acquisition read module and result storage module collected, through retraining module is further trained on original model basis again, strengthens the generalization ability of model.

Further, the retraining module comprises a category updating model, and foreign matter categories are added when new foreign matters appear, so that the intelligent tobacco shred foreign matter detection device has more perfect foreign matter detection capability.

Furthermore, the intelligent tobacco shred foreign matter detection device is connected with a display module and a result storage module, and displays the detected image information in real time for operators to monitor and follow-up check in real time; the image acquisition hardware platform is a digital camera using a CMOS camera.

On the other hand, the invention also provides an intelligent detection method for foreign matters in tobacco shreds, which is implemented by using the intelligent detection device for foreign matters in tobacco shreds, preferentially using a coarse detection method for foreign matters based on image processing under the condition that the base number of materials to be detected is large, and then using a deep learning method based on an Faster R-CNN model according to the following steps:

the method comprises the following steps: processing a dark edge of the to-be-detected material in a disordered large area by using a progressive median filtering method;

step two: performing binary separation on foreign matters and cut tobacco in the material to be detected by using a threshold value division method under an HSV color model;

step three: carrying out pixel positioning on the foreign matters by using a positioning frame selection method based on boundary tracking, and carrying out space mapping according to the position relation between the image acquisition hardware platform view field and the conveyor belt;

step four: deep learning based on the fast R-CNN model is adopted, foreign body characteristics are learned through training of the deep network model and are distinguished from background characteristics, detection capability is enhanced, and meanwhile the applicability of the detection method is improved.

Further, the first step specifically includes:

firstly, expanding an image of an original image under the size of a convolution kernel of 3 multiplied by 3; then, carrying out primary median filtering by using a 15 multiplied by 15 convolution kernel; then using the convolution kernel of the parameter [2-1,5, -1,0, -1,0] to sharpen the image; finally, a second median filter is performed using a 15 × 15 or 9 × 9 convolution kernel.

Further, the second step specifically includes performing image segmentation according to the characteristic difference between the cut tobacco and the foreign matter under an HSV color model, where the calculation formula of conversion of the HSV color model is as follows:

V＝max(R,G,B)

wherein, R, G and B represent three channels of color, R represents the value of a red channel, G represents the value of a green channel, and B represents the value of a blue channel; each pixel point has color information and consists of three primary colors of red, green and blue light, namely three components, the maximum value and the minimum value of the three components are selected from R, G and B arrays, and the maximum value is defined as the value of brightness V; the saturation S is determined according to the difference value between the maximum value and the minimum value, the value range is 0-100%, and the larger the value is, the more saturated the color is; the chroma H is discussed according to different situations of the color to which the maximum value belongs, the chroma H is measured by angles, the value range is 0-360 degrees, the chroma H is calculated from red in a counterclockwise direction, the red is 0 degree, the green is 120 degrees, the blue is 240 degrees, and the complementary colors of the red, the green and the blue are as follows: yellow is 60 °, cyan is 180 °, violet is 300 °.

Further, the fourth step specifically includes:

the first step is as follows: the method comprises the steps of applying a double-line data median augmentation method and a transfer learning technology to enhance the generalization capability and the convergence speed of a model, firstly applying the double-line data median augmentation method to perform offline augmentation on image data, putting the augmented data into the model, performing online augmentation by combining four transformation random combinations of translation, scaling, overturning and cutting during training, and then applying the transfer learning technology to transfer model parameters trained by a network in a certain large-volume data set to a network node to be trained so as to serve as initial node parameters of the training network;

the second step is that: a double-path characteristic pyramid network combined with a bottom-up path is introduced, so that the learning capacity of the model on multi-scale and small-scale characteristics is improved, the double-path characteristic pyramid network is used for expanding characteristic extraction, and the extracted characteristics are more and richer;

the third step: the problem of mismatching caused by quantization is relieved by using a bilinear interpolation method; the bilinear interpolation method uses a regional suggestion network and a backbone network, wherein the regional suggestion network generates a part of candidate frames firstly, then screens the candidate frames to obtain suggestion frames, and the suggestion frames are divided into a detection frame and a target frame; the coordinates of the corresponding pixel points of the suggestion frame on the original image are floating point type coordinates; when the suggestion frame is mapped to the feature map output by the backbone network, the floating point type coordinates of the corresponding suggestion frame area in the feature map are reserved, and the boundary is not quantized; when the pooling operation is carried out to divide the suggested frame area into a plurality of areas, the boundary of each area is not subjected to quantization processing; sampling in each area, obtaining a corresponding sampling value by using a bilinear interpolation method for the coordinates of the sampling center point of each area, and reserving the maximum sampling value as the maximum pooling result of the area;

the formula of the back propagation is

In the formula, L represents a loss function, x_iRepresenting pixel points, y, on the pre-pooling profile_rjRepresents the j-th point of the sample in the r-th suggestion frame after pooling, d (i, i (r, j)) represents the distance between two points, Δ h represents the difference between the abscissa of xi and xi (r, j), and Δ w represents the difference between the ordinate of xi and xi, and Δ h and Δ w act on the original gradient as the coefficients of the bilinear interpolation method in the above formula.

Further, the two-line data median augmentation method and the transfer learning technology comprise the following steps:

(1) the Faster R-CNN model puts the collected image into a feature extraction module to carry out multilayer convolution, excitation and pooling to obtain deep features, wherein the feature extraction module is in front of the feature map;

(2) using a regional suggestion network, on one hand, extracting the candidate frame according to a set anchor point and classifying positive and negative samples through a softmax excitation function, on the other hand, performing boundary regression on the positive samples selected by the suggestion frame, and performing calculation classification through the softmax excitation function, wherein the softmax excitation function generates a numerical value in an interval of 0-1, and partitions the interval according to a set classification problem, and the calculated numerical value belongs to which class in which interval, and the formula is as follows:

where wy.x is the input vector value;

(3) uniformly generating the suggestion frame in the step (2) to finish coarse positioning of the material to be detected;

(4) inputting the suggestion frame output by the area suggestion network and the original feature map output by the feature extraction module into an area-of-interest pooling module, wherein the area-of-interest pooling module converts the feature map region corresponding to the suggestion frame input into the area-of-interest pooling module into a new feature map with a consistent size at the position after the suggestion frame generates the area suggestion network;

(5) the classification uses a classification regression module and a full connection layer, the classification regression module is positioned at the part behind the region-of-interest pooling module, the classification regression module classifies and identifies the feature map of each suggestion frame through the full connection layer and a softmax excitation function, the confidence coefficient of each suggestion frame in each category is calculated, the category with the highest confidence coefficient is taken as the final category of a target framed by the suggestion frame, the categories are divided into two categories, tobacco shreds are one category, and foreign matters are one category;

(6) when the training is finished and the practical application is carried out, a detected target frame has a large amount of redundancy, and at the moment, a non-maximum suppression algorithm is adopted for pruning so as to generate a more accurate and concise detection frame; the non-maximum suppression algorithm is to suppress elements which are not maximum, search local maximum, and the process is as follows: assuming the set A, B first, then the degree of overlap IoU ═ a ═ B)/(a ═ B), then in the case of homogeneous categories, the scores of each of the target frames are sorted from large to small, taking out the score that is the highest among them; then, the overlapping degree IoU is calculated with all other remaining interested areas, wherein the interested areas refer to areas where detection targets possibly exist, and the interested area border with the overlapping degree IoU of the current interested area border being larger than the set overlapping degree IoU threshold value is removed; and finally, continuously selecting the region of interest with the highest score from the unprocessed region of interest borders, and repeating the process until all the regions of interest in each category are found.

Further, the two-way feature pyramid network is a simple two-way feature pyramid network in a PANet model, and the two-way feature pyramid network is preferably a two-way feature pyramid network in an EfficientDet model.

The invention has the beneficial effects that:

(1) aiming at the conditions that the quantity of tobacco shred materials is large and the probability of foreign matters is low, the foreign matter inspection efficiency can be ensured by adopting the rapid rough inspection algorithm based on the image processing algorithm with high inspection rate and high false inspection rate.

(2) The invention carries out fine detection on the roughly detected materials by using the improved Faster R-CNN model, improves the universality of the detection effect and the accuracy rate of the detection although the detection speed is reduced, and improves the detection rate by pertinently improving the small-size and multi-size characteristics of the foreign matters by using the Faster R-CNN model.

(3) The Faster R-CNN network abandons the Selective Search method of the traditional R-CNN network when extracting the candidate frame and uses the regional suggestion network (RPN).

(4) The invention improves the transmission structure of the characteristic pyramid network by using the PANet model to improve the image instance segmentation precision, and adds a bottom-up propagation path in a top-down fusion path. Furthermore, the invention preferably selects the EfficientDet model proposed by the Google team, adds an additional propagation path between the input and the output of the feature pyramid network at the same level, and fuses the feature graph of each layer with richer feature information without increasing additional calculation amount. In addition, the EfficientDet model introduces a weight concept to the propagation path, the feature maps in different layers are not simply and directly added, but weight values of different input feature maps are learned through a network so as to adapt to the importance degree of the features in different layers to a final detection result.

(5) The generalization capability and the convergence speed of the model are enhanced by applying a double-line data median augmentation method and a transfer learning technology; a two-path characteristic pyramid network combined with a bottom-up path is introduced, so that the learning capacity of the model on multi-scale and small-scale characteristics is improved; the use of bilinear interpolation alleviates the mismatch problem caused by quantization.

Drawings

In order to better express the technical scheme of the invention, the following drawings are used for explaining the invention:

FIG. 1 is a block diagram of a first system and a second system according to an embodiment;

FIG. 2 is a flow chart of a third algorithm of the embodiment;

FIG. 3 is a flow chart of a third progressive median filtering method according to an embodiment;

fig. 4 is a schematic diagram of a three-way simple feature pyramid network structure in the embodiment;

FIG. 5 is a three median augmented image contrast map of the embodiment;

FIG. 6 is a flow chart of data augmentation according to a third embodiment;

FIG. 7 is a flow chart of an embodiment of a three-transition learning;

FIG. 8 shows the structure of the triple fast R-CNN of the embodiment;

FIG. 9 is a schematic diagram of quantization-free sampling of each region in a region-of-interest matching (ROI Align) of the embodiment;

FIG. 10 is a schematic diagram illustrating a method for locating a three-pixel region according to an embodiment;

FIG. 11 is a schematic diagram of an embodiment of three-space mapping;

the reference numbers illustrate: 1. the device comprises a spreader, 2, a conveyor belt, 3, an image acquisition hardware platform, 4, a removing device, 5, an image processing software platform, 51, an image acquisition reading module, 52, a foreign matter detection module, 53, an algorithm selection module, 54, a rough detection module, 55, a Faster R-CNN model, 56, a result storage module, 57, a retraining module, 6, a display module, 7, tobacco shreds, 8 and foreign matters.

Detailed Description

Preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only for explaining the technical principle of the present invention, and do not limit the scope of the present invention.

It should be noted that in the description of the present invention, the terms of direction or positional relationship indicated by the terms "upper", "lower", "left", "right", "inner", "outer", etc. are based on the directions or positional relationships shown in the drawings, which are only for convenience of description, and do not indicate or imply that the device or element must have a specific orientation, be constructed in a specific orientation, and be operated, and thus, should not be construed as limiting the present invention.

Furthermore, it should be noted that, in the description of the present invention, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be, for example, fixedly connected, detachably connected, or integrally connected; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood by those skilled in the art according to specific situations.

The embodiment I and II refer to fig. 1, the intelligent detection device for the foreign matters in the tobacco shreds comprises a spreading device (1), a conveyor belt (2), an image acquisition hardware platform (3), a removing device (4) and an image processing software platform (5), wherein the image processing software platform (5) comprises an image acquisition reading module (51), a foreign matter detection module (52), an algorithm selection module (53), a rough detection module (54), a Faster R-CNN model (55), a result storage module (56), a retraining module (57) and a display module (6).

The working process of the invention is as follows: the material to be detected is sent into a spreading device (1), the spreading device (1) is connected with a conveyor belt (2), the material to be detected is spread on the conveyor belt (2) as uniformly and densely as possible and moved to an image acquisition hardware platform (3), the image acquisition hardware platform is a digital camera using a CMOS camera, the network communication image processing software platform (5) transmits the acquired image to the image acquisition reading module (51) for fixed cutting correction and then transmits the image to the foreign matter detection module (52), the corresponding detection of the acquired image is selected by an algorithm selection module (53) by using a rough detection module (54) or a Faster R-CNN model (55), if the collected image is detected to have foreign matters, the detection result is stored in a result storage module (56), and a display module (6) is connected with the result storage module (56) and displays the detected image information in real time for operators to monitor and follow-up check in real time; meanwhile, the foreign matter detection module transmits the area number with the foreign matter back to the removing device (4), and the material to be detected on the conveying belt (2) is divided into tobacco shreds (7) and the foreign matter (8) by the removing device (4).

The difference between the second embodiment and the first embodiment is that aiming at the tobacco shred foreign matter detection algorithm based on the improved Faster R-CNN model, as a large amount of image data is needed for model training, the initially available data volume is small and is a simulation image, a retraining module (57) is also designed when a software platform is built, the data of the material to be detected, which are collected by the image acquisition and reading module (51) and the result storage module (56), are further trained on the basis of the original model through the retraining module (57), and the generalization capability of the model is enhanced. Because the types of the tobacco foreign matters referred to in the design of the invention are not complete, the retraining module (57) also comprises a type updating model, and the types of the foreign matters are added when new foreign matters appear, so that the intelligent detection device for the tobacco foreign matters has more perfect capability of detecting the foreign matters.

The third embodiment of the invention, referring to fig. 2-10, is an intelligent detection method for foreign matters in tobacco shreds, which uses an intelligent detection device for foreign matters in tobacco shreds, preferentially uses a rough detection method for foreign matters based on image processing under the condition that the basic number of materials to be detected is large, and then uses a deep learning method based on a Faster R-CNN model, and comprises the following steps:

the method comprises the following steps: processing a disordered dark edge of the material to be detected in a large area by using a progressive median filtering method, and firstly expanding an image of an original image under the size of a 3 multiplied by 3 convolution kernel; then, carrying out primary median filtering by using a 15 multiplied by 15 convolution kernel; then using the convolution kernel of the parameter [2-1,5, -1,0, -1,0] to sharpen the image; finally, a second median filter is performed using a 15 × 15 or 9 × 9 convolution kernel.

Step two: binary separation is carried out on foreign matters (8) and cut tobacco (7) in the material to be detected by using a threshold value division method under an HSV color model; image segmentation is carried out according to the characteristic difference of the tobacco shreds (7) and the foreign matters (8) under the HSV color model, the disordered dark edges of the tobacco shreds are inhibited, and the calculation formula of conversion of the HSV color model is as follows:

V＝max(R,G,B)

wherein, R, G and B represent three channels of color, R represents the value of a red channel, G represents the value of a green channel, and B represents the value of a blue channel; each pixel point has color information and consists of three primary colors of red, green and blue light, namely three components, the maximum value and the minimum value of the three components are selected from R, G and B arrays, and the maximum value is defined as the value of brightness V; the saturation S is determined according to the difference value between the maximum value and the minimum value, the saturation S represents the degree of the color approaching the spectral color, one color can be regarded as the result of mixing certain spectral color and white, wherein the larger the proportion of the spectral color is, the higher the degree of the color approaching the spectral color is, the higher the saturation of the color is, the darker and gorgeous the color is, the white light component of the spectral color is 0, the saturation reaches the highest, the value range is usually 0-100%, and the larger the value is, the more saturated the color is; the chroma H is discussed according to different conditions of the color to which the maximum value belongs, the chroma H can be measured by angles, the value range is 0-360 degrees, the chroma H is calculated from red in a counterclockwise direction, the red is 0 degrees, the green is 120 degrees, the blue is 240 degrees, and the complementary colors of the red, the green and the blue are as follows: yellow 60 °, cyan 180 °, violet 300 °.

Step three: and (3) carrying out pixel positioning on the foreign matter (8) by using a positioning frame selection method based on boundary tracking, and carrying out space mapping according to the position relation between the image acquisition hardware platform view field and the conveyor belt. The positioning frame selection method based on boundary tracking comprises the following two steps: pixel region localization and spatial localization mapping.

First, pixel area positioning: referring to fig. 10, the frame of a picture is composed of the uppermost row, the lowermost row, the leftmost column, and the rightmost column of the picture in the binary image. With 4(8) connected scene, i.e. 1 pixel (pixel with gray value of 1) is 4(8) connected, and 0 pixel (pixel with gray value of 0) is 8(4) connected. In the 4(8) connected scene, if a 1 pixel has 0 pixels in its 8(4) connected domain, the 1 pixel is a boundary point. The boundary of a certain area is composed of a plurality of boundary points. Assume that there is a 1-pixel connected domain S1, a 0-pixel connected domain S2. If S2 directly surrounds S1, the boundary between S2 and S1 is called the outer boundary; if S1 directly surrounds S2, the boundary line between S2 and S1 is referred to as the hole boundary. Either the outer boundary or the hole boundary is composed of a number of 1 pixels, with 0 pixels not constituting an edge. The boundary starting point is divided into an outer boundary starting point and a hole boundary starting point. If a certain point satisfies both fig. 10(a) and fig. 10(b), the point is regarded as a boundary starting point. A complete boundary can be obtained from the boundary starting point by using a boundary tracking algorithm, and a new number is assigned to the complete boundary B each time a new complete boundary is obtained.

And then, by a raster scanning method, executing a boundary tracking algorithm while using raster scanning, thereby obtaining an output image, wherein the non-0 pixel region is the outline of the foreign matter (8), namely the pixel region of the foreign matter (8). The information of each foreign body (8) contour is composed of an array K, and the horizontal and vertical coordinates of each pixel point forming the boundary of the contour are recorded. The substantially rectangular area in which the foreign object (8) is located can be located according to the data provided by the array K. The rectangular area where each foreign object (8) is located is determined by four pieces of information, i.e., the abscissa (X) and the ordinate (Y) of the vertex at the upper left corner of the rectangle, and the Height (Height) and the width (Weight) of the rectangle. The specific calculation flow is shown in a formula.

X＝min(K[i][0])

Y＝min(K[i][1])

Height＝max(K[i][1])-Y

Weight＝max(K[i][0])-X

Secondly, space positioning mapping: according to the method, as shown in the figure 11, the image shot in the view field of the image acquisition hardware platform (3) is divided into pixel areas according to the space range, and in order to ensure the removal effectiveness, the pixel position where the foreign matter (8) is located is selected after the frame selection position, the left vertex and the right vertex of the foreign matter frame are selected to respectively judge the space areas, and the space area number where the foreign matter is located is obtained.

Step four: after the coarse detection, the detection rate of the algorithm is reduced or the false detection rate is improved due to the change of the material to be detected or the influence of other environmental factors in consideration of the limitation of the traditional image algorithm in solving the problem of foreign matter detection, namely the problem of poor algorithm universality. The corresponding image processing algorithm is designed according to different conditions, so that the algorithm redundancy is realized, the efficiency is reduced, and the detection intelligence is reduced by switching different algorithms according to different scenes, so that the foreign body characteristics are learned and distinguished from the background characteristics by training a deep network model by adopting deep learning based on an Faster R-CNN model, the detection capability is enhanced, and the application capability of the detection algorithm is improved at the same time, and the process is as follows:

the first step is as follows: aiming at the problem of insufficient tobacco shred material samples, a double-line data median augmentation method and a transfer learning technology are applied to enhance the generalization capability and the convergence speed of the model; firstly, a double-line data median augmentation method is used for off-line augmentation of image data, the double-line data median augmentation method refers to a mode combining off-line augmentation and on-line augmentation, namely a dark edge suppression algorithm of asymptotic median filtering in the step one is used, augmented data are put into a model, and the on-line augmentation is carried out in a training mode combining four transformation random combinations of translation, scaling, overturning and cutting. The two-line data median augmentation method is the prior art, and the effect graph of the two-line data median augmentation method is shown in figure 5.

And then, applying a transfer learning technology, wherein the transfer learning technology refers to another method for solving the problems of insufficient generalization capability, low convergence speed and the like of a small sample data set in training, the transfer learning technology is mainly used for learning related knowledge from a source domain and applying the related knowledge to the learning of a target domain to improve the learning capability of the target domain, and model parameters of a network trained in a certain large-volume data set are transferred to network nodes to be trained in deep learning so as to serve as initial node parameters of the training network.

The second step is that: aiming at the problem of missed detection caused by variable foreign matter scale and small common scale, a two-way feature pyramid network combined with a bottom-up path is introduced, the learning capability of the model on multi-scale and small-scale features is improved, the two-way feature pyramid network is used for expanding feature extraction, so that extracted features are more and richer, and the two-way feature pyramid network is a simple two-way feature pyramid network in a PANet model or a two-way feature pyramid network in an EfficientDet model.

Referring to fig. 4, the feature pyramid network is a top-down process, which transfers high-level strong semantic features to enhance the feature map of the whole network. However, such enhancement is only for semantic information, and is enhancement of the upper layer features to the lower layer features, and the feature information for positioning of the lower layer is not effectively transferred. In this regard, a crack still exists between the upper and lower layers of the feature. In the PANet model proposed in 2018, the transfer structure of the feature pyramid network is improved for the purpose of improving the accuracy of image instance segmentation, and a bottom-up propagation path is added in a top-down fusion path. The path is added to supplement the feature pyramid network, and is used for shortening a feature transmission path, so that features rich in positioning information at a lower layer can be completely transmitted to a feature map at a higher layer, and the accuracy of image instance segmentation on the feature map at the higher layer is improved. In the target detection, for trivial target objects such as tobacco foreign matters, the objects are sometimes densely stacked in a certain area of the material in the image, and in order to detect the stacked foreign matters more precisely one by one during the detection, the structure can be used for improving the thinning degree of the detection frame selection. A schematic diagram of a simple two-way feature pyramid network structure in the PANet model is shown in fig. 4. And the dimension of each layer is kept unchanged, and the P2-P5 layers are respectively subjected to one round of feature fusion, and the feature information transmitted from the bottom layer is sequentially superposed.

The propagation structure of the feature pyramid network is further improved in the EfficientDet model proposed by *** team. Compared with the PANet model which is simply added with a bottom-up propagation path, the EfficientDet model is additionally added with an additional propagation path between the input and the output of the feature pyramid network at the same level on the basis, so that the feature graph of each layer is fused with richer feature information without increasing additional calculated amount. In addition, the EfficientDet model introduces a weight concept to the propagation path, the feature maps in different layers are not simply and directly added, but weight values of different input feature maps are learned through a network so as to adapt to the importance degree of the features in different layers to a final detection result. The invention refers to the propagation path structure of the two-way characteristic pyramid in the EfficientDet model by combining the two-way characteristic pyramid network reference of the bottom-up path, does not add weight information, and does not adopt a cycle structure. N2-N5 are formed by fusing P2-P5 characteristic diagrams in a traditional characteristic pyramid network with two additional propagation paths, and P6 is still obtained by maximum pooling sampling of a P5 layer without modification. Each layer is equivalent to a convolution layer with different sizes to obtain feature maps with different scales, and then fusion is carried out, so that the extracted features after the network processing contain richer information. The third step: referring to fig. 8-9, in order to obtain higher target framing accuracy, the pooling algorithm of the pooling feature map module is changed to a bilinear interpolation method, so that the mismatching problem caused by quantization is relieved. The bilinear interpolation method comprises a regional suggestion network (RPN) and a backbone network (backbone), wherein the regional suggestion network (RPN) firstly generates a part of candidate frames, then screens the candidate frames to obtain suggestion frames, and the suggestion frames are divided into a detection frame and a target frame; the suggestion frame has corresponding pixel point coordinates on the original image and is floating point type coordinates; when the suggestion frame is mapped into a Feature map (Feature map) output by a backbone network (backbone), retaining the floating point type coordinates of the corresponding suggestion frame region in the Feature map (Feature map), and not quantizing the boundary; when the pooling operation is carried out to divide the suggested frame area into a plurality of areas (sections), the boundary of each area (section) is not subjected to quantization processing; sampling in each region (section), obtaining a corresponding sampling value by using a bilinear interpolation method for the sampling center point coordinate of each region (section), reserving the maximum sampling value as the maximum pooling result of the region, wherein the sampling center point is the coordinate of the sampling center point position of the central small frame in the nine small frames in the circle in the figure nine; the candidate box, the suggestion box, the target box, the detection box and the backbone network all refer to the steps of Faster R-CNN of FIG. 8.

Compared with the operation of performing quantization and rounding on a region of interest Pooling module (ROI Pooling) during two times of calculation, floating point number information is reserved in a region of interest matching (ROI Align) in the calculation process, floating point type information is also reserved at each sampling point extracted in the Pooling operation, data errors caused by quantization and rounding are reduced, and the precision of effective target region positioning frame selection is improved. The backbone network (backbone) is a network consisting of a series of convolutional layers, pooling layers and full connection layers and is used for extracting features of an input image to generate a Feature map, then generating a suggestion frame by a regional suggestion network (RPN), and finally classifying the suggestion frame through the full connection layers and a softmax excitation function and outputting the position of the suggestion frame. The backbone network is a feature extraction module, and the output is a feature graph. The feature in Classification in FIG. 8 is the full connection layer.

The formula of the back propagation is

The two-line data median augmentation method and the transfer learning technology comprise the following steps:

(1) the fast R-CNN model puts the collected image into a Feature extraction module to carry out multilayer convolution, excitation and pooling to obtain deep features, the Feature extraction module is in front of the Feature map, which is a problem of data progressive step, and the Feature extraction module can obtain the Feature map only by carrying out Feature extraction, so the Feature extraction module is in front of the Feature map. Common networks for the feature extraction module include VGG series, Resnet series, Densenet network and the like;

(2) the method for selecting the Search of the traditional R-CNN network is abandoned when the candidate frame is extracted by the fast R-CNN network, a regional suggestion network (RPN) is used, the candidate frame is extracted according to a set anchor point (anchors) and positive and negative samples are classified through a softmax excitation function, boundary regression is carried out on the positive samples selected by the suggestion frame on the other side, calculation and classification are carried out through the softmax excitation function, the softmax excitation function generates a numerical value in an interval of 0-1, intervals are divided according to a set classification problem, and the calculated numerical value belongs to which class in which interval, wherein the formula is as follows:

x is the input vector value;

(3) the two are unified to generate a suggestion frame at a suggestion frame generation layer (Proposal) to finish the coarse positioning of the material to be detected;

(4) inputting a suggestion frame output by a regional suggestion network (RPN) and an original feature map output by a feature extraction module into a region of interest Pooling module (ROI Pooling), converting a feature map region corresponding to the input suggestion frame into a new feature map with consistent size at the position of the region of interest Pooling module (ROI Pooling) after the region suggestion network is generated by the suggestion frame, thereby improving the detection efficiency;

(5) the classification regression module is a part behind the region-of-interest Pooling module (ROI Pooling), the classification regression module performs classification recognition on the feature map of each suggestion box through a full connection layer (feature) and a softmax excitation function, the confidence coefficient of each suggestion box in each category is calculated, and the category with the highest confidence coefficient is taken as the final category of a target framed by the suggestion box;

(6) when the training is finished and the practical application is carried out, a large amount of redundancy exists in the detected target frame, and at the moment, a non-maximum value suppression method is adopted for pruning so as to generate a more accurate and concise detection frame. The non-maximum suppression algorithm is to suppress elements which are not maximum, search local maximum, and the process is as follows: assuming the set A, B first, then the degree of overlap IoU ═ a ═ B)/(a ═ B), then in the case of homogeneous categories, the scores of each of the target frames are sorted from large to small, taking out the score that is the highest among them; homogeneous category refers to the same object, such as a person is a category and a dog is a category; the categories here are generic. The categories are divided into two categories, namely, the tobacco shreds (7) and the foreign matters (8); then, the overlapping degree IoU is calculated with all other remaining regions of interest, which refer to the region where the detection target may exist, for example, the region suggestion network (RPN) in fig. 8, and the resulting suggestion box generation layer (propofol) is the region of interest. Removing the region of interest border with the value of the current region of interest border overlapping IoU being greater than a set overlapping IoU threshold; and finally, continuously selecting the region of interest with the highest score from the unprocessed region of interest borders, and repeating the process until all the regions of interest in each category are found.

The above-mentioned embodiments only express several embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention.

Claims

1. The intelligent detection device for the foreign matters in the tobacco shreds is characterized by comprising a spreader, a conveyor belt, an image acquisition hardware platform, a removing device and an image processing software platform, wherein the image processing software platform comprises an image acquisition reading module, a foreign matter detection module, an algorithm selection module, a rough detection module, a Faster R-CNN model and a result storage module; the spreading device is connected with the conveying belt, the materials to be detected are evenly spread on the conveying belt and moved to the image acquisition hardware platform, the image acquisition hardware platform is in network communication with the image processing software platform, the acquired images are transmitted to the image acquisition reading module to be fixedly cut and corrected and then transmitted to the foreign matter detection module, the algorithm selection module is used for selecting the rough detection module or the fast R-CNN model to correspondingly detect the acquired images, the detection results are stored in the result storage module, and meanwhile, the foreign matter detection module transmits the region numbers of the foreign matters to the removing device.

2. The intelligent tobacco shred foreign matter detection device according to claim 1, further comprising a retraining module, wherein the retraining module is used for further training the data of the material to be detected, which are collected by the image acquisition and reading module and the result storage module, on the basis of an original model.

3. The intelligent tobacco shred foreign matter detection device according to claim 1, wherein the retraining module comprises a category updating model, and foreign matter categories are added when new foreign matters appear.

4. The intelligent tobacco shred foreign matter detection device according to claim 1, wherein the intelligent tobacco shred foreign matter detection device is connected with a display module, is connected with the result storage module, and displays the detected image information in real time for an operator to monitor in real time and view subsequently; the image acquisition hardware platform is a digital camera using a CMOS camera.

5. An intelligent tobacco foreign matter detection method is characterized in that the intelligent tobacco foreign matter detection device of any one of claims 1 to 4 is used, under the condition that the base number of materials to be detected is large, a foreign matter rough detection method based on image processing is preferentially used, then a deep learning method based on an Faster R-CNN model is used, and the intelligent tobacco foreign matter detection method is carried out according to the following steps:

step four: deep learning based on the Faster R-CNN model is adopted, foreign body characteristics are learned and distinguished from background characteristics through training of a deep network model.

6. The intelligent detection method for foreign matters in tobacco shreds according to claim 5, wherein the first step specifically comprises the following steps: firstly, expanding an image of an original image under the size of a convolution kernel of 3 multiplied by 3; then, carrying out primary median filtering by using a 15 multiplied by 15 convolution kernel; then using the convolution kernel of the parameter [2-1,5, -1,0, -1,0] to sharpen the image; finally, a second median filter is performed using a 15 × 15 or 9 × 9 convolution kernel.

7. The intelligent detection method for the foreign matters in the tobacco shreds according to claim 6, wherein the second step specifically comprises image segmentation according to the characteristic difference of the tobacco shreds and the foreign matters under an HSV color model, and the calculation formula of conversion of the HSV color model is as follows:

V＝max(R,G,B)

wherein, R, G and B represent three channels of color, R represents the value of a red channel, G represents the value of a green channel, and B represents the value of a blue channel; each pixel point has color information and consists of three primary colors of red, green and blue light, namely three components, the maximum value and the minimum value of the three components are selected from R, G and B arrays, and the maximum value is defined as the value of brightness V; the saturation S is determined according to the difference value between the maximum value and the minimum value, the value range is 0-100%, and the larger the value is, the more saturated the color is; the chroma H is discussed according to different situations of the color to which the maximum value belongs, the chroma H is measured by angles, the value range is 0-360 degrees, the chroma H is calculated from red in a counterclockwise direction, the red is 0 degrees, the green is 120 degrees, the blue is 240 degrees, and the complementary colors of the red, the green and the blue are as follows: yellow 60 °, cyan 180 °, violet 300 °.

8. The intelligent detection method for foreign matters in tobacco shreds according to claim 7, wherein the fourth step specifically comprises:

the first step is as follows: firstly, performing off-line augmentation on image data by using a double-line data median augmentation method, putting the augmented data into a model, performing on-line augmentation by combining four transformation random combinations of translation, scaling, overturning and cutting during training, and then migrating model parameters trained in a certain massive data set by a network into a network node to be trained by using a migration learning technology to serve as initial node parameters of the training network;

the second step is that: introducing two-path characteristic pyramid network extension characteristic extraction combined with a bottom-up path;

the third step: using a bilinear interpolation method, wherein the bilinear interpolation method comprises a regional suggestion network and a backbone network, the regional suggestion network firstly generates a part of candidate frames, then screening the candidate frames to obtain suggestion frames, and the suggestion frames are divided into a detection frame and a target frame; the coordinates of the corresponding pixel points of the suggestion frame on the original image are floating point type coordinates; when the suggestion frame is mapped to the feature map output by the backbone network, the floating point type coordinates of the corresponding suggestion frame area in the feature map are reserved, and the boundary is not quantized; when the pooling operation is carried out to divide the suggested frame area into a plurality of areas, the boundary of each area is not subjected to quantization processing; sampling in each area, obtaining a corresponding sampling value by using a bilinear interpolation method for the coordinates of the sampling center point of each area, and reserving the maximum sampling value as the maximum pooling result of the area;

the formula of the back propagation is

9. The intelligent tobacco shred foreign matter detection method according to claim 8, wherein the two-line data median value augmentation method and the transfer learning technology comprise the following steps:

where wy.x is the input vector value;

(3) uniformly generating the suggestion frame in the step (2) to finish the coarse positioning of the material to be detected;

(6) when practical application is carried out after training is finished, a non-maximum suppression algorithm is adopted for pruning, and a more accurate and concise detection frame is generated; the non-maximum suppression algorithm is to suppress elements which are not maximum, search local maximum, and the process is as follows: assuming the set A, B first, then the degree of overlap IoU ═ a ═ B)/(a ═ B), then in the case of homogeneous categories, the scores of each of the target frames are sorted from large to small, taking out the score that is the highest among them; then, the overlapping degree IoU is calculated with all other remaining interested areas, wherein the interested areas refer to areas where detection targets possibly exist, and the interested area border with the overlapping degree IoU of the current interested area border being larger than the set overlapping degree IoU threshold value is removed; and finally, continuously selecting the region of interest with the highest score from the unprocessed region of interest borders, and repeating the process until all the regions of interest in each category are found.

10. The intelligent tobacco foreign matter detection method according to claim 8, wherein the two-way characteristic pyramid network is a simple two-way characteristic pyramid network in a PANET model, and the two-way characteristic pyramid network is preferably a two-way characteristic pyramid network in an EfficientDet model.