CN116434002A - Smoke detection method, system, medium and equipment based on lightweight neural network - Google Patents
Smoke detection method, system, medium and equipment based on lightweight neural network Download PDFInfo
- Publication number
- CN116434002A CN116434002A CN202310297290.XA CN202310297290A CN116434002A CN 116434002 A CN116434002 A CN 116434002A CN 202310297290 A CN202310297290 A CN 202310297290A CN 116434002 A CN116434002 A CN 116434002A
- Authority
- CN
- China
- Prior art keywords
- smoke
- smoke detection
- image
- module
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 239000000779 smoke Substances 0.000 title claims abstract description 207
- 238000001514 detection method Methods 0.000 title claims abstract description 130
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 27
- 238000012549 training Methods 0.000 claims abstract description 63
- 230000001537 neural effect Effects 0.000 claims abstract description 41
- 238000000034 method Methods 0.000 claims abstract description 27
- 238000012360 testing method Methods 0.000 claims abstract description 27
- 230000004927 fusion Effects 0.000 claims abstract description 16
- 238000007781 pre-processing Methods 0.000 claims abstract description 16
- 230000006870 function Effects 0.000 claims description 18
- 238000000605 extraction Methods 0.000 claims description 11
- 238000005520 cutting process Methods 0.000 claims description 10
- 238000003860 storage Methods 0.000 claims description 10
- 238000005070 sampling Methods 0.000 claims description 8
- 230000015654 memory Effects 0.000 claims description 7
- 230000002776 aggregation Effects 0.000 claims description 3
- 238000004220 aggregation Methods 0.000 claims description 3
- 238000011478 gradient descent method Methods 0.000 claims description 3
- 238000002372 labelling Methods 0.000 claims description 3
- 238000011179 visual inspection Methods 0.000 claims description 2
- 230000000007 visual effect Effects 0.000 abstract description 5
- 238000004364 calculation method Methods 0.000 abstract description 4
- 230000006872 improvement Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 29
- 239000011159 matrix material Substances 0.000 description 11
- 238000013527 convolutional neural network Methods 0.000 description 8
- 230000000694 effects Effects 0.000 description 8
- 230000005540 biological transmission Effects 0.000 description 5
- 210000000988 bone and bone Anatomy 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000011835 investigation Methods 0.000 description 3
- 238000011176 pooling Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000011218 segmentation Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000012790 confirmation Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 230000000875 corresponding effect Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000009792 diffusion process Methods 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 210000003128 head Anatomy 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 230000006698 induction Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000007500 overflow downdraw method Methods 0.000 description 1
- 238000011020 pilot scale process Methods 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 210000001525 retina Anatomy 0.000 description 1
- 239000002689 soil Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/766—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using regression, e.g. by projecting features on hyperplanes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Image Analysis (AREA)
Abstract
The application relates to a smoke detection method, a system, a medium and equipment based on a lightweight neural network, and relates to the technical field of neural network visual detection, wherein the method comprises the steps of acquiring a smoke image and preprocessing the smoke image to obtain a data set, a test sample set and a training sample set; building a lightweight neural smoke detection model, and training the smoke detection model by using the training sample set to obtain a trained lightweight neural smoke detection model; and analyzing the test sample set by using the trained lightweight neural smoke detection model to obtain a detection value, wherein the detection value comprises smoke category and smoke coordinate information. Through the improvement of a backbone network, a multi-scale feature fusion module and a classification regression module in the smoke detection model, the calculation amount of smoke detection of the model can be reduced, the model is lighter, and meanwhile, the accuracy and the timeliness of smoke detection are improved.
Description
Technical Field
The invention relates to the technical field of neural network vision detection, in particular to a smoke detection method, system, medium and equipment based on a lightweight neural network.
Background
With the continuous and rapid increase of the total civil electricity consumption, the construction of power grids is continuously expanded, and the electric power is used as a high-risk industry, so that popularization and application of fireproof and safety technologies are increasingly emphasized. In order to ensure the safety of the power transmission line, the design and improvement of a fire disaster prevention method with timeliness and reliability are provided for the induction of the power transmission line.
The fire disaster is often accompanied by the appearance of characteristics such as smoke, high temperature, flame and the like, and the fire disaster detection technology is used as a parameter for fire disaster identification, so that the fire disaster is automatically detected as soon as possible and an alarm is given out, and the lives and properties of people are saved to the greatest extent. At present, mature products such as a temperature sensing detector, a smoke sensing detector, a flame detector and the like are respectively used for measuring smoke, temperature and flame generated in the fire disaster process. Early in a fire, smoke features are more visible than flame features and are more easily identified by the sensor. The detection of fire disaster conditions according to fire smoke images has become an important research direction for the current transmission line fire detection. If the smoke detection of the fire can be carried out before the fire occurs, the fire situation can be timely dealt with, and the loss of the power transmission line is reduced to the minimum. Therefore, it is of great importance to design a smoke detector with timeliness and reliability. CNN shows its particularly powerful function as convolutional neural network structures become deeper and wider. However, extending the architecture of neural networks typically results in more computations, which makes it impossible for most people to afford computationally intensive tasks such as object detection.
By searching, the closer prior art is obtained as follows:
the patent document with the publication number of CN109961042B discloses a smoke detection method combining a deep convolutional neural network and a visual change chart, wherein a suspected smoke region is initially detected by the deep convolutional neural network. Based on the basis, the visual change diagram is constructed based on the physical characteristics of smoke diffusion and based on the video motion change, and the SVM classifier is adopted to realize the secondary judgment of the smoke area.
The application publication number CN110956611A discloses a smoke detection method integrating a convolutional neural network, which comprises a suspected smoke acquisition module, a suspected smoke confirmation module and a smoke alarm module, wherein the suspected smoke acquisition module acquires an image containing suspected smoke after detecting an image acquired in real time through a Faster R-CNN model, the suspected smoke detection module detects a candidate region of the suspected smoke image by using the convolutional neural network to determine the image containing smoke, the smoke alarm module responds to the result processed by the suspected smoke confirmation module, so that the smoke detection cost is greatly reduced, the recognition efficiency and accuracy are improved, the recognition flexibility is high, no additional storage and calculation cost is avoided, the complexity of a system is reduced, and the smoke detection method is efficient and energy-saving.
According to the technical scheme, the smoke is detected through the neural network, the detection reliability of the smoke is improved, but how to extract the smoke characteristics in complex and changeable weather environments and specific scenes is a main challenge in the current smoke detection field. In addition, improving the timeliness of smoke detection and early warning in time to avoid loss is also a great difficulty.
Disclosure of Invention
In order to improve the detection effect of a smoke detection model, the application provides a smoke detection method, a system, a medium and equipment based on a lightweight neural network.
In a first aspect, the smoke detection method based on the lightweight neural network provided by the application adopts the following technical scheme:
a smoke detection method based on a lightweight neural network, comprising:
acquiring a smoke image and preprocessing the smoke image to obtain a data set, a test sample set and a training sample set;
building a lightweight neural smoke detection model, and training the lightweight neural smoke detection model by using the training sample set to obtain a trained lightweight neural smoke detection model;
and analyzing the test sample set by using the trained lightweight neural smoke detection model to obtain a detection value, wherein the detection value comprises smoke category and smoke coordinate information.
The further technical proposal is that: when the smoke image is preprocessed, the method comprises the following steps:
cutting the acquired initial image according to a threshold cutting method to obtain a smoke image only containing a smoke part;
labeling the smoke parts in the plurality of smoke images, wherein each labeled smoke image generates an xml file comprising a picture name and smoke position coordinates, and one xml file is a label;
all the labels are converted into txt format files and json format files to be used as data sets, and images are extracted from the data sets according to set proportions to be used as training sample sets and test sample sets.
The further technical proposal is that: the training of the lightweight neural smoke detection model by using the data set, when the trained lightweight neural smoke detection model is obtained, comprises the following steps:
loading a pre-training weight file, and initializing a smoke detection model; the pre-training file is a history accumulated training parameter;
inputting the training sample set into an initialized smoke detection model for training, outputting predicted smoke boundary frame coordinates and real smoke boundary frame coordinates, and generating a loss function loss;
returning to the loss function loss, updating training parameters according to a gradient descent method, and performing iterative operation;
and stopping training after the iterative operation reaches the preset iterative times to obtain the lightweight neural smoke detection model.
The further technical proposal is that: the lightweight neural smoke detection model comprises a backbone network feature extraction module, a multi-scale feature fusion module and a classification regression module;
the method for analyzing the test sample set by using the lightweight neural smoke detection model comprises the following steps of:
inputting images in a test sample set to a backbone network feature extraction module, sequentially passing through a Unit2 downsampling Unit and a plurality of Unit1 basic units, performing one-time splicing operation on the obtained output features and the output of the Unit2 downsampling Unit, and then obtaining a plurality of feature images through a channel shuffling module;
after receiving a plurality of feature images, the multi-scale feature fusion module adjusts the channel number and the length and width of the plurality of feature images to be the same as the output size through a 1X 1 convolution and up-sampling unit in a lateral connection mode, and adds and aggregates the obtained plurality of output feature images to obtain an aggregated feature image;
and inputting the aggregated feature map into the region candidate network by using the classification regression module, generating an anchor frame, and obtaining smoke boundary frame coordinates by using the size of the anchor frame as a priori frame and carrying out frame regression prediction.
In a second aspect, the application discloses a smoke detection system based on a lightweight neural network, which adopts the following technical scheme:
a smoke detection system based on lightweight neural network comprises
The image acquisition module is used for acquiring image information;
the image preprocessing module is used for preprocessing the acquired image information to obtain a data set, a test sample set and a training sample set; and
the light neural smoke detection model is used for analyzing input image information to obtain detection values, and the detection values comprise smoke types and smoke coordinate information.
The further technical proposal is that: the lightweight neural smoke detection model includes:
the backbone network feature extraction module is used for acquiring a plurality of feature graphs;
the multi-scale feature fusion network module is used for fusing the feature images to generate an aggregate feature image; and
and the classification regression module is used for obtaining the finally predicted smoke boundary frame coordinates according to the aggregation feature map.
The further technical proposal is that: the image preprocessing module comprises:
the image clipping unit clips the acquired initial image according to a threshold cutting method to obtain a smoke image only containing a smoke part;
the file generation unit is used for generating an xml file comprising a picture name and a smoke position coordinate for each marked smoke image, wherein one xml file is a label;
the sample set generating unit is used for converting all the labels into txt format files and json format files to serve as data sets, and extracting images from the data sets according to set proportions to serve as training sample sets and testing sample sets.
The further technical proposal is that: the model training module is used for carrying out iterative training on the lightweight neural smoke detection model.
In a third aspect, the present application discloses a computer readable storage medium, which adopts the following technical scheme:
a computer-readable storage medium, comprising:
a program which, when executed by a processor, is executed by a lightweight neural network based smoke detection method as claimed in any one of the first aspects.
In a fourth aspect, the application discloses a smoke risk visual detection device adopting the following technical scheme:
a smoke risk visual inspection apparatus comprising
One or more memories for storing instructions; and
one or more processors to invoke and execute the instructions from the memory to perform the lightweight neural network based smoke detection method according to any of the first aspects.
In summary, the present application includes the following beneficial technical effects:
through the improvement of a backbone network, a multi-scale feature fusion module and a classification regression module in the smoke detection model, the calculation amount of smoke detection of the model can be reduced, the model is lighter, and meanwhile, the accuracy and the timeliness of smoke detection are improved.
Drawings
FIG. 1 is a flow chart of an overall method provided by an embodiment of the present application;
FIG. 2 is a flow chart of a method for preprocessing an initial image provided in an embodiment of the present application;
FIG. 3 is a flowchart of a neural smoke detection model training method provided in an embodiment of the present application;
FIG. 4 is a flowchart of a neural smoke detection model detection method provided in an embodiment of the present application;
FIG. 5 is a schematic diagram of the components of a neural smoke detection model provided in an embodiment of the present application;
fig. 6 is a block diagram of a backbone network provided in an embodiment of the present application;
fig. 7 is a block diagram of a backbone network provided in an embodiment of the present application;
fig. 8 is a construction unit structure diagram of a backbone network provided in an embodiment of the present application;
FIG. 9 is a block diagram of a parallel attention module provided by an embodiment of the present application;
FIG. 10 is a block diagram of a spatial attention module and a channel attention module provided in an embodiment of the present application;
FIG. 11 is a block diagram of a multi-scale feature fusion network provided by an embodiment of the present application;
FIG. 12 is a classification regression module diagram provided by an embodiment of the present application;
fig. 13 is an original image to be detected provided in an embodiment of the present application;
fig. 14 is a diagram of detection effect of smoke in an original image according to an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. The following description of at least one exemplary embodiment is merely exemplary in nature and is in no way intended to limit the application, its application, or uses. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application, but the present application may be practiced in other ways other than those described herein, and persons skilled in the art will readily appreciate that the present application is not limited to the specific embodiments disclosed below.
Example 1
The invention discloses a smoke detection method based on a lightweight neural network, which comprises the following steps of.
Please refer to fig. 1:
step S100: and acquiring an initial image and preprocessing a smoke image to obtain a data set, a test sample set and a training sample set.
When the initial image is acquired, a smoke picture near the high-voltage transmission line can be shot through a professional color industrial camera with high pixels to serve as the initial image for smoke detection. The smoke of the initial image varies in color, concentration, and visibility. These images are recorded under various weather conditions, including sunny days, cloudy days, and cloudy. The method comprises the following steps when preprocessing an initial image.
Please refer to fig. 2:
step S110: and cutting the acquired initial image according to a threshold cutting method to obtain a smoke image only containing a smoke part.
And cutting all the original images containing the smoke to be detected, which are acquired by the industrial camera, by using a threshold segmentation method.
It will be appreciated that the background of the original image smoke is complex and has a significant amount of interference, such as soil, sky, light, buildings and vegetation. The smoke area in the initial picture only occupies a small part, the information proportion acting on smoke detection is small, if the size of the initial image is directly reduced in an equal proportion, a large amount of redundant information is calculated, and the timeliness requirement of the smoke detection is difficult to meet.
The image segmentation method is to select an optimal threshold according to the difference of average brightness of a background area and a smoke area in a gray level histogram of a smoke initial image. And separating the smoke outline part and the background of each original image of the smoke to be detected according to the image threshold value to obtain a processed image only containing the smoke part. The threshold segmentation method is used, so that the dependence on computer hardware is reduced, the detection precision is improved, the data volume is compressed, and the analysis and processing steps are simplified.
Step S120: and labeling the smoke parts in the plurality of smoke images, wherein each labeled smoke image generates an xml file comprising a picture name and smoke position coordinates, and one xml file is a label.
When the smoke part in the smoke image is marked, the smoke part in the smoke image can be marked manually, for example, by using photoshop to mark the smoke image manually, so that Chinese labels are added in the smoke image.
Step S130: all the labels are converted into txt format files and json format files to be used as data sets, and images are extracted from the data sets according to set proportions to be used as training sample sets and test sample sets.
In this embodiment, 20% of the images are extracted from the dataset as test sample sets and the remaining dataset as training sample sets.
Step S200: and establishing a lightweight neural smoke detection model, and training the smoke detection model by using the training sample set to obtain a trained lightweight neural smoke detection model.
The lightweight neural smog detection model after training is obtained by training the lightweight smog model.
Please refer to fig. 3:
step S210: and loading a pre-training weight file, and initializing a smoke detection model.
It should be appreciated that the pre-training weight file works poorly if training is started from 0. Therefore, the pre-training weight file adopts historical accumulated training parameters, namely model parameters which are disclosed at present and have good effects, and initializes momentum, batch processing size, weight value and iteration period, wherein the weight attenuation value is 0.0005, and the momentum value is 0.9. To obtain stable performance, the initial learning rate was set to 0.01, and an exponentially decaying learning rate was used, with a decay index of 0.9; the batch size is set to be 16, the iteration period is 150, and the parameters are input into the lightweight neural smoke detection network to improve the training effect.
Step S211: inputting the training sample set into an initialized lightweight neural smoke detection model for training, outputting predicted smoke boundary frame coordinates and real smoke boundary frame coordinates, and generating a loss function loss;
step S212: returning to the loss function loss, updating training parameters according to a gradient descent method, and performing iterative operation.
The loss function (loss function) is an operation function for measuring the difference degree between the predicted value f (x) and the true value Y of the model, and is a non-negative real value function, and the smaller the loss function is, the better the robustness of the model is.
The loss function is mainly used in the training stage of the model, after training data of each batch are sent into the model, a predicted value is output through forward propagation, and then the loss function calculates a difference value between the predicted value and a true value, namely the loss value. After the loss value is obtained, the model updates each parameter through back propagation to reduce the loss between the true value and the predicted value, so that the predicted value generated by the model is close to the true value, and the learning purpose is achieved.
Step S213: and stopping training after the iterative operation reaches the preset iterative times to obtain the lightweight neural smoke detection model.
If the iteration number is 150 in this embodiment, the training is stopped after the iteration number reaches 150.
Referring to fig. 5, the lightweight smoke detection model includes a backbone portion and a detection portion connected in sequence. The backbone part is a feature extraction network module integrated with a parallel grouping attention mechanism module, and the detection part is an improved Light-Head R-CNN, and is divided into a multi-scale feature fusion network module and a classification regression module. First, the resolution of the input smoke image is adjusted, and the image is sent to the backbone network to extract the characteristics. Wherein C3 comes from the output characteristic diagram of the backup network Cross Stage2, C4 comes from the output characteristic diagram of the backup network Cross Stage3, C5 comes from the output characteristic diagram of the backup network Cross Stage4, and Cg is the global characteristic diagram obtained by the global average pooling layer of C5. The four feature images are input into a multi-scale feature fusion network module, and the classification regression module and the FC full-connection layer are used for finally obtaining the type and the position of the smoke.
The lightweight neural smoke detection model comprises a backbone network feature extraction module, a multi-scale feature fusion module and a classification regression module. The method comprises the following steps when a test sample set is analyzed by using a lightweight neural smoke detection model to obtain a detection value.
Please refer to fig. 4:
step S220: and inputting the images in the test sample set to a backbone network feature extraction module, sequentially passing through a Unit2 downsampling Unit and a plurality of Unit1 basic units, performing one-time splicing operation on the obtained output features and the output of the Unit2 downsampling Unit, and then obtaining a plurality of feature images through a channel shuffling module.
Specifically, as shown in fig. 6, the backbone network structure diagram includes Conv1 convolution operation, a max pooling layer, a cross stage2 sub-module, a cross stage3 sub-module, a cross stage4 sub-module, conv5 convolution operation, and a global average pooling layer, which are sequentially connected.
The cross stage module of the backbone network is shown in fig. 7, and is composed of a downsampling module, a ShuffleNetV2 original feature extraction module and a Channel Shuffle module (Channel Shuffle). Substitution of Unit2 for the original cross-phase local network requires additional convolution to achieve downsampling. The input feature map sequentially passes through a Unit2 downsampling Unit and k Unit1 basic units, the obtained output features and the output of the Unit2 downsampling Unit are subjected to one-time splicing operation, and then the feature map is obtained through a channel shuffling module. Dividing an input feature map into two parts x in a channel dimension by a slicing operation 0 ＝[x 0′ ,x 0″ ]。x 0″ Obtaining a characteristic diagram x through k units 1 k Then the characteristic diagram x 0′ And x k The feature images x are obtained by splicing together in the channel dimension c Feature map x c Obtaining an output characteristic diagram x through a channel shuffling module s Represented by the following formula:
fk represents the operating function of the k-th layer of the network; xk represents the output feature map of the kth layer; x0, x1, x2, …, xk-1 represents the output profile of the previous layers; wk represents the weight of the kth layer;
representing convolution; [ x ]' 0 ,x k ]Representing Xk and x0', i.e., x, concatenated together c ；/>
Representing the channel shuffling module.
x 0′ And x 0′ The gradients across both branches are independent. Both branches do not reuse gradient information used by the other when updating weights. The structure of the cross stage2 is a down sampling Unit2 and 4 Unit1 basic units, the structure of the cross stage3 is a down sampling Unit2 and 8 Unit1 basic units, and the structure of the cross stage4 is a down sampling Unit2 and 4 Unit1 basic units. Backbone network structure information is shown in table 1:
table 1 details of backbone network
The structure composition of the building Unit of the backbone network is shown in fig. 8, the structure of the Unit basic Unit is that an input feature map is divided by channels, one branch keeps identical mapping, the other part sequentially passes through 1×1 convolution, 3×3 depth separable convolution and 1×1 convolution, and then one-time splicing operation is carried out on the other part and the part keeping identical mapping, the obtained output is introduced into a parallel grouping attention module, and then the feature map is obtained by a channel shuffling module (the convolution structure is composed of a convolution layer, batch standardization operation BN, an activation function Relu and the like). The structure of the Unit2 downsampling Unit is that one part is input characteristic diagram, the input characteristic diagram is subjected to 5×5 depth separable convolution sum and 1×1 convolution, the other part is subjected to 1×1 convolution, 5×5 depth separable convolution sum and 1×1 convolution, and the operation identical to that of the Unit1 is executed after the input characteristic diagram is spliced. Because the downsampling unit does not adopt a channel segmentation strategy, a characteristic diagram with doubled output channels is obtained.
The structure diagram of the parallel attention module is shown in fig. 9, and the feature grouping operation is performed, so that the input feature diagram is divided into a plurality of groups in the channel dimension to obtain a plurality of sub-features, and the channel of each sub-feature is divided into two branches. The spatial attention module and the channel attention module are shown in fig. 10, one branch passes through the spatial attention module, and an input feature map a (c×h×w) is first input into the convolution module to generate B (c×h×w) and C (c×h×w), and B and C are converted into (c×n) dimensions through reshape operation, where n=h×w, and N is the number of pixels. Then, the B matrix is transposed and multiplied by the C matrix, the result is input into softmax, a spatial attention diagram S (n×n) is obtained, two position similarities are linearly and positively correlated with the magnitude of Sji value, and the output is:
wherein Sji is the element of the j-th row and i-th column of the matrix; bi is the ith column of the B matrix; ci is the j-th column of the C matrix.
A is input into another convolution layer to generate a new feature map D (C.times.H.W), and is converted into C.times.N through reshape operation and then multiplied by the transpose of the space attention force diagram S, so that a matrix with the size of C.times.N is obtained, and the matrix is converted into the original size of C.times.H.times.W through reshape operation. This matrix is multiplied by a coefficient alpha and then the original signature a is added. The output is:
wherein Ej is the output spatial attention profile; di represents the ith element in D; aj represents the j-th element in a; the alpha value is a learnable parameter or a training parameter. The initial value of alpha is 0, which is updated continuously by training iterations.
The other branch is passed through the channel attention module, the feature map a (c×h×w) is converted into a matrix of c×n through reshape operation, and the output is:
ai represents the ith channel of A, aj represents the jth channel of A, xji measures the effect of the ith channel on the jth channel.
This notice is a matrix multiplication of the matrix a with the channel dimension changed to C X N by a reshape operation, and the resulting output (C X N) is then transformed to C X H X W by reshape operation and weighted by the original signature a. The output is:
wherein Ej is the output channel attention profile; beta is a learnable or training parameter initialized to 0.
And finally, splicing the obtained feature graphs together in the channel dimension, and obtaining the feature with the same dimension as the input feature through channel shuffling. Wherein the building block and the parallel attention module share a channel shuffling module.
Step S221: after receiving the multiple feature images, the multi-scale feature fusion module adjusts the channel number and the length and width of the multiple feature images to be the same as the output size through a 1×1 convolution and up-sampling unit in a lateral connection mode, and adds and aggregates the obtained multiple output feature images to obtain an aggregated feature image.
Specifically, as shown in fig. 11, C3 is an output feature map from the back bone network Cross Stage2, C4 is an output feature map from the back bone network Cross Stage3, C5 is an output feature map from the back bone network Cross Stage4, and Cg is a global feature map obtained by global averaging of C5. By adopting a lateral connection mode, the channel number and the length and the width of the four feature images are adjusted to be the same as the output size through a 1×1 convolution and up-sampling unit, and the output feature images P3, P4, P5 and Pg are obtained and added and aggregated.
The specific fusion method has the following calculation formula:
F BGFM ＝sigmoid(P 3 )·(P 3 +P 4 +P 5 +P g )
wherein F is BGFM The method is characterized in that the method is an output characteristic diagram of a multi-scale characteristic fusion network module, P3 is an output characteristic diagram obtained by 1X 1 convolution of C3, P4, P5 and Pg are output characteristic diagrams obtained by 1X 1 convolution and up-sampling units respectively of C4, C5 and Cg.
Wherein P is 3 Leading P by obtaining re-weighting coefficient through Sigmoid function 3 、P 4 、P 5 And P g And (5) fusion.
Step S222: and inputting the aggregated feature map into the region candidate network by using the classification regression module, generating an anchor frame, and obtaining smoke boundary frame coordinates by using the size of the anchor frame as a priori frame and carrying out frame regression prediction.
Specifically, as shown in fig. 12, the lightweight detection head using ThunderNet matches the backbone network. Inputting the feature map into RPN (regional candidate network), and using 6 scales {12 } for complex smoke feature in dataset 2 ,16 2 ,32 2 ,64 2 ,128 2 ,256 2 And 3 ratios {1:2,1:1,2:1} to yield ahchor (anchor box). And obtaining a boundary frame by using the size of the anchor frame as a priori frame through frame regression prediction, wherein the model regression aims at biasing coordinates between the real boxes and the anchor boxes, and obtaining final predicted boxes coordinates. Classifying the boundary frames by using a logistic classifier to obtain the smoke category and classification probability corresponding to each boundary frame; and then sorting the smoke categories and the classification probabilities of all the bounding boxes through NMS (IOU thresh=0.5), determining the smoke category corresponding to each bounding box, obtaining a predicted value, wherein the predicted value comprises smoke category and smoke position information, and calculating training loss between the predicted value and a true value through a loss function.
Step S300: and analyzing the test sample set by using the trained lightweight neural smoke detection model to obtain a detection value, wherein the detection value comprises smoke category and smoke coordinate information.
The network model performance is measured by Parameters, GFLOPs (calculated amount), mAP (detection precision) and the like.
The comparative data for the different models is shown in table 2:
TABLE 2 MAP and fps values for different models
Table 2 compares the detection effect of 10 deep learning object detection algorithms on smoke datasets, including models of Faster-RCNN, retina Net, and the newly proposed YOLOv 5. As can be seen from table 2, the calculated amount (GFLOPs) and the parameter amount (Parameters) of the model designed in the present application on the data set are optimal in all the comparison methods. The detection accuracy mAP of the method on the smoke data set is 74.2%. In conclusion, the LSDet achieves higher detection precision while achieving light weight. The original image before detection is shown in fig. 13, and the effect after detection is shown in fig. 14.
Example 2
Based on the same inventive concept, the application also discloses a smoke detection system based on the lightweight neural network, which comprises:
the image acquisition module is used for acquiring image information;
the image preprocessing module is used for preprocessing the acquired image information to obtain a data set, a test sample set and a training sample set; and
the light neural smoke detection model is used for analyzing input image information to obtain detection values, and the detection values comprise smoke types and smoke coordinate information.
Further, the lightweight neural smoke detection model includes:
the backbone network feature extraction module is used for acquiring a plurality of feature graphs;
the multi-scale feature fusion network module is used for fusing the feature images to generate an aggregate feature image; and
and the classification regression module is used for obtaining the finally predicted smoke boundary frame coordinates according to the aggregation feature map.
Further, the image preprocessing module includes:
the image clipping unit clips the acquired initial image according to a threshold cutting method to obtain a smoke image only containing a smoke part;
the file generation unit is used for generating an xml file comprising a picture name and a smoke position coordinate for each marked smoke image, wherein one xml file is a label;
the sample set generating unit is used for converting all the labels into txt format files and json format files to serve as data sets, and extracting images from the data sets according to set proportions to serve as training sample sets and testing sample sets.
Further, the system also includes a model training module for iteratively training the lightweight neural smoke detection model.
Example 3
The application also discloses a computer readable storage medium, characterized in that the computer readable storage medium comprises a program which, when executed by a processor, is executed by a smoke detection method based on a lightweight neural network as claimed in any one of the preceding claims.
Example 4
The embodiment of the application also discloses a smog risk visual detection device, which comprises:
one or more memories for storing instructions; and
one or more processors to invoke and execute the instructions from the memory to perform the lightweight neural network-based smoke detection method as recited in any preceding claim.
In the embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. The above described embodiments of the apparatus are merely illustrative, e.g. the division of the units is merely a logical functional division, and there may be other ways of dividing in practice; for example, multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical, or other forms.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiment of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units. The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
At present, the technical scheme of the invention has been subjected to pilot-scale test, namely, smaller-scale test of products before large-scale mass production; after the pilot test is completed, the use investigation of the user is performed in a small range, and the investigation result shows that the user satisfaction is higher; now, the preparation of the formal production of the product for industrialization (including intellectual property risk early warning investigation) is started.
Claims (10)
1. A smoke detection method based on a lightweight neural network, comprising:
acquiring a smoke image and preprocessing the smoke image to obtain a data set, a test sample set and a training sample set;
building a lightweight neural smoke detection model, and training the lightweight neural smoke detection model by using the training sample set to obtain a trained lightweight neural smoke detection model;
and analyzing the test sample set by using the trained lightweight neural smoke detection model to obtain a detection value, wherein the detection value comprises smoke category and smoke coordinate information.
2. The smoke detection method based on a lightweight neural network according to claim 1, wherein the preprocessing of the smoke image comprises:
cutting the acquired initial image according to a threshold cutting method to obtain a smoke image only containing a smoke part;
labeling the smoke parts in the plurality of smoke images, wherein each labeled smoke image generates an xml file comprising a picture name and smoke position coordinates, and one xml file is a label;
all the labels are converted into txt format files and json format files to be used as data sets, and images are extracted from the data sets according to set proportions to be used as training sample sets and test sample sets.
3. The smoke detection method based on a lightweight neural network according to claim 1, wherein when the lightweight neural smoke detection model is trained by using the data set, the method comprises:
loading a pre-training weight file, and initializing a smoke detection model; the pre-training file is a history accumulated training parameter;
inputting the training sample set into an initialized smoke detection model for training, outputting predicted smoke boundary frame coordinates and real smoke boundary frame coordinates, and generating a loss function loss;
returning to the loss function loss, updating training parameters according to a gradient descent method, and performing iterative operation;
and stopping training after the iterative operation reaches the preset iterative times to obtain the lightweight neural smoke detection model.
4. The smoke detection method based on a lightweight neural network according to claim 1, wherein the lightweight neural smoke detection model comprises a backbone network feature extraction module, a multi-scale feature fusion module and a classification regression module;
the method for analyzing the test sample set by using the lightweight neural smoke detection model comprises the following steps of:
inputting images in a test sample set to a backbone network feature extraction module, sequentially passing through a Unit2 downsampling Unit and a plurality of Unit1 basic units, performing one-time splicing operation on the obtained output features and the output of the Unit2 downsampling Unit, and then obtaining a plurality of feature images through a channel shuffling module;
after receiving a plurality of feature images, the multi-scale feature fusion module adjusts the channel number and the length and width of the plurality of feature images to be the same as the output size through a 1X 1 convolution and up-sampling unit in a lateral connection mode, and adds and aggregates the obtained plurality of output feature images to obtain an aggregated feature image;
and inputting the aggregated feature map into the region candidate network by using the classification regression module, generating an anchor frame, and obtaining smoke boundary frame coordinates by using the size of the anchor frame as a priori frame and carrying out frame regression prediction.
5. A smoke detection system based on a lightweight neural network, comprising
The image acquisition module is used for acquiring image information;
the image preprocessing module is used for preprocessing the acquired image information to obtain a data set, a test sample set and a training sample set; and
the light neural smoke detection model is used for analyzing input image information to obtain detection values, and the detection values comprise smoke types and smoke coordinate information.
6. The lightweight neural network-based smoke detection system of claim 5, wherein the lightweight neural smoke detection model comprises:
the backbone network feature extraction module is used for acquiring a plurality of feature graphs;
the multi-scale feature fusion network module is used for fusing the feature images to generate an aggregate feature image; and
and the classification regression module is used for obtaining the finally predicted smoke boundary frame coordinates according to the aggregation feature map.
7. The smoke detection system of claim 5 wherein said image preprocessing module comprises:
the image clipping unit clips the acquired initial image according to a threshold cutting method to obtain a smoke image only containing a smoke part;
the file generation unit is used for generating an xml file comprising a picture name and a smoke position coordinate for each marked smoke image, wherein one xml file is a label;
the sample set generating unit is used for converting all the labels into txt format files and json format files to serve as data sets, and extracting images from the data sets according to set proportions to serve as training sample sets and testing sample sets.
8. The smoke detection system of claim 5 further comprising a model training module for iteratively training the lightweight neural smoke detection model.
9. A computer-readable storage medium, the computer-readable storage medium comprising:
a program which, when executed by a processor, is executed by a lightweight neural network based smoke detection method as claimed in any one of claims 1 to 4.
10. A smoke risk visual inspection device, characterized by: comprising
One or more memories for storing instructions; and
one or more processors to invoke and execute the instructions from the memory to perform the lightweight neural network-based smoke detection method of any of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310297290.XA CN116434002A (en) | 2023-03-24 | 2023-03-24 | Smoke detection method, system, medium and equipment based on lightweight neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310297290.XA CN116434002A (en) | 2023-03-24 | 2023-03-24 | Smoke detection method, system, medium and equipment based on lightweight neural network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116434002A true CN116434002A (en) | 2023-07-14 |
Family
ID=87082504
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310297290.XA Pending CN116434002A (en) | 2023-03-24 | 2023-03-24 | Smoke detection method, system, medium and equipment based on lightweight neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116434002A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116721302A (en) * | 2023-08-10 | 2023-09-08 | 成都信息工程大学 | Ice and snow crystal particle image classification method based on lightweight network |
| CN116612336B (en) * | 2023-07-19 | 2023-10-03 | 浙江华诺康科技有限公司 | Method, apparatus, computer device and storage medium for classifying smoke in endoscopic image |
| CN116977634A (en) * | 2023-07-17 | 2023-10-31 | 应急管理部沈阳消防研究所 | Fire smoke detection method based on laser radar point cloud background subtraction |
-
2023
- 2023-03-24 CN CN202310297290.XA patent/CN116434002A/en active Pending
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116977634A (en) * | 2023-07-17 | 2023-10-31 | 应急管理部沈阳消防研究所 | Fire smoke detection method based on laser radar point cloud background subtraction |
| CN116977634B (en) * | 2023-07-17 | 2024-01-23 | 应急管理部沈阳消防研究所 | Fire smoke detection method based on laser radar point cloud background subtraction |
| CN116612336B (en) * | 2023-07-19 | 2023-10-03 | 浙江华诺康科技有限公司 | Method, apparatus, computer device and storage medium for classifying smoke in endoscopic image |
| CN116721302A (en) * | 2023-08-10 | 2023-09-08 | 成都信息工程大学 | Ice and snow crystal particle image classification method based on lightweight network |
| CN116721302B (en) * | 2023-08-10 | 2024-01-12 | 成都信息工程大学 | Ice and snow crystal particle image classification method based on lightweight network |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN116434002A (en) | Smoke detection method, system, medium and equipment based on lightweight neural network | |
| CN112801169B (en) | Camouflage target detection method, system, device and storage medium based on improved YOLO algorithm | |
| CN108038846A (en) | Transmission line equipment image defect detection method and system based on multilayer convolutional neural networks | |
| CN110598029A (en) | Fine-grained image classification method based on attention transfer mechanism | |
| CN112668579A (en) | Weak supervision semantic segmentation method based on self-adaptive affinity and class distribution | |
| CN113780211A (en) | Lightweight aircraft detection method based on improved yolk 4-tiny | |
| CN114529799A (en) | Aircraft multi-target tracking method based on improved YOLOV5 algorithm | |
| CN109784283A (en) | Based on the Remote Sensing Target extracting method under scene Recognition task | |
| CN112560827B (en) | Model training method, model training device, model prediction method, electronic device, and medium | |
| CN110909672A (en) | Smoking action recognition method based on double-current convolutional neural network and SVM | |
| CN115482418B (en) | Semi-supervised model training method, system and application based on pseudo-negative labels | |
| CN111368634A (en) | Human head detection method, system and storage medium based on neural network | |
| CN114511710A (en) | Image target detection method based on convolutional neural network | |
| CN116824585A (en) | Aviation laser point cloud semantic segmentation method and device based on multistage context feature fusion network | |
| CN110728186B (en) | Fire detection method based on multi-network fusion | |
| CN111898418A (en) | Human body abnormal behavior detection method based on T-TINY-YOLO network | |
| CN115393666A (en) | Small sample expansion method and system based on prototype completion in image classification | |
| CN114881286A (en) | Short-time rainfall prediction method based on deep learning | |
| CN114511785A (en) | Remote sensing image cloud detection method and system based on bottleneck attention module | |
| CN113327253A (en) | Weak and small target detection method based on satellite-borne infrared remote sensing image | |
| CN117218545A (en) | LBP feature and improved Yolov 5-based radar image detection method | |
| CN115272412B (en) | Edge calculation-based low-small slow target detection method and tracking system | |
| CN117132910A (en) | Vehicle detection method and device for unmanned aerial vehicle and storage medium | |
| CN117152528A (en) | Insulator state recognition method, insulator state recognition device, insulator state recognition apparatus, insulator state recognition program, and insulator state recognition program | |
| CN116863271A (en) | Lightweight infrared flame detection method based on improved YOLO V5 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |