CN117787690A - Hoisting operation safety risk identification method and identification device - Google Patents
Hoisting operation safety risk identification method and identification device Download PDFInfo
- Publication number
- CN117787690A CN117787690A CN202311740839.4A CN202311740839A CN117787690A CN 117787690 A CN117787690 A CN 117787690A CN 202311740839 A CN202311740839 A CN 202311740839A CN 117787690 A CN117787690 A CN 117787690A
- Authority
- CN
- China
- Prior art keywords
- image
- target
- crane
- hoisting
- frame
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 60
- 238000001514 detection method Methods 0.000 claims description 88
- 239000011159 matrix material Substances 0.000 claims description 43
- 238000006073 displacement reaction Methods 0.000 claims description 15
- 230000004927 fusion Effects 0.000 claims description 13
- 230000001502 supplementing effect Effects 0.000 claims description 12
- 238000004422 calculation algorithm Methods 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 9
- 238000003062 neural network model Methods 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 claims description 8
- 239000013589 supplement Substances 0.000 claims description 7
- 238000010606 normalization Methods 0.000 claims description 5
- 238000003860 storage Methods 0.000 claims description 5
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 238000007621 cluster analysis Methods 0.000 claims description 4
- 238000012937 correction Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 230000008859 change Effects 0.000 description 4
- 238000000605 extraction Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000005520 cutting process Methods 0.000 description 2
- 230000033001 locomotion Effects 0.000 description 2
- 238000011176 pooling Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000003702 image correction Methods 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000009545 invasion Effects 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Landscapes
- Image Analysis (AREA)
Abstract
The invention provides a hoisting operation safety risk identification method and an identification device, wherein the identification method comprises the following steps: acquiring a hoisting scene image; acquiring an object identification image: inputting the hoisting scene image into an improved YOLO v5 model to obtain a target identification image; acquiring a corrected identification image: correcting the target identification image to obtain a corrected identification image; identifying a security risk: identifying safety risks in hoisting operation according to the safety relationship between the worker target and the crane target in the corrected identification image; wherein the improved YOLO v5 model is obtained by replacing the C3 module in the backbone network of the YOLO v5 model with a C2f module. The invention can effectively detect potential dangerous situations in complex hoisting scenes, improves the safety performance and has certain instantaneity and robustness.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a hoisting operation safety risk identification method and an identification device.
Background
The installation and the positioning of large-scale mechanical equipment in the industrial field are usually realized by means of hoisting equipment, the process involves complex interaction of the large-scale mechanical equipment and field workers, and a large safety risk exists, so that the research and the utilization of artificial intelligence technology to assist in detecting the construction safety and early warning the safety of the construction field become particularly important. However, in practical engineering application, the hoisting operation generally has the conditions of unfixed operation position, non-uniform crane appearance, complicated operation environment and the like, so that the mode of performing target detection after image cutting processing in the prior art cannot meet the identification requirement of safety risks in the hoisting operation.
Disclosure of Invention
The invention provides a hoisting operation safety risk identification method and a hoisting operation safety risk identification device, which are used for solving the defect that a mode of carrying out target detection after image cutting processing in the prior art cannot meet the safety risk identification requirement in hoisting operation, and realizing the identification of the safety risk in hoisting operation.
The invention provides a hoisting operation safety risk identification method, which comprises the following steps:
acquiring a hoisting scene image;
acquiring an object identification image: inputting the hoisting scene image into an improved YOLO v5 model to obtain a target identification image;
acquiring a corrected identification image: correcting the target identification image to obtain a corrected identification image;
identifying a security risk: identifying safety risks in hoisting operation according to the safety relationship between the worker target and the crane target in the corrected identification image;
wherein the improved YOLO v5 model is obtained by replacing the C3 module in the backbone network of the YOLO v5 model with a C2f module.
According to the hoisting operation safety risk identification method provided by the invention, the hoisting scene image is input into an improved YOLO v5 model to obtain a target identification image, and the method specifically comprises the following steps:
obtaining graphic features and semantic features: inputting the hoisting scene image into a backbone network of an improved YOLO v5 model to obtain graphic features;
acquiring fusion characteristics: inputting the graphic features into a neck structure of an improved YOLO v5 model to perform multi-scale feature fusion to obtain fusion features;
acquiring an object identification image: inputting the fusion characteristics into the head structure of the improved YOLO v5 model to obtain a target identification image;
the target identification image comprises a target category, a width and height of a target detection frame, a center point and a confidence level.
According to the hoisting operation safety risk identification method provided by the invention, the target identification image is corrected to obtain a corrected identification image, and the hoisting operation safety risk identification method specifically comprises the following steps:
acquiring a crane target compensation image: matching the target identification image with the crane template image to remove false crane targets in the target identification image and supplement missed crane targets into the target identification image to obtain a crane target compensation image;
acquiring a corrected identification image: and inputting the crane target compensation image and the continuous multi-frame hoisting scene image into a neural network model LightGlue for lightweight feature matching so as to supplement the missed detection target into the crane target compensation image and obtain a corrected identification image.
According to the hoisting operation safety risk identification method provided by the invention, the target identification image is matched with the crane template image to obtain the crane target compensation image, and the method specifically comprises the following steps:
computing co-occurrence matrix C of target recognition images respectively I (a, b) co-occurrence matrix C with crane template image T (a, b); wherein, co-occurrence matrix C I (a, b) and C T (a, b) are both employedCalculating to obtain;
wherein C (a, b) represents a co-occurrence matrix; z represents a normalization factor; p and q represent two pixel points; sigma represents a preset proportional parameter; []Indicating that the operation of judging true or false is true 1 and false is 0; i p Pixel value representing pixel point p, I q A pixel value representing a pixel point q, d representing a distance between the pixel points p and q;
co-occurrence matrix C of object recognition image I (a, b) conversion into a corresponding inter-point mutual information matrix M I (a, b) co-occurrence matrix C of crane template image T (a, b) conversion to corresponding inter-point mutual information momentArray M T (a, b); wherein, the inter-point information matrix M I (a, b) and M T (a, b) are both employedCalculating to obtain;
wherein M (a, b) represents an inter-point information matrix; h (a) represents the prior probability of the occurrence of the pixel value a, h (b) represents the prior probability of the occurrence of the pixel value b, and h represents the standard histogram;
based on inter-site mutual information matrix M I (a, b) and M T The elements in (a, b) obtaining a matching region of the target identification image and the crane template image;
R＝arg R max(S R )，
wherein arg represents a reverse operation, arg R max(S R ) Is represented by S R R, R represents the matching region at maximum, S R Representing the similarity under a certain matching region,
wherein m is I (a, b) represents an element in an inter-point mutual information matrix of a target identification image currently to be measured, m T (a, b) representing elements in the inter-point information matrix of the crane template image; t represents a crane template image; i represents the current target identification image to be tested.
And eliminating the false detected crane target in the target identification image according to the matching area, and supplementing the missed crane target into the target identification image to obtain a crane target compensation image.
According to the hoisting operation safety risk identification method provided by the invention, the crane target compensation image and the continuous multi-frame hoisting scene image are input into the neural network model LightGlue for lightweight feature matching so as to obtain the corrected identification image, and the method specifically comprises the following steps:
performing neural network model LightGlue lightweight characteristic matching on the crane target compensation image and the continuous multi-frame hoisting scene image to obtain coordinate information of matched paired characteristic points;
according to the coordinate information of the matched paired characteristic points, the characteristic points in the hoisting scene images of the previous frame and the current frame are in one-to-one correspondence;
and deducing the inter-frame displacement of the detection frame according to the inter-frame displacement of the feature points, and supplementing the missed detection target into a crane target compensation image to obtain a corrected identification image.
Further, the method comprises the steps of reasoning the inter-frame displacement of the detection frame according to the inter-frame displacement of the feature points, and supplementing the missed detection target into a crane target compensation image to obtain a corrected identification image, and specifically comprises the following steps:
calculating according to the center point of the detection frame in the previous frame of lifting scene image to obtain the center point of the detection frame in the current frame of lifting scene image;
obtaining the width of the detection frame in the current frame lifting scene image according to the width calculation of the detection frame in the previous frame lifting scene image, and obtaining the height of the detection frame in the current frame lifting scene image according to the height calculation of the detection frame in the previous frame lifting scene image;
obtaining a detection frame in the current frame hoisting scene image according to the center point of the detection frame in the current frame hoisting scene image and the width and the height of the detection frame in the current frame hoisting scene image;
and supplementing the detection frame in the current frame hoisting scene image into the crane target compensation image to obtain a corrected identification image.
According to the hoisting operation safety risk identification method provided by the invention, the safety risk in hoisting operation is identified according to the safety relationship between the worker target and the crane target in the corrected identification image, and the method specifically comprises the following steps:
acquiring an identification image corrected by a frame-by-frame image in a video;
judging the safety relationship between the worker target and the crane target in the corrected identification image of each frame: and when the position of the worker target is in the area refusing space of the hoisting operation, judging that the worker target has safety risk.
According to the hoisting operation safety risk identification method provided by the invention, the region refusing space of the hoisting operation is a cone space, the vertex of the cone space is a lifting hook, and the bottom surface of the cone space can cover an effective operation range.
According to the hoisting operation safety risk identification method provided by the invention, the safety relationship between the worker target and the crane target in the identification image corrected by each frame is judged, and the method specifically comprises the following steps:
for each detected crane target, an isosceles triangle area is defined by taking the midpoint of the upper edge of the crane detection frame and the lower edge as boundaries;
detecting Harris corner points in the isosceles triangle area to obtain corner points of the delimited area;
for the detected worker target, carrying out Harris corner detection in the worker detection frame to obtain a worker detection corner;
performing cluster analysis on the delimited area corner points and the worker detection corner points by adopting a DBSCAN clustering algorithm so as to divide the delimited area corner points and the worker detection corner points into different clusters;
calculating the proportion of the corner points of the delimited area and the corner points detected by workers to the same cluster, and comparing the proportion with a preset safety threshold value; when the proportion is larger than a safety threshold, judging that the safety relationship between the worker and the crane is a dangerous condition; when the safety relationship between the worker and the crane obtained by continuous several frames of images is determined as a dangerous situation, it is finally determined that a continuous dangerous situation exists.
The hoisting operation safety risk identification method provided by the invention further comprises the following steps:
when the safety risk of the worker is identified, a prompt word is marked at the top of the worker detection frame, the event that the worker enters the hoisting area in a violation manner is identified and recorded, and the real-time potential safety hazard warning of the industrial field is carried out.
The invention also provides a hoisting operation safety risk identification device, which comprises:
the hoisting scene image acquisition module is used for acquiring hoisting scene images;
the target identification image acquisition module is used for inputting the hoisting scene image into the improved YOLO v5 model to obtain a target identification image;
the corrected identification image acquisition module is used for correcting the target identification image to obtain a corrected identification image;
the safety risk identification module is used for identifying the safety risk in the hoisting operation according to the safety relationship between the worker target and the crane target in the corrected identification image;
wherein the improved YOLO v5 model is obtained by replacing the C3 module in the backbone network of the YOLO v5 model with a C2f module.
The invention also provides a non-transitory computer readable storage medium, on which is stored a computer program which, when executed by a processor, implements the lifting operation safety risk identification method of any one of the above.
According to the hoisting operation safety risk identification method provided by the invention, the obtained hoisting scene image is input into the improved YOLO v5 model to obtain the target identification image, and the target identification image is corrected to obtain the corrected identification image; then, according to the safety relationship between the worker target and the crane target in the corrected identification image, identifying the safety risk in hoisting operation, thereby being capable of determining whether potential safety hazards exist or not and giving an alarm as appropriate; the invention can effectively detect potential dangerous situations in complex hoisting scenes, improves the safety performance and has certain instantaneity and robustness.
Drawings
In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a hoisting operation safety risk identification method provided by the invention;
fig. 2 is a schematic diagram of a network structure of an improved YOLO v5 model in the hoisting operation security risk identification method provided by the invention;
FIG. 3 is a schematic structural diagram of C2f in a modified YOLO v5 model in the hoisting operation safety risk identification method provided by the invention;
fig. 4 is a schematic structural diagram of a hoisting operation safety risk identification device provided by the invention.
Reference numerals:
401: hoisting the scene image acquisition module; 402: a target recognition image acquisition module; 403: a corrected identification image acquisition module; 404: and a security risk identification module.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden on the person of ordinary skill in the art based on embodiments of the present invention, are within the scope of the present invention.
The prior art mainly focuses on solving the problem of safety risk identification of hoisting operation by optimizing a target identification algorithm and constructing a data set under a specific environment. However, because of the unique security detection requirements and the complex background environment in the lifting scene, the existing algorithm cannot be suitable for the identification of security risks in the lifting scene.
The hoisting operation safety risk identification method and the hoisting operation safety risk identification device provided by the invention are described below with reference to fig. 1-4.
As shown in fig. 1, the hoisting operation security risk identification method provided by the invention comprises the following steps:
s1, acquiring a hoisting scene image.
The hoisting scene image can be obtained by shooting the hoisting scene by a camera. The camera can be arranged on the inspection robot.
S2, acquiring a target identification image:
the lifting scene image is input into a modified YOLO v5 model as shown in fig. 2, and an object recognition image is obtained. Specifically, the targets in the target recognition image may be workers, cranes, and the like in the lifting scene.
Among these, the existing YOLO v5 is an end-to-end deep learning model that uses convolutional neural networks to learn the characteristics of objects in images and uses multi-scale prediction and network segmentation to detect and locate targets.
The existing YOLO v5 model includes a backbone network backbone, a neck structure neg, and a head structure head. The network structure of the YOLO v5 model is a full convolution network, i.e. the network structure consists of convolution layers, batch normalization layers and the like, and does not contain a full connection layer.
The backbone network backbone is mainly used for extracting characteristics of an input image. Backbone network backbone mainly comprises Conv module, C3 module and SPPF (Spatial Pyramid Poolng Fast, fast spatial pyramid pooling) module. The Conv module is used for downsampling, dimension-increasing and dimension-decreasing, normalization, nonlinearity and the like on the feature map. And the C3 module is used for increasing the depth and receptive field of the backbone network and improving the feature extraction capability. The SPPF module is used for extracting and encoding the characteristics of the image under different scales.
The neck structure neg is used for carrying out multi-scale feature fusion on the feature map. The head structure head is used to make the final regression prediction.
The invention improves a backbone network backbone in the existing YOLO v5 model, and introduces a C2f module shown in figure 3. And replacing the C3 module in the backbone network backbone by using the C2f module. The C2f module is used for feature extraction and dimension reduction, and the input feature map is converted into higher-level abstract features through multi-layer convolution and pooling. Compared with the localization and shallow feature extraction of the C3 module, the backbone network backup in the improved YOLO v5 model can capture low-level features such as edges and textures in an input image at the same time, can capture details and local information in the input image, and is more suitable for the recognition of complex industrial environments in a hoisting scene.
The backbone network backbone in the modified YOLO v5 model includes a layer 5 Conv module, a layer 4C 2f module, and a layer 1 SPPF module. The convolution kernel sizes k of the first, second, third and fifth Conv modules are all 6, and the step sizes s are all 2; the convolution kernel size k of the fourth layer Conv module is 7, and the step s is 2.
The improved YOLO v5 model can significantly improve the average accuracy (mAP) value by introducing a C2f module; the safety behavior detection capability of hoisting operators can be remarkably improved by increasing the convolution kernel size of the fourth Conv module.
S3, acquiring a corrected identification image:
and correcting the target identification image to obtain a corrected identification image.
The correction of the target identification image is mainly realized by removing false detection targets in the target identification image and supplementing missed detection targets into the target identification image.
The corrected identification image comprises the corrected target category, the corrected width and height of the target detection frame, the center point and the confidence coefficient.
S4, identifying safety risk:
and identifying the safety risk in the hoisting operation according to the safety relationship between the worker target and the crane target in the corrected identification image.
The safety risk between the worker and the crane can be judged by judging the safety distance between the worker and the crane in the hoisting scene.
In one possible embodiment, the hoisting scene image is input into a modified YOLO v5 model to obtain an object identification image, which specifically includes:
s21, obtaining graphic features and semantic features:
and inputting the hoisting scene image into a backbone network of the improved YOLO v5 model to obtain the graphic characteristics.
S22, acquiring fusion characteristics:
inputting the graphic features into the neck structure of the improved YOLO v5 model to perform multi-scale feature fusion, and obtaining fusion features.
S23, acquiring a target identification image, and inputting the fusion characteristic into a head structure of an improved YOLO v5 model to obtain the target identification image;
the target identification image comprises a target category, a width and height of a target detection frame, a center point and a confidence level.
In this embodiment, 640×640×3 lifting scene images may be input into the modified YOLO v5 model. The backbone network backup in the improved YOLO v5 model processes the input 640 x 3 lifting scene image to obtain 80 x 128 characteristic diagram, 40 x 256 characteristic diagram and 20 x 512 characteristic diagram of the backbone network backup output. The feature maps output by the three backbone network backbones are input into a neck structure neg in the modified YOLO v5 model, and 80 x 128 feature maps, 40 x 256 feature maps and 20 x 512 feature maps output by the neck structure neg are obtained. The feature images output by the three neck structures neg are input into a head structure head to obtain a final target identification image.
Specifically, the target identification image comprises a target category, a width and height of a target detection frame, a center point and a confidence level. The target category may be a worker target, a crane target, and the like.
The invention focuses on the characteristic of the hoisting scene, wherein, the general body type of the crane target is larger, and occupies a larger area in the image; worker targets are generally smaller, occupying less area in the image. The invention can effectively enlarge the receptive field of the crane by adjusting the convolution kernel size of the Conv module in the backbone network backbone, and simultaneously, the invention can effectively recognize the worker target with relatively smaller size in the image.
In one possible embodiment, since the environmental background of the hoisting operation is complex and the targets are in a moving state during the hoisting operation, the problem that the targets are frequently blocked under the moving view angle of the camera is caused, and therefore, false detected targets in the target identification image need to be removed and missed targets need to be compensated, so that the missed targets are supplemented into the target identification image.
Correcting the target identification image to obtain a corrected identification image, which specifically comprises the following steps:
s31, acquiring a crane target compensation image:
matching the target identification image with the crane template image to remove false crane targets in the target identification image and supplement missed crane targets into the target identification image to obtain a crane target compensation image.
Wherein, the crane template image can be obtained from a crane template library. The building process of the crane template library comprises the following steps: and carrying out all-round shooting and interception on the cranes of the common model from different distances, and establishing a crane template library by using the obtained images.
S32, acquiring a corrected identification image:
and inputting the crane target compensation image and the continuous multi-frame hoisting scene image into a neural network model LightGlue for lightweight feature matching so as to supplement the missed detection target into the crane target compensation image and obtain a corrected identification image.
In this embodiment, matching the target recognition image with the crane template image specifically includes:
s311, respectively calculating co-occurrence matrix C of the target identification images I (a, b) co-occurrence matrix C with crane template image T (a,b)；
Wherein, the co-occurrence matrix C of the target identification image I (a, b) co-occurrence matrix C of crane template image T (a, b) are calculated by adopting a formula (1);
in the formula (1), C (a, b) represents a co-occurrence matrix; z represents a normalization factor; p and q represent two pixel points; sigma represents a preset proportional parameter; []Indicating that the operation of judging true or false is true 1 and false is 0; i p Pixel value representing pixel point p, I q A pixel value representing a pixel point q, and d represents a distance between the pixel points p and q.
The co-occurrence matrix C of the target recognition image is calculated by using the formula (1) I (a, b), p and q each represent a pixel point in the target recognition image; calculating co-occurrence matrix of crane template image by adopting (1)When p and q each represent a pixel point in the crane template image.
S312, co-occurrence matrix C of target identification image I (a, b) into a corresponding PMI (Pointwise Mutual Information, inter-point information) matrix M I (a, b) co-occurrence matrix C of crane template image T (a, b) conversion to the corresponding PMI matrix M T (a,b)；
Wherein PMI matrix M I (a, b) and M T (a, b) are calculated by adopting a formula (2);
in the formula (2), M (a, b) represents a PMI matrix; h (a) represents the prior probability of the occurrence of the pixel value a, h (b) represents the prior probability of the occurrence of the pixel value b, and h represents the standard histogram.
The PMI matrix M is calculated by the formula (2) I When (a, b), h (a) represents the prior probability that the pixel value a appears in the target identification image, and h (b) represents the prior probability that the pixel value b appears in the target identification image. Calculating PMI matrix M using (2) T When (a, b), h (a) represents the prior probability of the occurrence of the pixel value a in the crane template image, and h (b) represents the prior probability of the occurrence of the pixel value b in the crane template image.
S313, according to the inter-point mutual information matrix M I (a, b) and M T The elements in (a, b) obtaining a matching region of the target identification image and the crane template image;
R＝arg R max(S R ) (3)
in the formula (3), arg represents a reverse operation, arg R max(S R ) Is represented by S R R, R represents the matching region at maximum, S R Representing the similarity under a certain matching region,
in the formula (4), m I (a, b) represents the object currently under testIdentifying elements in the inter-point information matrix of the image, m T (a, b) representing elements in the inter-point information matrix of the crane template image; t represents a crane template image; i represents the current target identification image to be tested.
S315, eliminating the false detected crane target in the target identification image according to the matching area, and supplementing the missed crane target into the target identification image to obtain a crane target compensation image.
The invention adopts CoTM (Co-occurrence based template matching) to perform priori knowledge template matching on the target identification image and the crane template image, realizes template matching requirement through Co-occurrence statistics, has certain robustness to misalignment and deformation, has the advantage of rapid calculation, and can promote the generalization capability of the algorithm.
In this embodiment, a crane target compensation image and a continuous multi-frame lifting scene image are input into a neural network model LightGlue to perform lightweight feature matching, so as to supplement a missed detection target into the crane target compensation image, and obtain a corrected identification image, which specifically includes:
s321, performing neural network model LightGlue lightweight characteristic matching on the crane target compensation image and the continuous multi-frame hoisting scene image to obtain coordinate information of the matched paired characteristic points.
S322, according to the coordinate information of the matched paired characteristic points, the characteristic points in the hoisting scene images of the previous frame and the current frame are in one-to-one correspondence.
Assuming that the average moving speed of the feature points is equal to the moving speed of the detection frame; the average zoom speed of the feature points is equal to the zoom speed of the detection frame.
S323, reasoning the inter-frame displacement of the detection frame according to the inter-frame displacement of the feature points, and supplementing the missed detection target into a crane target compensation image to obtain a corrected identification image, wherein the process comprises the following steps:
calculating according to the center point of the detection frame in the previous frame of lifting scene image to obtain the center point of the detection frame in the current frame of lifting scene image:
obtaining the width of the detection frame in the current frame lifting scene image according to the width calculation of the detection frame in the previous frame lifting scene image, and obtaining the height of the detection frame in the current frame lifting scene image according to the height calculation of the detection frame in the previous frame lifting scene image:
in the formulae (5) to (6), (a) c ,b c ) Representing the center point of a detection frame in a lifting scene image of a previous frame, (p) c ,q c ) Representing the center point of a detection frame in a lifting scene image of a current frame, (m) x ,m y ) Representing the average displacement from the previous frame hoisting scene image to the characteristic point of the current frame hoisting scene image; w (w) 1 Representing the width, w, of a detection frame in a lifting scene image of a previous frame 2 Representing the width of a detection frame in a lifting scene image of a current frame, and h 1 Indicating the height of a detection frame in a hoisting image of a previous frame, h 2 Representing the height of a detection frame in a hoisting image of a current frame; r is (r) x Representing the average density change from the previous frame of hoisting scene image to the transverse characteristic point of the current frame of hoisting scene image, and r y Representing the average density change of feature points in the longitudinal direction from a previous frame of hoisting scene image to a current frame of hoisting scene image; sigma (sigma) a Representing the abscissa standard deviation sigma of characteristic points in a hoisting scene image of a previous frame b Representing the ordinate standard deviation sigma of characteristic points in a previous frame of hoisting scene image p Representing the abscissa standard deviation sigma of characteristic points in the lifting scene image of the current frame q And the ordinate standard deviation of the characteristic points in the hoisting scene image of the current frame is represented.
And obtaining the detection frame in the current frame lifting scene image according to the center point of the detection frame in the current frame lifting scene image and the width and the height of the detection frame in the current frame lifting scene image.
And supplementing the detection frame in the current frame hoisting scene image into the crane target compensation image to obtain a corrected identification image.
In this embodiment, in order to save calculation resources, the mathematical expectation of the next inter-frame displacement may be calculated according to the displacement condition of the crane detection frame in the image of the last few frames (specifically, 10 frames to 20 frames), so as to avoid feature matching on the lifting scene image of each frame.
Wherein motion estimation of image displacementAnd motion estimation of image size +.>The method comprises the following steps of:
in the formula (7), Δt represents the number of elapsed interval frames;representing the displacement of the feature point and the detection frame, +.>The density change of the feature points and the width-height change of the detection frame are represented.
In one possible embodiment, the identifying the safety risk in the hoisting operation according to the safety relationship between the worker target and the crane target in the corrected identification image specifically includes:
s41, acquiring an identification image after the frame-by-frame image correction in the video.
S42, judging the safety relationship between the worker target and the crane target in the identification image after each frame correction;
specifically, if the position of the worker target is in the region refusing space of the hoisting operation, the safety risk of the worker is judged, and the safety risk in the hoisting operation is identified.
After step S4, the hoisting operation security risk identification method provided by the invention further includes the following steps:
when the safety risk of the worker is identified, a prompt word is marked at the top of the worker detection frame, the event that the worker enters the hoisting area in a violation manner is identified and recorded, and the real-time potential safety hazard warning of the industrial field is carried out.
According to the hoisting safety regulations, the region refusing space for hoisting operation is a cone which takes the lifting hook as a vertex and covers the effective operation range with the radius of the bottom surface. If the area refuses the invasion of personnel in the space, the dangerous situation is judged to exist.
Specifically, the safety relationship between the worker target and the crane target in the identification image after each frame correction is judged, and the process is as follows:
s421, regarding each detected crane object, an isosceles triangle area is defined by taking the middle point of the upper edge of the crane detection frame and the lower edge as boundaries.
The range of the isosceles triangle area can be further reduced, so that the calculated amount of detection of Harris corner points (the Harris corner point method is to judge whether the corner points are the corner points by calculating the content difference value in the pixel neighborhood area) is reduced, and the response speed is improved.
S422, harris corner points are detected in the isosceles triangle areas, and the corner points of the demarcation areas are obtained.
S423, carrying out Harris corner detection on the detected worker target in the worker detection frame to obtain a worker detection corner.
S424, performing cluster analysis on the corner points of the delimited area and the corner points detected by workers by adopting a DBSCAN clustering algorithm;
wherein, DBSCAN is an unsupervised clustering algorithm that can be used to divide corner points into different clusters. Through cluster analysis, all corner points can be divided into different clusters.
S425, calculating the proportion of corner points of the delimited area and corner points detected by workers to the same cluster, and comparing the proportion with a preset safety threshold;
if the ratio is greater than the safety threshold, a safety relationship between the worker and the crane is determined to be a dangerous condition.
If the safety relationship between the worker and the crane, which is obtained by successive frames of images, is determined as a dangerous situation, it is finally determined that there is a continuous dangerous situation.
The frame number threshold of the image can be set empirically and is optimized and adjusted.
The hoisting operation safety risk identification method provided by the invention can effectively detect potential dangerous situations in complex hoisting scenes, improves safety performance, has certain instantaneity and robustness, and can be widely applied to identification of safety risks in actual production environments.
Wherein the improved YOLO v5 model is obtained by replacing the C3 module in the backbone network of the YOLO v5 model with a C2f module. Wherein,
the hoisting operation safety risk recognition method provided by the invention is based on an improved YOLO v5 model, integrates a priori knowledge template matching algorithm and a mobile visual angle association recognition algorithm, and processes the position relationship between workers and a crane through characteristic point clustering, so that whether potential safety hazards exist or not can be determined, and an alarm is given as appropriate.
According to the hoisting operation safety risk identification method provided by the invention, the C2f module is introduced into the YOLO v5 model, so that the average precision (mAP) value can be obviously improved, and particularly under the condition that the fifth layer convolution kernel size in the backbone network back of the improved YOLO v5 model is increased, the safety behavior detection capability of hoisting operators can be obviously improved. Compared with the prior art, the detection effect of the invention is more effective.
The hoisting operation safety risk identification device provided by the invention is described below, and the hoisting operation safety risk identification device described below and the hoisting operation safety risk identification method described above can be correspondingly referred to each other.
As shown in fig. 4, the hoisting operation safety risk identification device provided by the invention comprises a hoisting scene image acquisition module 401, a target identification image acquisition module 402, a corrected identification image acquisition module 403 and a safety risk identification module 404.
The lifting scene image acquisition module 401 is used for acquiring lifting scene images;
the target recognition image acquisition module 402 is used for inputting the hoisting scene image into the improved YOLO v5 model to obtain a target recognition image;
the corrected recognition image obtaining module 403 is configured to correct the target recognition image to obtain a corrected recognition image;
the safety risk identification module 404 is configured to identify a safety risk in the hoisting operation according to a safety relationship between the worker target and the crane target in the corrected identification image;
wherein the improved YOLO v5 model is obtained by replacing the C3 module in the backbone network of the YOLO v5 model with a C2f module.
In another aspect, the present invention also provides a computer program product, where the computer program product includes a computer program, where the computer program can be stored on a non-transitory computer readable storage medium, and when the computer program is executed by a processor, the computer can execute the hoisting operation security risk identification method provided by the above methods, and the method includes:
acquiring a hoisting scene image;
acquiring an object identification image: inputting the hoisting scene image into an improved YOLO v5 model to obtain a target identification image;
acquiring a corrected identification image: correcting the target identification image to obtain a corrected identification image;
identifying a security risk: identifying safety risks in hoisting operation according to the safety relationship between the worker target and the crane target in the corrected identification image;
wherein the improved YOLO v5 model is obtained by replacing the C3 module in the backbone network of the YOLO v5 model with a C2f module.
In yet another aspect, the present invention further provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform the hoisting operation security risk identification method provided by the above methods, the method comprising:
acquiring a hoisting scene image;
acquiring an object identification image: inputting the hoisting scene image into an improved YOLO v5 model to obtain a target identification image;
acquiring a corrected identification image: correcting the target identification image to obtain a corrected identification image;
identifying a security risk: identifying safety risks in hoisting operation according to the safety relationship between the worker target and the crane target in the corrected identification image;
wherein the improved YOLO v5 model is obtained by replacing the C3 module in the backbone network of the YOLO v5 model with a C2f module.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and practice the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the method described in the respective embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art will appreciate that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (12)
1. The hoisting operation safety risk identification method is characterized by comprising the following steps of:
acquiring a hoisting scene image;
acquiring an object identification image: inputting the hoisting scene image into an improved YOLO v5 model to obtain a target identification image;
acquiring a corrected identification image: correcting the target identification image to obtain a corrected identification image;
identifying a security risk: identifying safety risks in hoisting operation according to the safety relationship between the worker target and the crane target in the corrected identification image;
wherein the improved YOLO v5 model is obtained by replacing the C3 module in the backbone network of the YOLO v5 model with a C2f module.
2. The hoisting operation safety risk identification method according to claim 1, wherein the step of inputting the hoisting scene image into an improved YOLO v5 model to obtain the target identification image specifically comprises the following steps:
obtaining graphic features and semantic features: inputting the hoisting scene image into a backbone network of an improved YOLO v5 model to obtain graphic features;
acquiring fusion characteristics: inputting the graphic features into a neck structure of an improved YOLO v5 model to perform multi-scale feature fusion to obtain fusion features;
acquiring an object identification image: inputting the fusion characteristics into the head structure of the improved YOLO v5 model to obtain a target identification image;
the target identification image comprises a target category, a width and height of a target detection frame, a center point and a confidence level.
3. The hoisting operation safety risk identification method according to claim 2, wherein the correcting the target identification image to obtain a corrected identification image specifically comprises:
acquiring a crane target compensation image: matching the target identification image with the crane template image to remove false crane targets in the target identification image and supplement missed crane targets into the target identification image to obtain a crane target compensation image;
acquiring a corrected identification image: and inputting the crane target compensation image and the continuous multi-frame hoisting scene image into a neural network model LightGlue for lightweight feature matching so as to supplement the missed detection target into the crane target compensation image and obtain a corrected identification image.
4. The hoisting operation safety risk identification method according to claim 3, wherein the matching of the target identification image and the crane template image to obtain a crane target compensation image specifically comprises:
computing co-occurrence matrix C of target recognition images respectively I (a, b) co-occurrence matrix C with crane template image T (a, b); wherein, co-occurrence matrix C I (a, b) and C T (a, b) are both employedCalculating to obtain;
wherein C (a, b) represents a co-occurrence matrix; z represents a normalization factor; p and q represent two pixel points; the E represents a preset proportion parameter; []Indicating that the operation of judging true or false is true 1 and false is 0; i p Pixel value representing pixel point p, I q A pixel value representing a pixel point q, d representing a distance between the pixel points p and q;
co-occurrence matrix C of object recognition image I (a, b) conversion into a corresponding inter-point mutual information matrix M I (a, b) co-occurrence matrix C of crane template image T (a, b) conversionFor the corresponding inter-point mutual information matrix M T (a, b); wherein, the inter-point information matrix M I (a, b) and M T (a, b) are both employedCalculating to obtain;
wherein M (a, b) represents an inter-point information matrix; h (a) represents the prior probability of the occurrence of the pixel value a, h (b) represents the prior probability of the occurrence of the pixel value b, and h represents the standard histogram;
based on inter-site mutual information matrix M I (a, b) and M T The elements in (a, b) obtaining a matching region of the target identification image and the crane template image;
R＝arg R max(S R )，
wherein arg represents a reverse operation, arg R max(S R ) Is represented by S R R, R represents the matching region at maximum, S R Representing the similarity under a certain matching region,
wherein m is I (a, b) represents an element in an inter-point mutual information matrix of a target identification image currently to be measured, m T (a, b) representing elements in the inter-point information matrix of the crane template image; t represents a crane template image; i represents the current target identification image to be tested.
And eliminating the false detected crane target in the target identification image according to the matching area, and supplementing the missed crane target into the target identification image to obtain a crane target compensation image.
5. The hoisting operation safety risk identification method according to claim 4, wherein the step of inputting the crane target compensation image and the continuous multi-frame hoisting scene image into the neural network model LightGlue for lightweight feature matching to obtain the corrected identification image specifically comprises the following steps:
performing neural network model LightGlue lightweight characteristic matching on the crane target compensation image and the continuous multi-frame hoisting scene image to obtain coordinate information of matched paired characteristic points;
according to the coordinate information of the matched paired characteristic points, the characteristic points in the hoisting scene images of the previous frame and the current frame are in one-to-one correspondence;
and deducing the inter-frame displacement of the detection frame according to the inter-frame displacement of the feature points, and supplementing the missed detection target into a crane target compensation image to obtain a corrected identification image.
6. The hoisting operation safety risk identification method according to claim 5, wherein the step of inferring the inter-frame displacement of the detection frame according to the inter-frame displacement of the feature points and supplementing the missed detection target into a crane target compensation image to obtain a corrected identification image comprises the following steps:
calculating according to the center point of the detection frame in the previous frame of lifting scene image to obtain the center point of the detection frame in the current frame of lifting scene image;
obtaining the width of the detection frame in the current frame lifting scene image according to the width calculation of the detection frame in the previous frame lifting scene image, and obtaining the height of the detection frame in the current frame lifting scene image according to the height calculation of the detection frame in the previous frame lifting scene image;
obtaining a detection frame in the current frame hoisting scene image according to the center point of the detection frame in the current frame hoisting scene image and the width and the height of the detection frame in the current frame hoisting scene image;
and supplementing the detection frame in the current frame hoisting scene image into the crane target compensation image to obtain a corrected identification image.
7. The hoisting operation safety risk identification method according to claim 1, wherein the identifying the safety risk in the hoisting operation according to the safety relationship between the worker target and the crane target in the corrected identification image specifically comprises:
acquiring an identification image corrected by a frame-by-frame image in a video;
judging the safety relationship between the worker target and the crane target in the corrected identification image of each frame: and when the position of the worker target is in the area refusing space of the hoisting operation, judging that the worker target has safety risk.
8. The hoisting operation safety risk identification method according to claim 7, wherein the region refusing space of the hoisting operation is a cone space, the vertex of the cone space is a lifting hook, and the bottom surface of the cone space can cover an effective operation range.
9. The hoisting operation safety risk identification method according to claim 7, wherein the determining the safety relationship between the worker target and the crane target in the identification image after each frame correction specifically comprises:
for each detected crane target, an isosceles triangle area is defined by taking the midpoint of the upper edge of the crane detection frame and the lower edge as boundaries;
detecting Harris corner points in the isosceles triangle area to obtain corner points of the delimited area;
for the detected worker target, carrying out Harris corner detection in the worker detection frame to obtain a worker detection corner;
performing cluster analysis on the delimited area corner points and the worker detection corner points by adopting a DBSCAN clustering algorithm so as to divide the delimited area corner points and the worker detection corner points into different clusters;
calculating the proportion of the corner points of the delimited area and the corner points detected by workers to the same cluster, and comparing the proportion with a preset safety threshold value; when the proportion is larger than a safety threshold, judging that the safety relationship between the worker and the crane is a dangerous condition; when the safety relationship between the worker and the crane obtained by continuous several frames of images is determined as a dangerous situation, it is finally determined that a continuous dangerous situation exists.
10. The hoisting operation safety risk identification method according to claim 1, further comprising:
when the safety risk of the worker is identified, a prompt word is marked at the top of the worker detection frame, the event that the worker enters the hoisting area in a violation manner is identified and recorded, and the real-time potential safety hazard warning of the industrial field is carried out.
11. Hoisting operation safety risk recognition device, characterized by includes:
the hoisting scene image acquisition module is used for acquiring hoisting scene images;
the target identification image acquisition module is used for inputting the hoisting scene image into the improved YOLO v5 model to obtain a target identification image;
the corrected identification image acquisition module is used for correcting the target identification image to obtain a corrected identification image;
the safety risk identification module is used for identifying the safety risk in the hoisting operation according to the safety relationship between the worker target and the crane target in the corrected identification image;
wherein the improved YOLO v5 model is obtained by replacing the C3 module in the backbone network of the YOLO v5 model with a C2f module.
12. A non-transitory computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor implements a lifting operation safety risk recognition method according to any of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311740839.4A CN117787690A (en) | 2023-12-18 | 2023-12-18 | Hoisting operation safety risk identification method and identification device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311740839.4A CN117787690A (en) | 2023-12-18 | 2023-12-18 | Hoisting operation safety risk identification method and identification device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117787690A true CN117787690A (en) | 2024-03-29 |
Family
ID=90380819
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311740839.4A Pending CN117787690A (en) | 2023-12-18 | 2023-12-18 | Hoisting operation safety risk identification method and identification device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117787690A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117994594A (en) * | 2024-04-03 | 2024-05-07 | 武汉纺织大学 | Power operation risk identification method based on deep learning |
-
2023
- 2023-12-18 CN CN202311740839.4A patent/CN117787690A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117994594A (en) * | 2024-04-03 | 2024-05-07 | 武汉纺织大学 | Power operation risk identification method based on deep learning |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102129893B1 (en) | Ship tracking method and system based on deep learning network and average movement | |
| CN108509859B (en) | Non-overlapping area pedestrian tracking method based on deep neural network | |
| CN107123131B (en) | Moving target detection method based on deep learning | |
| CN107133973B (en) | Ship detection method in bridge collision avoidance system | |
| CN113052876B (en) | Video relay tracking method and system based on deep learning | |
| CN111079518B (en) | Ground-falling abnormal behavior identification method based on law enforcement and case handling area scene | |
| CN109448326B (en) | Geological disaster intelligent group defense monitoring system based on rapid image recognition | |
| CN104966304A (en) | Kalman filtering and nonparametric background model-based multi-target detection tracking method | |
| CN111161312B (en) | Object trajectory tracking and identifying device and system based on computer vision | |
| Bedruz et al. | Real-time vehicle detection and tracking using a mean-shift based blob analysis and tracking approach | |
| CN117787690A (en) | Hoisting operation safety risk identification method and identification device | |
| CN111414807A (en) | Tidal water identification and crisis early warning method based on YO L O technology | |
| CN116993970A (en) | Oil and gas pipeline excavator occupation pressure detection method and system based on yolov5 | |
| CN117557565B (en) | Detection method and device for lithium battery pole piece | |
| CN109558877B (en) | KCF-based offshore target tracking algorithm | |
| CN115345821A (en) | Steel coil binding belt loosening abnormity detection and quantification method based on active visual imaging | |
| Duan et al. | Real time road edges detection and road signs recognition | |
| CN117475353A (en) | Video-based abnormal smoke identification method and system | |
| CN113052139A (en) | Deep learning double-flow network-based climbing behavior detection method and system | |
| CN117011280A (en) | 3D printed concrete wall quality monitoring method and system based on point cloud segmentation | |
| CN111986233A (en) | Large-scene minimum target remote sensing video tracking method based on feature self-learning | |
| CN116259008A (en) | Water level real-time monitoring method based on computer vision | |
| CN113658223B (en) | Multi-row person detection and tracking method and system based on deep learning | |
| WO2018110377A1 (en) | Video monitoring device | |
| Shan et al. | Maritime target detection based on electronic image stabilization technology of shipborne camera |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |