CN114219687B - Intelligent construction safety hidden danger identification method integrating man-machine vision - Google Patents
Intelligent construction safety hidden danger identification method integrating man-machine vision Download PDFInfo
- Publication number
- CN114219687B CN114219687B CN202111290335.8A CN202111290335A CN114219687B CN 114219687 B CN114219687 B CN 114219687B CN 202111290335 A CN202111290335 A CN 202111290335A CN 114219687 B CN114219687 B CN 114219687B
- Authority
- CN
- China
- Prior art keywords
- hidden danger
- image
- target
- point
- gazing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000010276 construction Methods 0.000 title claims abstract description 62
- 238000000034 method Methods 0.000 title claims abstract description 39
- 230000000007 visual effect Effects 0.000 claims abstract description 31
- 238000012549 training Methods 0.000 claims abstract description 28
- 238000001514 detection method Methods 0.000 claims abstract description 26
- 230000008569 process Effects 0.000 claims abstract description 17
- 239000011159 matrix material Substances 0.000 claims abstract description 16
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 13
- 230000004424 eye movement Effects 0.000 claims description 20
- 230000009466 transformation Effects 0.000 claims description 12
- 238000006243 chemical reaction Methods 0.000 claims description 7
- 238000011176 pooling Methods 0.000 claims description 6
- 239000000463 material Substances 0.000 claims description 5
- 239000013598 vector Substances 0.000 claims description 5
- 238000010586 diagram Methods 0.000 claims description 4
- 238000013507 mapping Methods 0.000 claims description 4
- 238000012544 monitoring process Methods 0.000 claims description 4
- 238000012706 support-vector machine Methods 0.000 claims description 4
- 230000002776 aggregation Effects 0.000 claims description 2
- 238000004220 aggregation Methods 0.000 claims description 2
- 238000003384 imaging method Methods 0.000 claims description 2
- 230000001788 irregular Effects 0.000 claims description 2
- 230000002093 peripheral effect Effects 0.000 claims description 2
- 238000012545 processing Methods 0.000 claims description 2
- 238000013528 artificial neural network Methods 0.000 claims 1
- 238000005259 measurement Methods 0.000 claims 1
- 231100001267 hazard identification Toxicity 0.000 abstract description 3
- 238000007689 inspection Methods 0.000 abstract description 2
- 210000001508 eye Anatomy 0.000 description 29
- 238000002474 experimental method Methods 0.000 description 12
- 230000006870 function Effects 0.000 description 7
- 238000007726 management method Methods 0.000 description 6
- 239000011664 nicotinic acid Substances 0.000 description 5
- 230000008447 perception Effects 0.000 description 5
- 238000009435 building construction Methods 0.000 description 4
- 210000001747 pupil Anatomy 0.000 description 4
- 238000012360 testing method Methods 0.000 description 4
- 239000003086 colorant Substances 0.000 description 3
- 210000004087 cornea Anatomy 0.000 description 3
- 210000001525 retina Anatomy 0.000 description 3
- 230000019771 cognition Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000002265 prevention Effects 0.000 description 2
- 230000006399 behavior Effects 0.000 description 1
- 210000005252 bulbus oculi Anatomy 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 210000003128 head Anatomy 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 230000001678 irradiating effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 230000000704 physical effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000010408 sweeping Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
- G06Q50/26—Government or public services
- G06Q50/265—Personal security, identity or safety
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/047—Probabilistic or stochastic networks
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Business, Economics & Management (AREA)
- Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Tourism & Hospitality (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computer Security & Cryptography (AREA)
- Development Economics (AREA)
- Educational Administration (AREA)
- Probability & Statistics with Applications (AREA)
- Economics (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- Image Analysis (AREA)
Abstract
The intelligent recognition method of construction potential safety hazards integrating human-computer vision comprises the steps of firstly adopting an eye tracker to track an eye jump process, and obtaining a target saliency map based on human eye experience; then identifying a hidden danger image training database to obtain a target saliency map based on primary visual characteristics; training convolutional neural network parameters of hidden danger parts; and finally, establishing a hidden danger knowledge semantic discrimination model, calculating a similarity matrix of hidden danger information input and a hidden danger knowledge base, and realizing hidden danger automatic discrimination. According to the intelligent recognition method for the construction potential safety hazard integrating the man-machine vision, the traditional manual inspection construction potential safety hazard is changed into the automatic detection of the construction potential safety hazard by a machine, and the labor cost is saved. The mode of detecting the potential safety hazards by the expert is applied to the machine, so that the machine has the capability of detecting the potential safety hazards by the expert to automatically detect the potential safety hazards, and the potential safety hazard identification degree is high.
Description
Technical Field
The invention relates to the technical field of construction safety, in particular to an intelligent identification method for construction safety hidden danger integrating human-machine vision.
Background
The potential safety hazards of construction are mainly caused by unsafe states of people, objects and management, and the potential safety hazards have the characteristics of concealment, danger, contingency, causality and the like. The potential safety hazards of construction are not easy to find, gradually appear along with the progress of construction, are excited under certain conditions, and gradually develop into safety accidents. In construction, hidden dangers have the characteristic of contingency, and contingencies can be developed into necessary events without being emphasized. For potential safety hazards, early discovery, early treatment and early prevention should be achieved.
At present, construction potential safety hazards need to be identified, detected and modified on site by technicians, and real-time, comprehensive and efficient construction is difficult to achieve depending on experience and professional level of the technicians. In the common potential safety hazard and prevention measures in building construction, the safety precaution measure is as follows: the method comprises the steps of (1) improving safety management consciousness (2) establishing a sound safety production responsibility system (3) increasing safety education work (4) checking and accepting (5) construction equipment according to standards and specifications, and (6) increasing safety supervision force according to standard use. Articles of research on investigation methods of hidden danger in building construction, common potential safety hazards and countermeasure in high-rise building construction and discussion on countermeasure of common potential safety hazards in building construction are all introduction of potential safety hazard countermeasure from management perspective. In the existing patent document CN111145046a, a digital management method and system for the construction process of hydraulic and hydroelectric engineering are introduced, and the digital management platform is used for realizing the on-site monitoring and improving the operation level and engineering quality of the construction process. The drawbacks of the above method are: the hidden danger cannot be automatically identified, the hidden danger still needs to be identified manually, the accuracy of the hidden danger identification manually depends on experience of people, and the dependency on the people is too high.
The automatic identification of the construction potential safety hazards improves the safety management efficiency on one hand, monitors operators on the other hand, reduces the safety violation behaviors of the construction site, has the characteristics of high potential hazard identification degree, all weather and low cost, and has great significance for reducing the safety accidents.
Disclosure of Invention
The invention aims to solve the defects of the prior art and provides an intelligent recognition method for construction potential safety hazards by fusing man-machine vision.
The technical scheme adopted by the invention is as follows:
The intelligent recognition method of construction potential safety hazards integrating human-computer vision comprises the steps of firstly adopting an eye tracker to track an eye jump process, and obtaining a target saliency map based on human eye experience; then identifying a hidden danger image training database to obtain a target saliency map based on primary visual characteristics; training convolutional neural network parameters of hidden danger parts; and finally, establishing a hidden danger knowledge semantic discrimination model to realize hidden danger automatic discrimination.
The intelligent identification method for the construction potential safety hazards by fusing man-machine vision comprises the following steps:
step one: firstly, according to an eye movement instrument experiment, obtaining eye movement characteristic parameters of a tested group;
Secondly, carrying out coordinate conversion on the fixation clustering center points, solving a hidden danger detection optimal path, carrying out image stitching on hidden danger scenes on the hidden danger detection optimal path to obtain a large-view image, and further accurately positioning hidden danger positions in the large-view image according to group fixation point aggregation characteristics to obtain a target saliency map S 1 based on primary visual characteristics;
Thirdly, analyzing the eye movement data, distinguishing gazing significant areas from non-gazing significant areas, inputting training potential safety hazard images, and obtaining a target significant map S 2 based on eye gazing experience;
Step four, further identifying an image target on the basis of accurately positioning the hidden danger exposure part, and training convolutional neural network parameters of the hidden danger part by combining the established hidden danger image database;
step five, finally establishing a hidden danger knowledge semantic discrimination model, wherein the process is as follows: and calculating a similarity matrix Sim of the hidden danger information input and the hidden danger knowledge base, and combining to form hidden danger judging information so as to realize automatic hidden danger judgment.
The intelligent identification method for the construction potential safety hazards by fusing man-machine vision comprises the following steps:
step 1: and taking a large number of hidden danger pictures and videos acquired on site as pre-experiment materials, establishing a group visual cognition experiment sample library, and providing a training database for a construction potential safety hazard machine visual recognition model.
Step 2: the construction safety manager with abundant experience is recruited as a tested group, sample images are displayed one by one before the eyes of the tested group, each sample image stimulates for a certain time, and the tested group dictates hidden danger positions and features. And recording hidden danger identification accuracy of each tested group, and selecting the tested group with high hidden danger identification accuracy and good identification stability. Taking the screened test population of the preliminary experiment as the test population of the formal experiment.
Step 3: according to the construction process, inviting the head-wearing eye tracker equipment of the experimental tested group to enter an experimental scene, and searching potential safety hazards by all the experimental tested groups according to respective experience. And acquiring eye movement characteristic parameters such as a first visual angle video, a fixation point coordinate, a fixation point duration and the like when the tested group identifies hidden danger through an eye movement instrument.
Step 4: converting the gaze point coordinates acquired by the eye tracker into target point coordinates under a world coordinate system, converting coordinates of gaze cluster center points, taking the gaze cluster center points as key positioning points of an automatic glance path of a camera monitoring construction site, taking the sum of shortest glance times of all construction sites as a target function, and solving an optimal path for hidden danger detection. And performing image stitching on a plurality of clear local hidden danger scenes of the hidden danger detection path.
Step 5: according to the group fixation point gathering characteristics, further accurately positioning hidden danger positions in the large-view image, extracting parameters such as red, green and blue color channels of an input image, pixel point coordinates of the image and the like according to a hidden danger image training database, analyzing basic characteristics such as scene image brightness I, color RGB, direction LB and the like of different scales l, acquiring saliency maps I (c theta S), RGB (c theta S) and LB (c theta S) of brightness, color and direction characteristics, normalizing the three saliency maps, combining characteristic weights w I、wRGB、wLB, and establishing a target saliency map S 1 based on primary visual characteristics:
S1＝wI·I(cΘs)+wRGB·RGB(cΘs)+wLB·LB(cΘs) (4)
Step 6: gaze time weighting Gao Situ GS t, gaze sequence weighting map GS r, and gaze center distance weighting map GS d are established with each gaze point r time, sequence, and euclidean distance d (r, v) to cluster center point v as weights, respectively. The GS image is input into a linear support vector machine for training, a gazing salient region and a non-gazing salient region are distinguished, an attention model based on a human eye gazing experience target salient map is constructed, optimal w and b parameters of a linear hyperplane model are solved, and a target salient map based on the human eye gazing experience is extracted S 2:
Step 7: the construction method of the two types of target saliency maps is integrated, and a visual bionic perception model of the construction potential safety hazard is built. And optimizing model weight parameters according to the hidden danger image training database, and training a machine vision algorithm to learn hidden danger recognition experience of human eyes.
Step 8: classifying a large number of experiment materials such as hidden danger pictures, videos and eye movement data acquired on site according to hidden danger classification to form a convolutional neural network framework, deeply acquiring image characteristic information, finding out candidate areas possibly containing targets, acquiring image characteristics contained in the candidate areas, inputting the image characteristics into a full-connection layer, accessing a softmax classification function to realize classification and identification of target images, and training convolutional neural network parameters of hidden danger parts by combining an established hidden danger image database.
Step 9: adopting a bounding box algorithm to establish a two-dimensional plane enveloping space of a construction target, converting the space position and the relation data thereof into semantic concept expression, and establishing a hidden danger knowledge semantic discrimination model, wherein the establishment process is as follows: and (3) comparing the hidden danger standard knowledge base, extracting basic semantic concepts V 1、V2 of hidden danger information input and the hidden danger standard knowledge base, calculating the similarity S between semantic elements V 1、V2, outputting a maximum matching degree semantic concept set max { Sim }, and combining to form hidden danger judging information to realize hidden danger automatic judging.
The invention discloses an intelligent recognition method for construction potential safety hazards by fusing man-machine vision, which has the following technical effects:
1) The invention converts the traditional manual inspection construction potential safety hazard into the automatic detection construction potential safety hazard of the machine, thereby saving the labor cost.
2) According to the invention, the mode of detecting the potential safety hazard by the expert is applied to the machine, so that the machine has the capability similar to the capability of detecting the potential safety hazard by the expert to automatically detect the potential safety hazard, and the potential safety hazard identification degree is high.
Drawings
Fig. 1 is a diagram of a hidden danger visual bionic perception model.
FIG. 2 is a flow chart of image object recognition for hidden danger sites.
Fig. 3 is a construction hidden danger semantic discrimination flowchart.
Detailed Description
The intelligent identification method for the construction potential safety hazards by fusing man-machine vision comprises the following steps:
1. and taking a large number of hidden danger pictures and videos acquired on site as pre-experiment materials, establishing a group visual cognition experiment sample library, and providing a training database for a construction potential safety hazard machine visual recognition model.
2. The construction safety manager with abundant experience is recruited as a tested group, sample images are displayed one by one before the eyes of the tested group, each sample image stimulates for a certain time, and the tested group dictates hidden danger positions and features. And recording hidden danger identification accuracy of each tested group, and selecting the tested group with high hidden danger identification accuracy and good identification stability. The test population screened by the preliminary experiment is taken as the test population for the formal experiment.
3. According to the construction progress, inviting the expert group to enter an experimental scene by using the head-mounted eye tracker device, and making all experiments to be tested to patrol in the construction site according to the respective experience searching route. And acquiring eye movement characteristic parameters such as a first visual angle video, a fixation point coordinate, a fixation point duration and the like when the hidden danger is detected and identified through an eye movement instrument.
4. According to the principle of eye imaging and eye tracker data acquisition, pupil-cornea reflection technology is used, and the basic principle of the technology is that (1) infrared rays are used for irradiating eyes; (2) Collecting infrared light reflected from the cornea and retina using a camera; (3) Due to the physiological structure and physical properties of the eyeball, on the premise that the relative positions of the light source and the head are unchanged, a light spot formed by cornea reflection does not move and (4) the direction of light reflected on the retina indicates the direction of the pupil (light source light enters from the pupil and retina reflected light exits from the pupil); and finally (5) calculating the direction of eye movement according to the angle between the cornea and the pupil reflected light. And combining the horizontal visual angle alpha and the vertical visual angle beta of the eyes of the tested group to the target point, and calculating x, y and z coordinate axis visual angle scaling factor px α(d)、pyβ(d)、pzα (d) according to the measured distance d between the hidden danger part space point and the tested group. And scaling the fixation point coordinate (v x,vy,vz) based on the eye tracker coordinate system, and calculating the fixation point space coordinate (vz x,vzy,vzz) under the same coordinate origin to realize the space information mapping of the fixation point of the eye tracker.
vzx＝pxα×vx,vzy＝pyβ×vy,vzz＝pzα×vz (1);
Wherein: px α、pyβ、pzα represents the x, y, z direction ratio of the coordinates of the gaze point in the actual spatial position to the coordinates in the eye tracker, respectively, with the eye tracker as the origin of coordinates. vz x、vzy、vzz represents the actual spatial position coordinates of the gaze point with the eye tracker as the origin of coordinates, respectively. Taking the difference between the eye movement instrument coordinate system and the world space coordinate system into consideration, taking the eye point space coordinate as a basis, taking the eye point physical coordinates (X, Y, Z) based on the origin of the camera as a reference, establishing a space coordinate system conversion equation, solving a coordinate system conversion matrix M, and converting the eye point coordinates acquired by the eye movement instrument into target point coordinates under the world space coordinate system.
[X,Y,Z]＝M[vzx,vzy,vzz] (2)；
And carrying out coordinate transformation on the fixation clustering center point, wherein the fixation clustering center point refers to a center point of the fixation point gathered in a certain range, and is used as a key positioning point of an automatic glance path of a camera monitoring construction site, and solving an optimal path for hidden danger detection by taking the sum of the shortest time of glances of all the construction sites as a target function. And performing image stitching on a plurality of clear local hidden danger scenes of the hidden danger detection path.
The specific process of solving the optimal path for hidden danger detection is as follows:
Calculating the stay time TS K of each scene according to the sum of the accumulated time Sigma ms of the gazing points in each gazing clustering area, taking the shortest total time min Sigma TS K required for sweeping all construction areas as an objective function, taking the priority order zeta K of the detection parts into consideration, establishing a construction potential safety hazard detection traversal model based on the traveling business problem, and solving the optimal potential hazard detection optimal traversal path by using a particle swarm algorithm. And according to the hidden danger detection path, combining the key gazing area on the path, collecting a large amount of hidden danger partial images, and providing an image data basis for hidden danger identification.
Traveling salesman problem of construction potential safety hazard detection: and (3) taking the best hidden danger detection effect (time and recognition rate of detecting all construction areas) as an objective function, comprehensively considering constraint conditions such as the priority order of detection parts, the stay time of key gazing areas, the number of acquired images (the number of personnel and mechanical equipment, the operation type, the hidden danger occurrence frequency and severity degree) and the like.
The image stitching process of the hidden danger scene of the hidden danger detection path is as follows:
firstly, setting shooting overlapping rate of two adjacent scene images on a hidden danger detection path, and ensuring that the shot adjacent images have overlapping areas.
Secondly, carrying out multi-scale space division on the hidden danger image, extracting a pixel change amplitude obvious region between the same scale plane and different scales, and obtaining a stable characteristic point set P= { P 1,p2,...,pn };
Then, calculating the gradient direction theta (x, y) and the gradient amplitude m (x, y) of the pixel (x, y) of each feature point, describing the feature point according to the feature vector p (theta (x, y), m (x, y)) of the feature point, and forming a multidimensional feature description subset;
And finally, traversing and calculating Euclidean distance l between the feature points in the adjacent images according to the feature vectors of the feature points, and taking the minimum distance as a matching feature point:
Based on an image feature point matching strategy algorithm, adjacent images with overlapping areas are spliced into large-view hidden danger images, and image acquisition work of a construction scene is completed on a path most likely to have the construction hidden danger, and the construction hidden danger is included.
5. The specific process of obtaining the primary visual feature target saliency map S 1 is as follows:
And (5.1) further accurately positioning hidden danger positions in the large-view image according to the group fixation point gathering characteristics. According to a hidden danger image training database, extracting parameters such as red r l, green g l, blue b l three-color channels of an input image, pixel point coordinates (x, y) of the image and the like, extracting basic characteristics such as scene image brightness I, color RGB, direction LB and the like of different scales l, and training a target significant attention model based on primary visual characteristics:
In the formula (3): i l denotes scene luminance at l scale, r l denotes red channel of input image at l scale, g l denotes green channel of input image at l scale, b l denotes blue channel of input image at l scale, RGB l denotes three color feature at l scale, f RGBl(rl,gl,bl) denotes color feature function of three colors, LB (x, y) denotes directional feature of pixel point coordinates (x, y), and f LB (x, y) denotes function of directional feature of pixel point coordinates (x, y).
(5.2) Obtaining saliency maps I (c theta S), RGB (c theta S) and LB (c theta S) of brightness, color and direction characteristics aiming at scale factors of high resolution c and low resolution S in an image scale space by calculating gradient difference theta between a central pixel and a peripheral pixel of an image area, normalizing the three saliency maps, combining characteristic weights w I、wRGB、wLB, and establishing a target saliency map S 1 based on primary visual characteristics:
S1＝wI·I(cΘs)+wRGB·RGB(cΘs)+wLB·LB(cΘs) (4)
In the formula (4): s 1 denotes a target saliency map based on primary visual features, w I、wRGB、wLB denotes feature weights of saliency maps of luminance, color, and direction features, and I (cΘs), RGB (cΘs), and LB (cΘs) denote saliency maps of luminance, color, and direction features.
6. The specific process of obtaining the target saliency map S 2 based on the eye gaze experience is as follows:
(6.1) respectively establishing a gazing time weighting Gao Situ GS t, a gazing sequence weighting map GS r and a gazing center distance weighting map GS d by taking Euclidean distances d (r, v) of each gazing point r time, sequence and cluster center point v as weights. Based on the weighted three graphs of weights α, β, γ, a multi-target detection gaze Gao Situ GS is formed:
GS＝αGSt+βGSr+γGSd (5)
(6.2) carrying out two classifications on the multi-target detection gazing Gao Situ GS by adopting a linear support vector machine, wherein the two classifications are used for searching a hyperplane and distinguishing gazing significant areas from non-gazing significant areas. The GS image is input into a linear support vector machine for training, a hyperplane is found, gazing significant points and non-gazing significant points are distinguished, and a linear hyperplane solving model is as follows:
in the formula (6), w is a weight, b is a deviation vector, and y is a sample true category label.
Through inputting training potential safety hazard images, solving optimal parameters w and b, constructing an attention model based on a target saliency map of human eye fixation experience, and extracting a target saliency map S 2 based on human eye fixation experience.
7. And (3) constructing a construction potential safety hazard visual bionic perception model by integrating a target significant attention model based on primary visual characteristics and human eye fixation experience, wherein the flow is shown in figure 1. Training the hidden danger identification experience of a machine vision learner, and accurately positioning the hidden danger exposure part. The construction safety hidden danger visual bionic perception model is constructed as follows:
Firstly, inputting images in a hidden danger image training database, then obtaining a primary visual characteristic target saliency map, and combining the primary visual characteristic target saliency map and the primary visual characteristic target saliency map to form a construction potential safety hazard visual bionic perception model.
8. On the basis of accurately positioning the hidden danger exposure part, the image target is further identified, and the specific process is as follows:
and (8.1) classifying a large number of experiment materials such as hidden danger pictures, videos, eye movement data and the like acquired on site according to hidden danger classification, and providing basic data for training convolutional neural network parameters.
(8.2) Inputting the original image of the hidden trouble part into the convolutional neural network VGG16, alternately and repeatedly processing the original image through a convolutional layer Conv and a pooling layer Pooling of the VGG16 network, and extracting an image feature vector through a full-connection layer and a softmax classification layer to obtain a convolutional feature map of the image of the hidden trouble part. The process of generating the convolution signature is as follows:
The red, green and blue are three primary colors, the colors which can be resolved by human eyes can be basically synthesized, the convolution operation is carried out on the original image of the input hidden trouble part, the three characteristic filters of red, green and blue are utilized to obtain the output of a three-channel, the strength of a numerical value at a certain position in the channel is the reaction to the strength of the current characteristic, namely the convolution characteristic diagram,
And (8.3) aiming at the extracted convolution feature map, adopting a EdgeBoxes method, utilizing the texture, edge color and other information of the feature map, determining the number of contours in the candidate frame and the number of edges overlapped with the edges of the candidate frame, finding out a candidate region possibly containing a target, and generating a regression boundary of the candidate region. And recombining the candidate region and the convolution feature diagram corresponding to the candidate region in a pooling layer, accessing a full-connection layer and a softmax classification layer, and detecting the target category and the region boundary. And training the parameters of the convolutional neural network by combining the established hidden danger image database. The hidden danger part image target recognition flow is shown in figure 2.
9. The hidden danger semantic relation reasoning process is as follows:
(9.1) taking irregular shape characteristics of targets such as constructors, machines and equipment into consideration, adopting a bounding box algorithm to establish a two-dimensional plane enveloping space of the construction target, and respectively extracting a maximum coordinate point (x, y) max and a minimum coordinate point (x, y) min of the enveloping space.
(9.2) Detecting image coordinate data of the target through the trained convolutional neural network parameters, calculating an image scaling transformation matrix a 1, a symmetrical transformation matrix a 2 and an angle transformation matrix a 3, and constructing a transformation matrix of coordinates (x, y) and actual coordinates (x ', y') on the image target graphRealizing the position mapping transformation of the image target and the actual space:
(9.3) establishing a hidden danger knowledge semantic discrimination model, wherein the process is as follows:
Converting the spatial position and the relation data thereof into semantic concept expression, comparing the semantic concept expression with a hidden danger standard knowledge base, extracting basic semantic concepts V 1、V2 of hidden danger information input and hidden danger standard knowledge base, positioning the structural position VL 1、VL2 of V 1、V2 in a semantic network, and measuring the distance Dist (VL 1,VL2) between network nodes of the two and the maximum hierarchy maxVL of the semantic network. In summary, the similarity S between the semantic elements V 1、V2 is calculated:
traversing the similarity between m semantic concepts VM= { V 11,V12,···,V1m } of hidden danger text input and n semantic concepts VN= { V 21,V22,···,V2n } in a hidden danger specification knowledge base by using a semantic element similarity algorithm, and calculating a similarity matrix Sim of hidden danger information input and the knowledge base:
In the formula (9), VM is m semantic concepts VM= { V 11,V12,···,V1m } of hidden danger text input, VN is n semantic concepts VN= { V 21,V22,···,V2n } in hidden danger specification knowledge base, and the VM is in a Sim matrix The similarity between any hidden danger text and the hidden danger standard knowledge base is obtained.
And outputting a maximum matching degree semantic concept set max { Sim }, and combining to form hidden danger judging information by using a hidden danger semantic similarity algorithm, so as to realize hidden danger automatic judgment. As shown in fig. 3.
The hidden danger is automatically judged, namely, the construction site is identified by utilizing machine vision, the obtained information is judging information of whether the hidden danger is needed to be judged, the similarity calculation is carried out on the judging information and a hidden danger standard knowledge base, if the judging information is matched with hidden danger information in the knowledge base, the hidden danger is identified in the construction site, early warning and correction are needed to be carried out, then a knowledge case is formed and integrated into the hidden danger standard knowledge base, and if the judging information is not matched with the hidden danger information in the knowledge base, the hidden danger is not hidden danger. The hidden danger automatic discrimination can automatically monitor and identify the construction potential safety hazard, has the advantages of high identification degree, all weather and low cost, and has great significance for reducing the safety accidents.
Claims (3)
1. The intelligent identification method for the construction potential safety hazards by fusing man-machine vision is characterized by comprising the following steps of:
step one: firstly, according to an eye movement instrument, obtaining eye movement characteristic parameters;
Secondly, carrying out coordinate conversion on the fixation clustering center points, solving a hidden danger detection optimal path, carrying out image stitching on hidden danger scenes on the hidden danger detection optimal path to obtain a large-view image, and further accurately positioning hidden danger positions in the large-view image according to group fixation point aggregation characteristics to obtain a target saliency map S 1 based on primary visual characteristics;
Thirdly, analyzing the eye movement data, distinguishing gazing significant areas from non-gazing significant areas, inputting training potential safety hazard images, and obtaining a target significant map S 2 based on eye gazing experience;
Step four, further identifying an image target on the basis of accurately positioning the hidden danger exposure part, and training convolutional neural network parameters of the hidden danger part by combining the established hidden danger image database;
step five, finally establishing a hidden danger knowledge semantic discrimination model, calculating a similarity matrix of hidden danger information input and a hidden danger knowledge base, combining to form hidden danger discrimination information, and integrating an overall process to realize hidden danger automatic discrimination;
In the second step, a target saliency map S 1 based on the primary visual characteristics is obtained, which includes the following steps:
Step 2.1: according to a hidden danger image training database, extracting coordinate parameters of red, green and blue color channels of an input image and pixel points of the image, analyzing basic characteristics of scene image brightness I, color RGB and direction LB of different scales, and training a target significant attention model based on primary visual characteristics:
（3）；
Step 2.2: for scale factors of high resolution c and low resolution S in an image scale space, calculating gradient difference values theta of central pixels and peripheral pixels of an image area, acquiring saliency maps I (c theta S), RGB (c theta S) and LB (c theta S) of brightness, color and direction features, normalizing the three saliency maps, combining feature weights w I、wRGB、wLB, and establishing a target saliency model S 1 based on primary visual features:
（4）；
In the third step, a target saliency map S 2 based on the eye fixation experience is obtained, which includes the following steps:
Step 3.1: respectively establishing a gazing time weighting Gao Situ GS t, a gazing sequence weighting map GS r and a gazing center distance weighting map GS d by taking the time and sequence of each gazing point r and the Euclidean distance d (r, v) from the clustering center point v as weights; based on the weighted three graphs of weights α, β, γ, a multi-target detection gaze Gao Situ GS is formed:
（5）；
Step 3.2: inputting the GS image into a linear support vector machine for training, constructing an attention model based on a target salient point of human eye gazing experience, solving optimal w and b parameters of a linear hyperplane model, carrying out two-classification on multi-target detection gazing Gao Situ GS, extracting a target salient point S 2 based on human eye gazing experience, finding out a hyperplane gazing salient point and a non-gazing salient point, and distinguishing a gazing salient region and a non-gazing salient region;
（6）；
in the fourth step, the image target is further identified, and the method specifically comprises the following steps:
step 4.1: classifying a large number of hidden trouble pictures, videos and eye movement data experimental materials acquired on site according to hidden trouble classification, and providing basic data for training convolutional neural network parameters of hidden trouble parts;
Step 4.2: inputting the original image of the hidden trouble part into a convolutional neural network VGG16, alternately and repeatedly processing the original image through a convolutional layer Conv and a pooling layer Pooling of the VGG16 network, and extracting an image feature vector through a full-connection layer and a softmax classification layer to obtain a convolutional feature map of the image of the hidden trouble part;
Step 4.3: aiming at the extracted convolution feature map, adopting EdgeBoxes method, utilizing texture and edge color information of the feature map to determine the number of contours in the candidate frame and the number of edges overlapped with the edges of the candidate frame, finding out candidate regions possibly containing targets, and generating regression boundary of the candidate regions; recombining the candidate region and the convolution feature diagram corresponding to the candidate region in a pooling layer, accessing a full-connection layer and a softmax classification layer, and detecting the target category and the region boundary; training convolutional neural network parameters of hidden danger parts by combining the established hidden danger image database;
The fifth step comprises the following steps:
Step 5.1: taking irregular shapes of constructors, machines and equipment targets into consideration, adopting a bounding box algorithm to establish a two-dimensional plane enveloping space of the construction target, and respectively extracting a maximum coordinate point (x, y) max and a minimum coordinate point (x, y) min of the enveloping space;
Step 5.2: detecting image coordinate data of a target through convolution neural network parameters of hidden danger parts, calculating an image scaling transformation matrix a 1, a symmetrical equal transformation matrix a 2 and an angle transformation matrix a 3, and constructing a transformation matrix of coordinates (x, y) and actual coordinates (x ', y') on an image target graph Realizing the position mapping transformation of the image target and the actual space:
（7）；
step 5.3: the hidden danger knowledge semantic discrimination model is established to realize hidden danger automatic discrimination, and the process is as follows:
Converting the spatial position and the relation data thereof into semantic concept expression, comparing the semantic concept expression with a hidden danger standard knowledge base, extracting basic semantic concepts V1 and V2 of hidden danger information input and the hidden danger standard knowledge base, positioning the structural positions VL1 and VL2 of V1 and V2 in a semantic network, and measuring the distances Dist (VL 1 and VL 2) between network nodes of the two semantic networks and the maximum level maxVL of the semantic network; in summary, the similarity S between the semantic elements V1, V2 is calculated:
（8）；
Traversing the similarity between m semantic concepts VM= { V11, V12, …, V1m } of the hidden danger text input and n semantic concepts VN= { V21, V22, …, V2n } in the hidden danger standard knowledge base by using a semantic element similarity algorithm, and calculating a similarity matrix Sim of the hidden danger information input and the knowledge base:
（9）；
and outputting a maximum matching degree semantic concept set max { Sim }, combining to form hidden danger discrimination information, and integrating the whole flow to realize hidden danger automatic discrimination through a hidden danger semantic similarity algorithm.
2. The intelligent recognition method of construction safety hidden trouble fusing man-machine vision according to claim 1, wherein the intelligent recognition method is characterized by comprising the following steps: in the first step, the first visual angle video, the fixation point coordinates and the long eye movement characteristic parameters during fixation point identification of the tested group are acquired through an eye movement instrument.
3. The intelligent recognition method of construction safety hidden trouble fusing man-machine vision according to claim 1, wherein the intelligent recognition method is characterized by comprising the following steps: in the second step, according to the eye imaging and the eye tracker data acquisition principle, the eye movement horizontal view angle alpha and the vertical view angle beta of the target point of the tested group are combined, the x, y and z coordinate axis view angle scaling px α(d)、pyβ(d)、pzα (d) is calculated according to the measurement distance d between the hidden trouble part space point and the tested group, and the fixation point coordinate based on the eye tracker coordinate system is obtainedCalculating the space coordinate/>, of the fixation point under the same origin of coordinates according to scalingThe space information mapping of the eye point of the eye movement instrument is realized;
（1）；
Taking the difference between the eye-tracker coordinate system and the world space coordinate system into consideration, taking the space coordinate of the point of regard as a basis, taking the real coordinate (X, Y, Z) of the point of regard based on the origin of the camera as a reference, establishing a space coordinate system conversion equation, solving a coordinate system conversion matrix M, and converting the point of regard coordinate acquired by the eye-tracker into a target point coordinate under the world space coordinate system;
（2）；
coordinate transformation is carried out on the fixation cluster center point, the fixation cluster center point is used as a key positioning point of an automatic glancing path of a camera monitoring construction site, the shortest sum of glances of all construction sites is used as an objective function, and an optimal path for hidden danger detection is solved; and performing image stitching on a plurality of clear local hidden danger scenes of the hidden danger detection path.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111290335.8A CN114219687B (en) | 2021-11-02 | 2021-11-02 | Intelligent construction safety hidden danger identification method integrating man-machine vision |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111290335.8A CN114219687B (en) | 2021-11-02 | 2021-11-02 | Intelligent construction safety hidden danger identification method integrating man-machine vision |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114219687A CN114219687A (en) | 2022-03-22 |
| CN114219687B true CN114219687B (en) | 2024-06-14 |
Family
ID=80696456
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111290335.8A Active CN114219687B (en) | 2021-11-02 | 2021-11-02 | Intelligent construction safety hidden danger identification method integrating man-machine vision |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114219687B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115908954B (en) * | 2023-03-01 | 2023-07-28 | 四川省公路规划勘察设计研究院有限公司 | Geological disaster hidden danger identification system and method based on artificial intelligence and electronic equipment |
| CN116310760B (en) * | 2023-04-06 | 2023-09-22 | 河南禹宏实业有限公司 | Intelligent water conservancy monitoring system based on machine vision |
| CN116630752B (en) * | 2023-07-25 | 2023-11-17 | 广东南方电信规划咨询设计院有限公司 | Construction site target object identification method and device based on AI algorithm |
| CN117952964B (en) * | 2024-03-25 | 2024-06-07 | 重庆医科大学绍兴柯桥医学检验技术研究中心 | Fundus medical image analysis method based on computer vision technology |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110069773A (en) * | 2019-03-12 | 2019-07-30 | 中国华能集团有限公司 | A kind of security risk intelligent identification Method, device and storage medium |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107833220B (en) * | 2017-11-28 | 2021-06-11 | 河海大学常州校区 | Fabric defect detection method based on deep convolutional neural network and visual saliency |
| CN111354028B (en) * | 2020-02-19 | 2022-05-31 | 山东大学 | Binocular vision-based power transmission channel hidden danger identification and tracking method |
| CN111272874A (en) * | 2020-03-04 | 2020-06-12 | 河南理工大学 | Automatic detection device for hidden roof collapse danger of roadway |
-
2021
- 2021-11-02 CN CN202111290335.8A patent/CN114219687B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110069773A (en) * | 2019-03-12 | 2019-07-30 | 中国华能集团有限公司 | A kind of security risk intelligent identification Method, device and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 眼动试验在建筑业安全隐患识别研究中的应用与展望;廖彬超;刘梅;徐晴雯;王燕卿;王小云;;中国安全科学学报;20161115(11);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114219687A (en) | 2022-03-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114219687B (en) | Intelligent construction safety hidden danger identification method integrating man-machine vision | |
| CN103886308B (en) | A kind of pedestrian detection method of use converging channels feature and soft cascade grader | |
| CN106250870B (en) | A kind of pedestrian's recognition methods again of joint part and global similarity measurement study | |
| CN110502965A (en) | A kind of construction safety helmet wearing monitoring method based on the estimation of computer vision human body attitude | |
| US20190147283A1 (en) | Deep convolutional neural networks for crack detection from image data | |
| CN110598736A (en) | Power equipment infrared image fault positioning, identifying and predicting method | |
| CN110175576A (en) | A kind of driving vehicle visible detection method of combination laser point cloud data | |
| CN109118479A (en) | Defects of insulator identification positioning device and method based on capsule network | |
| CN108734143A (en) | A kind of transmission line of electricity online test method based on binocular vision of crusing robot | |
| CN109635875A (en) | A kind of end-to-end network interface detection method based on deep learning | |
| CN112734692A (en) | Transformer equipment defect identification method and device | |
| CN110910378B (en) | Bimodal image visibility detection method based on depth fusion network | |
| CN109583324A (en) | A kind of pointer meters reading automatic identifying method based on the more box detectors of single-point | |
| CN108898085A (en) | Intelligent road disease detection method based on mobile phone video | |
| CN105844621A (en) | Method for detecting quality of printed matter | |
| CN107085696A (en) | A kind of vehicle location and type identifier method based on bayonet socket image | |
| CN106845502A (en) | It is a kind of to visualize guidance method for the Wearable servicing unit of overhaul of the equipments and overhaul of the equipments | |
| CN111985499B (en) | High-precision bridge apparent disease identification method based on computer vision | |
| CN113435282B (en) | Unmanned aerial vehicle image ear recognition method based on deep learning | |
| CN113920107A (en) | Insulator damage detection method based on improved yolov5 algorithm | |
| CN114399672A (en) | Railway wagon brake shoe fault detection method based on deep learning | |
| CN115272652A (en) | Dense object image detection method based on multiple regression and adaptive focus loss | |
| CN114812403B (en) | Large-span steel structure hoisting deformation monitoring method based on unmanned plane and machine vision | |
| CN109086803A (en) | A kind of haze visibility detection system and method based on deep learning and the personalized factor | |
| WO2023087420A1 (en) | Thermal infrared vision-based parking apron human body action recognition method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |