CN117974398A

CN117974398A - Construction safety supervision system based on multi-source data

Info

Publication number: CN117974398A
Application number: CN202410235225.9A
Authority: CN
Inventors: 吕迅; 王光伟; 彭凡; 马文豪; 韩建伟
Original assignee: Henan Hongtian Construction Engineering Co ltd
Current assignee: Henan Hongtian Construction Engineering Co ltd
Priority date: 2024-03-01
Filing date: 2024-03-01
Publication date: 2024-05-03

Abstract

The present disclosure relates to a construction safety supervision system based on multi-source data. It comprises the following steps: performing key frame sampling and then feature extraction on the obtained construction site monitoring video to obtain a construction site construction semantic feature map; after the noise reduction treatment is carried out on the obtained construction site sound data, carrying out feature analysis and gamma correction on the construction site sound data after noise reduction to obtain a corrected construction site sound waveform semantic feature map; and the construction site construction semantic feature map and the corrected construction site sound waveform semantic feature map are subjected to feature cross fusion network to obtain multi-mode monitoring semantic features of the construction site so as to determine whether abnormal construction exists in the construction site. Therefore, the safety condition of the construction site can be monitored in real time, and potential safety hazards can be found in time, so that the construction safety management level can be effectively improved, the construction safety accident rate is reduced, and casualties and property loss are reduced.

Description

Construction safety supervision system based on multi-source data

Technical Field

The disclosure relates to the technical field of construction safety, in particular to a construction safety supervision system based on multi-source data.

Background

Construction safety is an important component of the construction industry and is a complex and difficult task. Many potential safety hazards exist in a construction site, such as high-altitude operation, mechanical equipment operation, electric operation and the like, and the potential safety hazards can cause safety accidents at any time. Therefore, the potential safety hazard of the construction site is discovered and eliminated in time, and the method has important significance for guaranteeing the construction safety.

However, the traditional construction safety supervision method mainly depends on manual inspection and checking, a large amount of manpower and material resources are needed in the mode, the inspection frequency and coverage area are limited, that is, the manual inspection and checking only can cover limited areas, blind areas often exist, and some potential safety hazards are easily missed. Moreover, manual inspection and examination is easily affected by subjective factors, and the expertise level and experience of the inspector are limited, so that the potential safety hazard is easily identified inaccurately.

Accordingly, an optimized construction safety supervision system is desired.

Disclosure of Invention

This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

In a first aspect, the present disclosure provides a construction safety supervision system based on multi-source data, the system comprising:

the construction site monitoring video acquisition module is used for acquiring a construction site monitoring video acquired by the camera;

the construction site sound data acquisition module is used for acquiring construction site sound data acquired by the sound sensor;

the key frame sampling module is used for carrying out key frame sampling on the construction site monitoring video to obtain a sequence of construction site monitoring key frames;

The construction site construction semantic feature extraction module is used for extracting features of the sequence of the construction site monitoring key frames through a construction site construction semantic feature extractor based on a deep neural network model so as to obtain a construction site construction semantic feature map;

the construction site sound waveform semantic feature analysis correction module is used for carrying out feature analysis and gamma correction on the construction site sound data after noise reduction treatment on the construction site sound data so as to obtain a corrected construction site sound waveform semantic feature map;

the construction site multi-mode monitoring semantic feature expression module is used for enabling the construction site construction semantic feature map and the corrected construction site sound waveform semantic feature map to be subjected to feature cross fusion to obtain construction site multi-mode monitoring semantic features;

The construction site abnormality detection module is used for determining whether abnormal construction exists on the construction site or not based on the construction site multi-mode monitoring semantic features.

Optionally, the deep neural network model is a three-dimensional convolutional neural network model.

Optionally, the construction site sound waveform semantic feature analysis and correction module includes: the construction site sound waveform feature extraction unit is used for carrying out noise reduction treatment on the construction site sound data, and then passing the noise-reduced construction site sound data through a construction site sound waveform feature extractor based on a convolutional neural network model to obtain a construction site sound waveform semantic feature map; the waveform semantic feature gamma correction module is used for carrying out gamma correction on the construction site sound waveform semantic feature map to obtain a corrected construction site sound waveform semantic feature map.

Optionally, the waveform semantic feature gamma correction module is configured to: gamma correction is carried out on the construction site sound waveform semantic feature map by the following gamma correction formula so as to obtain a corrected construction site sound waveform semantic feature map; wherein, the gamma correction formula is:

L _i is a normalized feature map of the construction site sound waveform semantic feature map, A, B, C and D are adjustment super-parameters, and L is the corrected construction site sound waveform semantic feature map.

Optionally, the construction site multi-mode monitoring semantic feature expression module is used for: processing the construction site construction semantic feature map and the corrected construction site sound waveform semantic feature map through the feature cross fusion network according to the following fusion formula to obtain a construction site multi-mode monitoring semantic feature map as the construction site multi-mode monitoring semantic feature; wherein, the fusion formula is:

x_g＝AvgPool(x_cat)+MaxPool(x_cat)

x_out＝x₁*ω₁+x₂*ω₂

Wherein x _cat is a multi-modal semantic feature map of the construction site, which is obtained by aggregating the semantic feature map of the construction site construction and the corrected semantic feature map of the sound waveform of the construction site along the channel dimension, avgPool (·) represents global average pooling processing of each feature matrix in the feature map along the channel dimension, maxPool (·) represents maximum pooling processing of each feature matrix in the feature map along the channel dimension, x _g is a pooled fusion feature map, Full connection processing with node number C/r, r representing scaling super parameter, C representing channel number of the pooling fusion feature map, reLU (·) representing ReLU function, z _r representing channel feature map,/>And/>Respectively representing the full connection processing of different node numbers, wherein z ₁ is a first full connection characteristic diagram, z ₂ is a second full connection characteristic diagram,/>Index operation for the first full connection feature map,/>And performing exponential operation on the second full-connection feature map, wherein omega ₁ [ i ] is a first weight, omega ₂ [ i ] is a second weight, x ₁ and x ₂ are the construction site construction semantic feature map and the corrected construction site sound waveform semantic feature map respectively, and x _out is the construction site multi-mode monitoring semantic feature map.

Optionally, the construction site abnormality detection module includes: the construction site multi-mode monitoring semantic feature optimization unit is used for performing feature distribution optimization on the construction site multi-mode monitoring semantic feature map to obtain an optimized construction site multi-mode monitoring semantic feature map; the abnormal construction detection module is used for enabling the optimized construction site multi-mode monitoring semantic feature diagram to pass through a construction safety monitoring result generator based on a classifier to obtain a construction safety monitoring result, and the construction safety monitoring result is used for indicating whether abnormal construction exists on the construction site.

Optionally, the job site multi-mode monitoring semantic feature optimizing unit includes: the linear change subunit is used for converting each feature matrix along the channel dimension in the multi-mode monitoring semantic feature map of the construction site into a square matrix through linear transformation so as to obtain a multi-mode monitoring semantic feature map of the construction site after conversion; and the optimization correction subunit is used for carrying out optimization correction based on the adjacent feature matrix on the converted multi-mode monitoring semantic feature map of the construction site by taking the feature matrix as a unit so as to obtain the optimized multi-mode monitoring semantic feature map of the construction site.

Optionally, the abnormal construction detection module includes: the matrix unfolding unit is used for unfolding the optimized multi-mode monitoring semantic feature map of the construction site into classification feature vectors according to row vectors or column vectors; the full-connection coding unit is used for carrying out full-connection coding on the classification characteristic vectors by using a plurality of full-connection layers of the classifier so as to obtain coded classification characteristic vectors; and the classification unit is used for passing the coding classification feature vector through a Softmax classification function of the classifier to obtain the classification result.

By adopting the technical scheme, the construction semantic feature map of the construction site is obtained by performing key frame sampling and then feature extraction on the obtained construction site monitoring video; after the noise reduction treatment is carried out on the obtained construction site sound data, carrying out feature analysis and gamma correction on the construction site sound data after noise reduction to obtain a corrected construction site sound waveform semantic feature map; and the construction site construction semantic feature map and the corrected construction site sound waveform semantic feature map are subjected to feature cross fusion network to obtain multi-mode monitoring semantic features of the construction site so as to determine whether abnormal construction exists in the construction site. Therefore, the safety condition of the construction site can be monitored in real time, and potential safety hazards can be found in time, so that the construction safety management level can be effectively improved, the construction safety accident rate is reduced, and casualties and property loss are reduced.

Additional features and advantages of the present disclosure will be set forth in the detailed description which follows.

Drawings

The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by reference to the following detailed description when taken in conjunction with the accompanying drawings. The same or similar reference numbers will be used throughout the drawings to refer to the same or like elements. It should be understood that the figures are schematic and that elements and components are not necessarily drawn to scale. In the drawings:

FIG. 1 is a block diagram illustrating a multi-source data based construction safety supervision system in accordance with an example embodiment.

Fig. 2 is a flow chart illustrating a construction safety supervision method based on multi-source data according to an example embodiment.

Fig. 3 is a block diagram of an electronic device, according to an example embodiment.

Fig. 4 is an application scenario diagram illustrating a multi-source data based construction safety supervision system according to an example embodiment.

Detailed Description

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure have been shown in the accompanying drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustration purposes only and are not intended to limit the scope of the present disclosure.

It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order and/or performed in parallel. Furthermore, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.

The term "including" and variations thereof as used herein are intended to be open-ended, i.e., including, but not limited to. The term "based on" is based at least in part on. The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments. Related definitions of other terms will be given in the description below.

It should be noted that the terms "first," "second," and the like in this disclosure are merely used to distinguish between different devices, modules, or units and are not used to define an order or interdependence of functions performed by the devices, modules, or units.

It should be noted that references to "one", "a plurality" and "a plurality" in this disclosure are intended to be illustrative rather than limiting, and those of ordinary skill in the art will appreciate that "one or more" is intended to be understood as "one or more" unless the context clearly indicates otherwise.

The names of messages or information interacted between the various devices in the embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

Many potential safety hazards exist in a construction site, such as high-altitude operation, mechanical equipment operation, electric operation and the like, and the potential safety hazards can cause safety accidents at any time. Therefore, the potential safety hazard of the construction site is discovered and eliminated in time, and the method has important significance for guaranteeing the construction safety.

The potential safety hazards of high-altitude operation include: the high-altitude operation personnel fall from the high place, and the casualties can be caused. When working aloft, the object falls from the high place, and the ground personnel can be injured or equipment can be damaged. The personnel in the high altitude contact the electrified body, which may cause electric shock accidents.

The potential safety hazards of mechanical equipment operation include: mechanical equipment fails, and casualties or equipment damage may result. Improper operation of mechanical equipment may cause injury or damage to the equipment. Improper maintenance of the machine may result in the occurrence of machine failure.

The potential safety hazards of electrical operation include: the electrical circuit malfunctions, which may cause an electric shock accident or fire. The electrical equipment malfunctions, which may cause an electric shock accident or fire. Improper electrical operation may cause an electric shock accident or fire.

Other potential safety hazards include: sundries are accumulated on a construction site, and tripping and sliding accidents can be caused. Insufficient illumination on the construction site can cause that personnel cannot see the construction environment clearly, and accidents are easy to occur. The construction site is not smooth in ventilation, and oxygen deficiency or poisoning of personnel can be caused.

Measures for eliminating potential safety hazards of construction sites are as follows: safety education and training are enhanced, safety education and training are carried out on constructors, and safety consciousness and skills of constructors are improved. And establishing a sound safety management system, formulating a perfect safety management system, defining the safety responsibility of personnel at all levels, and establishing the sound safety management system. And safety inspection and supervision are enhanced, the construction site is periodically subjected to safety inspection, and potential safety hazards are timely found and eliminated. The necessary safety protection facilities are provided, so that the safety of the constructors is ensured by providing the constructors with the necessary safety protection facilities such as safety helmets, safety belts, safety nets and the like. An emergency plan is established, perfect emergency plans are formulated, responsibilities and emergency measures of personnel at all levels are clarified, and timely and effective coping is ensured when safety accidents occur. Strengthen safe cultural construction, build good safe atmosphere, improve constructor's security consciousness and responsibility sense.

By adopting the measures, the occurrence of construction safety accidents can be effectively prevented and reduced, and the construction safety is ensured.

However, the conventional construction safety supervision method mainly relies on manual inspection and examination, and has the following defects: the manual inspection and examination requires a large amount of manpower and material resources, and particularly in a large-scale construction site, a large amount of personnel and time are required to be invested for inspection and examination. Manual inspection and examination can only cover a limited area, often has blind areas, and easily leaks some potential safety hazards. The manual inspection and examination is easily affected by subjective factors of inspectors, such as experience, professional level, responsibility center and the like, and the identification of potential safety hazards is easy to be inaccurate. The professional level and experience of manual inspection and inspection personnel are limited, and inaccurate or untimely identification of potential safety hazards is easily caused. The manual inspection and examination can only be performed at a specific time, and the safety condition of the construction site cannot be monitored in real time. The records of manual inspection and inspection are often paper and difficult to trace back and manage. These defects lead to the difficulty in effectively identifying and eliminating potential safety hazards in the traditional construction safety supervision method, and easily lead to the occurrence of safety accidents.

With the development of information technology, a construction safety supervision system based on multi-source data fusion is generated. The system can collect video, sound, image and other data of a construction site by utilizing various devices such as cameras, sensors and the like, and analyze and process the data through a data fusion technology, so that potential safety hazards of the construction site can be found in time, and decision support is provided for construction safety management.

Based on the above, in the technical scheme of the application, a construction safety supervision system based on multi-source data is provided, which can monitor and collect construction site monitoring videos in real time through a camera, simultaneously monitor and collect construction site sound data in real time by utilizing a sound sensor, and introduce artificial intelligence-based data processing and analysis algorithm at the rear end to perform collaborative analysis of the construction site monitoring videos and the construction site sound data, so as to realize real-time monitoring of safety conditions of construction sites, such as abnormal construction, so as to discover potential safety hazards in time. Therefore, the safety condition of the construction site can be monitored in real time, and potential safety hazards can be found in time, so that the construction safety management level can be effectively improved, the construction safety accident rate is reduced, and casualties and property loss are reduced.

In order to solve the problems, the disclosure provides a construction safety supervision system based on multi-source data, which performs key frame sampling and then feature extraction on an obtained construction site monitoring video to obtain a construction site construction semantic feature map; after the noise reduction treatment is carried out on the obtained construction site sound data, carrying out feature analysis and gamma correction on the construction site sound data after noise reduction to obtain a corrected construction site sound waveform semantic feature map; and the construction site construction semantic feature map and the corrected construction site sound waveform semantic feature map are subjected to feature cross fusion network to obtain multi-mode monitoring semantic features of the construction site so as to determine whether abnormal construction exists in the construction site. Therefore, the safety condition of the construction site can be monitored in real time, and potential safety hazards can be found in time, so that the construction safety management level can be effectively improved, the construction safety accident rate is reduced, and casualties and property loss are reduced.

Specific embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

As shown in fig. 1, the construction safety supervision system 100 includes:

the construction site monitoring video acquisition module 101 is used for acquiring a construction site monitoring video acquired by a camera;

a construction site sound data acquisition module 102 for acquiring construction site sound data acquired by the sound sensor;

A key frame sampling module 103, configured to sample the key frame of the construction site monitoring video to obtain a sequence of construction site monitoring key frames;

the construction site construction semantic feature extraction module 104 is configured to perform feature extraction on the sequence of the construction site monitoring key frames through a construction site construction semantic feature extractor based on a deep neural network model to obtain a construction site construction semantic feature map;

The construction site sound waveform semantic feature analysis and correction module 105 is used for performing feature analysis and gamma correction on the construction site sound data after noise reduction treatment on the construction site sound data to obtain a corrected construction site sound waveform semantic feature map;

The construction site multi-mode monitoring semantic feature expression module 106 is configured to obtain construction site multi-mode monitoring semantic features by using the construction site construction semantic feature map and the corrected construction site sound waveform semantic feature map through a feature cross fusion network;

A construction site abnormality detection module 107, configured to determine whether abnormal construction exists in a construction site based on the construction site multi-mode monitoring semantic features.

The deep neural network model is a three-dimensional convolutional neural network model.

Specifically, in the technical scheme of the application, firstly, a construction site monitoring video collected by a camera is obtained, and construction site sound data collected by a sound sensor is obtained. It will be appreciated that direct analysis of video can take up significant computing resources due to the large amount of video data. Therefore, in the technical scheme of the application, the key frame sampling is required to be carried out on the construction site monitoring video so as to obtain a sequence of the construction site monitoring key frames. In particular, key frames herein refer to representative frames in a video that may reflect the primary content and changes of the video. By carrying out key frame sampling on the video, the data volume of the video can be greatly reduced, and meanwhile, the main information of the video is reserved, so that the analysis efficiency and the accuracy of identifying potential safety hazards on a construction site are improved.

Then, the key frames in the construction site monitoring video are considered to contain important information of the construction site, and meanwhile, a lot of information is not necessary for identifying potential safety hazards of the construction site. Therefore, in order to extract features with semantic information from video data so as to more accurately identify potential safety hazards of a construction site, in the technical scheme of the application, the sequence of the construction site monitoring key frames is further subjected to feature mining in a construction site construction semantic feature extractor based on a three-dimensional convolutional neural network model so as to extract construction semantic features about the construction site in the sequence of the construction site monitoring key frames, including scenes and personnel actions displayed in video, so that a construction site construction semantic feature map is obtained. By extracting construction semantic features of the construction site, the construction condition of the construction site can be analyzed, so that potential safety hazards can be identified more accurately.

In one embodiment of the present disclosure, the job site sound waveform semantic feature analysis correction module includes: the construction site sound waveform feature extraction unit is used for carrying out noise reduction treatment on the construction site sound data, and then passing the noise-reduced construction site sound data through a construction site sound waveform feature extractor based on a convolutional neural network model to obtain a construction site sound waveform semantic feature map; the waveform semantic feature gamma correction module is used for carrying out gamma correction on the construction site sound waveform semantic feature map to obtain a corrected construction site sound waveform semantic feature map.

It should be appreciated that, due to the abundant information contained in the sound data of the construction site, various noise interferences, such as machine sounds, workers' noise, etc., are also generally generated in the construction site, and these noises affect the quality and accuracy of the sound data. Therefore, in order to extract meaningful semantic features from sound data to help a system better understand conditions and potential safety hazards of a construction site, in the technical scheme of the application, after noise reduction processing is required to be performed on the sound data of the construction site, feature mining is performed on the sound data of the construction site after noise reduction in a construction site sound waveform feature extractor based on a convolutional neural network model so as to extract waveform semantic feature information of the sound data of the construction site, and thus a construction site sound waveform semantic feature map is obtained. By extracting the semantic features of the sound waveform of the construction site, the sound information of the construction site can be better analyzed to identify potential safety hazards or abnormal conditions possibly existing, so that the accuracy and the efficiency of safety management are improved.

Further, considering that the quality and readability of sound data of a construction site are low, the recognition capability and understanding capability of sound information of the construction site are poor. Therefore, in order to improve the quality and the readability of sound data and better support the subsequent identification and analysis of potential safety hazards, in the technical scheme of the application, gamma correction is further carried out on the construction site sound waveform semantic feature map so as to obtain a corrected construction site sound waveform semantic feature map. After the gamma correction processing, the details of the sound data can be clearer, and the information in the sound data can be analyzed more accurately. That is, gamma correction can make visual effects of sound data more balanced and natural, so that the sound data can be more easily understood and analyzed, thereby improving recognition ability of sound information of a construction site. And, gamma correction can help eliminate nonlinear effects in sound data, making the data more accurate and reliable. The method is beneficial to optimizing the expression form of the sound data and improving the quality and the readability of the sound data, so that the safety hidden trouble identification and analysis of a construction site are better supported.

In one embodiment of the present disclosure, the waveform semantic feature gamma correction module is configured to: gamma correction is carried out on the construction site sound waveform semantic feature map by the following gamma correction formula so as to obtain a corrected construction site sound waveform semantic feature map; wherein, the gamma correction formula is:

It should be appreciated that since video and sound are two different data modalities in a job site, they contain complementary semantic information for the job site. Specifically, the construction semantic feature map of the construction site and the corrected construction site sound waveform semantic feature map respectively represent the construction semantic feature and the sound waveform semantic feature of the construction site. Therefore, in order to effectively cross and integrate the characteristics of video and sound data of a construction site, so as to improve the characteristic representation capability of the characteristics and enable a system to better understand the condition of the construction site, in the technical scheme of the application, the construction semantic feature map of the construction site and the corrected construction site sound waveform semantic feature map are further subjected to a characteristic cross fusion network to obtain a multi-mode monitoring semantic feature map of the construction site, so that more comprehensive and rich multi-mode feature representation is obtained, and more accurate identification and analysis of potential safety hazards of the construction site are supported. It should be understood that, unlike direct simple addition of feature graphs, the feature cross fusion network can integrate global information on channels to assign corresponding weights to the construction site construction semantic feature graphs and the corrected construction site sound waveform semantic feature graphs, so that the network can select appropriate features for response and training, and thus fuse feature information from different sources or modalities, and thus obtain a more comprehensive and accurate construction site semantic feature representation. That is, the feature cross fusion network can fuse semantic feature information of video and sound data of a construction site together by learning the relationship and complementarity between semantic features of different modalities. The fused feature map combines the semantic information of the video and sound data of the construction site, can provide more comprehensive audio-visual information, and is helpful for the system to analyze the condition of the construction site more comprehensively and identify potential safety hazards or abnormal conditions.

In one embodiment of the disclosure, the job site multimodal monitoring semantic feature expression module is configured to: processing the construction site construction semantic feature map and the corrected construction site sound waveform semantic feature map through the feature cross fusion network according to the following fusion formula to obtain a construction site multi-mode monitoring semantic feature map as the construction site multi-mode monitoring semantic feature; wherein, the fusion formula is:

x_g＝AvgPool(x_cat)+MaxPool(x_cat)

x_out＝x₁*ω₁+x₂*ω₂

In one embodiment of the present disclosure, the job site anomaly detection module includes: the construction site multi-mode monitoring semantic feature optimization unit is used for performing feature distribution optimization on the construction site multi-mode monitoring semantic feature map to obtain an optimized construction site multi-mode monitoring semantic feature map; the abnormal construction detection module is used for enabling the optimized construction site multi-mode monitoring semantic feature diagram to pass through a construction safety monitoring result generator based on a classifier to obtain a construction safety monitoring result, and the construction safety monitoring result is used for indicating whether abnormal construction exists on the construction site.

Further, the construction site multi-mode monitoring semantic feature optimizing unit comprises: the linear change subunit is used for converting each feature matrix along the channel dimension in the multi-mode monitoring semantic feature map of the construction site into a square matrix through linear transformation so as to obtain a multi-mode monitoring semantic feature map of the construction site after conversion; and the optimization correction subunit is used for carrying out optimization correction based on the adjacent feature matrix on the converted multi-mode monitoring semantic feature map of the construction site by taking the feature matrix as a unit so as to obtain the optimized multi-mode monitoring semantic feature map of the construction site.

In the above technical scheme, the construction site construction semantic feature map and the corrected construction site sound waveform semantic feature map respectively represent construction site action semantic features of the monitoring video of the construction site based on three-dimensional convolution coding and sound waveform semantic features generated by the construction site. In this way, when the construction site construction semantic feature map and the corrected construction site sound waveform semantic feature map pass through a feature cross fusion network, the semantic feature distribution differences of all feature matrixes of the construction site construction semantic feature map and the corrected construction site sound waveform semantic feature map directly affect the distribution integrity among all feature matrixes of the construction site multi-mode monitoring semantic feature map in consideration of the source mode semantic distribution differences between the construction site monitoring video and the construction site sound data and the convolution coding dimension differences of the source mode semantic distribution differences and the source mode semantic distribution differences.

Based on the above, the applicant firstly converts each feature matrix in the multi-mode monitoring semantic feature map of the construction site into a square matrix through linear transformation, and then corrects the multi-mode monitoring semantic feature map of the construction site by taking the feature matrix as a unit based on optimization of adjacent feature matrices, specifically expressed as follows; converting each feature matrix along the channel dimension in the construction site multi-mode monitoring semantic feature map into a square matrix through linear transformation to obtain a converted construction site multi-mode monitoring semantic feature map; performing optimization correction based on adjacent feature matrixes on the converted multi-mode monitoring semantic feature map of the construction site by using the following optimization formula as a unit of the feature matrixes to obtain the optimized multi-mode monitoring semantic feature map of the construction site; wherein, the optimization formula is:

Wherein M _i and M _i+1 are respectively the ith and (i+1) th feature matrices in the converted multi-modal monitoring semantic feature map of the construction site, and And/>Global mean of feature matrices M _i and M _i+1, respectively,/>Is the transpose matrix of the (i+1) th feature matrix in the multi-modal monitoring semantic feature map of the construction site after the conversion of the multi-modal monitoring semantic feature map of the construction site,/>Is the reciprocal of the (i+1) th feature matrix in the multi-modal monitoring semantic feature map of the construction site after the conversion of the multi-modal monitoring semantic feature map of the construction site, M' _i+1 is the (i+1) th feature matrix in the multi-modal monitoring semantic feature map of the construction site after the optimization, and is the (i)/>Represents matrix multiplication, +..

The method comprises the steps of carrying out robust aggregation and sub-sampling proposal on each characteristic value of a characteristic matrix of a multi-mode monitoring semantic characteristic map of a construction site by taking the characteristic matrix of the multi-mode monitoring semantic characteristic map of the construction site as a seed point of scene transmission in a channel dimension along the center of the channel distribution, so as to carry out directional constraint transmission on a distribution boundary frame of an adjacent characteristic matrix on the basis of participation of each characteristic value of the characteristic matrix of the multi-mode monitoring semantic characteristic map of the construction site, and further improve the integrity of characteristic representation of the multi-mode monitoring semantic characteristic map of the construction site on the basis of context correlation of the whole multi-mode monitoring semantic characteristic map of the construction site from bottom to top along the channel dimension, thereby improving the accuracy of classification results obtained by a classifier of the multi-mode monitoring semantic characteristic map of the construction site. Therefore, the safety condition of the construction site can be monitored in real time, so that potential safety hazards can be found in time, the construction safety management level can be effectively improved, the occurrence rate of construction safety accidents is reduced, and casualties and property loss are reduced.

And then, the optimized multi-mode monitoring semantic feature map of the construction site passes through a construction safety monitoring result generator based on a classifier to obtain a construction safety monitoring result, wherein the construction safety monitoring result is used for indicating whether abnormal construction exists in the construction site. That is, the multi-mode monitoring semantic features of the construction site are utilized to conduct classification processing, so that the safety conditions of the construction site, such as abnormal construction, are monitored in real time, potential safety hazards can be found in time, the occurrence rate of construction safety accidents is reduced, and casualties and property loss are reduced. Accordingly, in a specific example of the present application, when a potential safety hazard is found, the safety supervision system may timely send an early warning to the constructor to remind the constructor to take necessary safety measures. Meanwhile, the safety supervision system can also send the safety hidden trouble information to construction managers so that the construction managers can take measures in time to eliminate the safety hidden trouble.

In one embodiment of the present disclosure, the abnormal construction detection module includes: the matrix unfolding unit is used for unfolding the optimized multi-mode monitoring semantic feature map of the construction site into classification feature vectors according to row vectors or column vectors; the full-connection coding unit is used for carrying out full-connection coding on the classification characteristic vectors by using a plurality of full-connection layers of the classifier so as to obtain coded classification characteristic vectors; and the classification unit is used for passing the coding classification feature vector through a Softmax classification function of the classifier to obtain the classification result.

In summary, by adopting the above scheme, the construction site monitoring video can be monitored and collected in real time through the camera, meanwhile, the sound sensor is utilized to monitor and collect the sound data of the construction site in real time, and the artificial intelligence-based data processing and analyzing algorithm is introduced into the rear end to perform collaborative analysis on the construction site monitoring video and the construction site sound data, so that the safety condition of the construction site, such as real-time monitoring of abnormal construction, is realized, and the potential safety hazard can be found in time. Therefore, the safety condition of the construction site can be monitored in real time, and potential safety hazards can be found in time, so that the construction safety management level can be effectively improved, the construction safety accident rate is reduced, and casualties and property loss are reduced.

FIG. 2 is a flow chart illustrating a method of construction safety supervision based on multi-source data, as shown in FIG. 2, according to an example embodiment, the method comprising:

step 201, acquiring a construction site monitoring video acquired by a camera;

Step 202, acquiring construction site sound data acquired by a sound sensor;

Step 203, performing key frame sampling on the construction site monitoring video to obtain a sequence of construction site monitoring key frames;

204, extracting features of the sequence of the construction site monitoring key frames through a construction site construction semantic feature extractor based on a deep neural network model to obtain a construction site construction semantic feature map;

Step 205, after noise reduction processing is performed on the construction site sound data, performing feature analysis and gamma correction on the noise-reduced construction site sound data to obtain a corrected construction site sound waveform semantic feature map;

206, the construction site construction semantic feature map and the corrected construction site sound waveform semantic feature map are subjected to feature cross fusion network to obtain construction site multi-mode monitoring semantic features;

step 207, determining whether abnormal construction exists in the construction site based on the multi-mode monitoring semantic features of the construction site.

Referring now to fig. 3, a schematic diagram of an electronic device 600 suitable for use in implementing embodiments of the present disclosure is shown. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and the like, and stationary terminals such as digital TVs, desktop computers, and the like. The electronic device shown in fig. 3 is merely an example and should not be construed to limit the functionality and scope of use of the disclosed embodiments.

As shown in fig. 3, the electronic device 600 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 601, which may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage means 608 into a Random Access Memory (RAM) 603. In the RAM603, various programs and data required for the operation of the electronic apparatus 600 are also stored. The processing device 601, the ROM602, and the RAM603 are connected to each other through a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.

In general, the following devices may be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, and the like; an output device 607 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 608 including, for example, magnetic tape, hard disk, etc.; and a communication device 609. The communication means 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. While fig. 3 shows an electronic device 600 having various means, it is to be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may be implemented or provided instead.

In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a non-transitory computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via communication means 609, or from storage means 608, or from ROM 602. The above-described functions defined in the methods of the embodiments of the present disclosure are performed when the computer program is executed by the processing device 601.

It should be noted that the computer readable medium described in the present disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.

In some embodiments, the clients, servers may communicate using any currently known or future developed network protocol, such as HTTP (HyperText TransferProtocol ), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the internet (e.g., the internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.

The computer readable medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device.

Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, including, but not limited to, an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules described in the embodiments of the present disclosure may be implemented in software or hardware. The name of the module is not limited to the module itself in some cases, and for example, the test parameter obtaining module may also be described as "a module for obtaining the device test parameter corresponding to the target device".

The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a system on a chip (SOC), a Complex Programmable Logic Device (CPLD), and the like.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

In accordance with one or more embodiments of the present disclosure, example 1 provides a construction safety supervision system based on multi-source data, the system comprising:

Example 2 provides the system of example 1, the deep neural network model being a three-dimensional convolutional neural network model, in accordance with one or more embodiments of the present disclosure.

In accordance with one or more embodiments of the present disclosure, example 3 provides the system of example 2, the job site acoustic waveform semantic feature analysis correction module comprising:

the construction site sound waveform feature extraction unit is used for carrying out noise reduction treatment on the construction site sound data, and then passing the noise-reduced construction site sound data through a construction site sound waveform feature extractor based on a convolutional neural network model to obtain a construction site sound waveform semantic feature map;

the waveform semantic feature gamma correction module is used for carrying out gamma correction on the construction site sound waveform semantic feature map to obtain a corrected construction site sound waveform semantic feature map.

In accordance with one or more embodiments of the present disclosure, example 4 provides the system of example 3, the waveform semantic feature gamma correction module to: gamma correction is carried out on the construction site sound waveform semantic feature map by the following gamma correction formula so as to obtain a corrected construction site sound waveform semantic feature map;

Wherein, the gamma correction formula is:

In accordance with one or more embodiments of the present disclosure, example 5 provides the system of example 4, the job site multimodal monitoring semantic feature expression module to: processing the construction site construction semantic feature map and the corrected construction site sound waveform semantic feature map through the feature cross fusion network according to the following fusion formula to obtain a construction site multi-mode monitoring semantic feature map as the construction site multi-mode monitoring semantic feature;

Wherein, the fusion formula is:

x_g＝AvgPool(x_cat)+MaxPool(x_cat)

x_out＝x₁*ω₁+x₂*ω₂

In accordance with one or more embodiments of the present disclosure, example 6 provides the system of example 5, the job site anomaly detection module comprising:

The construction site multi-mode monitoring semantic feature optimization unit is used for performing feature distribution optimization on the construction site multi-mode monitoring semantic feature map to obtain an optimized construction site multi-mode monitoring semantic feature map;

the abnormal construction detection module is used for enabling the optimized construction site multi-mode monitoring semantic feature diagram to pass through a construction safety monitoring result generator based on a classifier to obtain a construction safety monitoring result, and the construction safety monitoring result is used for indicating whether abnormal construction exists on the construction site.

According to one or more embodiments of the present disclosure, example 7 provides the system of example 6, the job site multimodal monitoring semantic feature optimization unit comprising:

The linear change subunit is used for converting each feature matrix along the channel dimension in the multi-mode monitoring semantic feature map of the construction site into a square matrix through linear transformation so as to obtain a multi-mode monitoring semantic feature map of the construction site after conversion;

And the optimization correction subunit is used for carrying out optimization correction based on the adjacent feature matrix on the converted multi-mode monitoring semantic feature map of the construction site by taking the feature matrix as a unit so as to obtain the optimized multi-mode monitoring semantic feature map of the construction site.

In accordance with one or more embodiments of the present disclosure, example 8 provides the system of example 7, the abnormal construction detection module comprising:

the matrix unfolding unit is used for unfolding the optimized multi-mode monitoring semantic feature map of the construction site into classification feature vectors according to row vectors or column vectors;

The full-connection coding unit is used for carrying out full-connection coding on the classification characteristic vectors by using a plurality of full-connection layers of the classifier so as to obtain coded classification characteristic vectors; and

And the classification unit is used for passing the coding classification feature vector through a Softmax classification function of the classifier to obtain the classification result.

Fig. 4 is an application scenario diagram illustrating a multi-source data based construction safety supervision system according to an example embodiment. As shown in fig. 4, in this application scenario, first, a construction site monitoring video acquired by a camera is acquired (e.g., C1 as illustrated in fig. 4); acquiring job site sound data (e.g., C2 as illustrated in fig. 4) collected by the sound sensor; then, the acquired construction site monitoring video and construction site sound data are input into a server (e.g., S as illustrated in fig. 4) deployed with a construction safety supervision algorithm based on the multi-source data, wherein the server is capable of processing the construction site monitoring video and the construction site sound data based on the construction safety supervision algorithm of the multi-source data to determine whether there is an abnormal construction at the construction site.

The foregoing description is only of the preferred embodiments of the present disclosure and description of the principles of the technology being employed. It will be appreciated by persons skilled in the art that the scope of the disclosure referred to in this disclosure is not limited to the specific combinations of features described above, but also covers other embodiments which may be formed by any combination of features described above or equivalents thereof without departing from the spirit of the disclosure. Such as those described above, are mutually substituted with the technical features having similar functions disclosed in the present disclosure (but not limited thereto).

Moreover, although operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. In certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are example forms of implementing the claims. The specific manner in which the various modules perform the operations in the apparatus of the above embodiments have been described in detail in connection with the embodiments of the method, and will not be described in detail herein.

Claims

1. A construction safety supervision system based on multi-source data, comprising:

2. The multi-source data based construction safety supervision system according to claim 1, wherein the deep neural network model is a three-dimensional convolutional neural network model.

3. The multi-source data based construction safety supervision system according to claim 2, wherein the construction site sound waveform semantic feature analysis correction module comprises:

4. The multi-source data based construction safety supervision system according to claim 3, wherein the waveform semantic feature gamma correction module is configured to: gamma correction is carried out on the construction site sound waveform semantic feature map by the following gamma correction formula so as to obtain a corrected construction site sound waveform semantic feature map;

Wherein, the gamma correction formula is:

5. The multi-source data based construction safety supervision system according to claim 4, wherein the job site multi-modal monitoring semantic feature expression module is configured to: processing the construction site construction semantic feature map and the corrected construction site sound waveform semantic feature map through the feature cross fusion network according to the following fusion formula to obtain a construction site multi-mode monitoring semantic feature map as the construction site multi-mode monitoring semantic feature;

Wherein, the fusion formula is:

x_g＝AvgPool(x_cat)+MaxPool(x_cat)

x_out＝x₁*ω₁+x₂*ω₂

6. The multi-source data based construction safety supervision system according to claim 5, wherein the job site anomaly detection module comprises:

7. The multi-source data based construction safety supervision system according to claim 6, wherein the job site multi-modal monitoring semantic feature optimization unit comprises:

8. The multi-source data based construction safety supervision system according to claim 7, wherein the abnormal construction detection module comprises: