CN114463594A - Multi-mode deep learning power generation equipment abnormity integrated identification method and equipment - Google Patents
Multi-mode deep learning power generation equipment abnormity integrated identification method and equipment Download PDFInfo
- Publication number
- CN114463594A CN114463594A CN202111400102.9A CN202111400102A CN114463594A CN 114463594 A CN114463594 A CN 114463594A CN 202111400102 A CN202111400102 A CN 202111400102A CN 114463594 A CN114463594 A CN 114463594A
- Authority
- CN
- China
- Prior art keywords
- data
- power generation
- modal
- module
- generation equipment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000010248 power generation Methods 0.000 title claims abstract description 91
- 238000000034 method Methods 0.000 title claims abstract description 43
- 238000013135 deep learning Methods 0.000 title claims abstract description 25
- 238000001514 detection method Methods 0.000 claims abstract description 69
- 230000004927 fusion Effects 0.000 claims description 78
- 238000012549 training Methods 0.000 claims description 36
- 238000000605 extraction Methods 0.000 claims description 35
- 238000013507 mapping Methods 0.000 claims description 34
- 230000005856 abnormality Effects 0.000 claims description 25
- 230000007246 mechanism Effects 0.000 claims description 19
- 238000007781 pre-processing Methods 0.000 claims description 16
- 238000013528 artificial neural network Methods 0.000 claims description 15
- 230000006870 function Effects 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 11
- 230000002159 abnormal effect Effects 0.000 claims description 10
- 238000004140 cleaning Methods 0.000 claims description 10
- 230000002457 bidirectional effect Effects 0.000 claims description 7
- 230000010354 integration Effects 0.000 claims description 6
- 238000000926 separation method Methods 0.000 claims description 6
- 238000010276 construction Methods 0.000 claims description 4
- 230000002902 bimodal effect Effects 0.000 claims description 3
- 238000013527 convolutional neural network Methods 0.000 claims description 2
- 230000008030 elimination Effects 0.000 claims description 2
- 238000003379 elimination reaction Methods 0.000 claims description 2
- 238000007526 fusion splicing Methods 0.000 claims description 2
- 230000008569 process Effects 0.000 abstract description 7
- 238000012544 monitoring process Methods 0.000 description 6
- 238000007689 inspection Methods 0.000 description 4
- 238000013473 artificial intelligence Methods 0.000 description 3
- 238000012512 characterization method Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000010801 machine learning Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000009467 reduction Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 238000003745 diagnosis Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000036541 health Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000012806 monitoring device Methods 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 239000003245 coal Substances 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000007500 overflow downdraw method Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Testing And Monitoring For Control Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a method and equipment for integrally identifying abnormity of power generation equipment through multi-mode deep learning. By the method and the device, the phenomena of missing report, false report and wrong report in the process of detecting the abnormity of the power generation equipment can be avoided, and the accuracy rate of predicting the abnormity detection of the power generation equipment is improved.
Description
Technical Field
The invention relates to the technical field of deep learning, artificial intelligence, neural networks and new energy, in particular to a multimode deep learning power generation equipment abnormity integrated identification method and device, computer equipment and a storage medium.
Background
New energy such as clean energy, artificial intelligence, big data and other core technologies are combined to form an integrated key technology, so that multi-modal data such as voice, image, video, text and the like are increasingly collected and artificially collected in application scenes such as photovoltaic scenes, wind turbines, offshore power generation and the like, and how are the maximum or optimal values of the mass data taken into account and how are effective features extracted? How to apply rich composite feature characterizations to the business systems of an application scenario? How to optimize combined feature characterization capabilities? These puzzles have become a challenge and a problem in industry, academia, and industry, and the prior art has not addressed the problem in terms of multimodal data and model fusion.
At present, on one hand, in the field of new energy resources, such as a fan and a photovoltaic scene, only single modal data and multi-modal data are used for obtaining effective characteristic representation insufficiency, such as characteristic representation information redundancy, inaccurate characteristic representation, insufficient characteristic representation precision and other defects, then, the functions of equipment abnormity detection and fault diagnosis, system defect early warning, system operation and maintenance and the like cannot meet the requirements or requirements of various services, particularly, when a certain equipment fault occurs, the phenomena of wrong report, missing report, late report and the like can occur when the output result is judged according to a single modal data construction model, and thus, inestimable loss is brought to a user in the field of new energy resources, including major accidents such as personal casualties, monitoring of the operation condition of individual equipment, equipment health inspection and the like; on the other hand, the traditional machine learning methods such as GMM, SVM, bayes or the traditional machine learning method combined learning in the past cannot meet the reliability and robustness requirements of a traditional machine learning model constructed by single-modal data and multi-modal data, and along with the rapid development of artificial intelligence and particularly deep learning, the method makes full use of the multi-modal data obtained by photovoltaic, wind turbine and offshore power generation equipment in the new energy field and the model fusion method to become the major trend of various application scenes and function requirements, particularly, the deep learning has strong feature extraction and characterization capability, and benefits such as cost reduction, efficiency improvement and the like for the new energy field, such as reduction of the risk of manual inspection operation, intelligent health and monitoring of the system, intelligent analysis of various operation indexes, labor and time saving, reduction of investment cost and the like in the operation analysis, so a new technical solution is needed to facilitate computers, and, GPU, data, etc.
Disclosure of Invention
The invention provides a multi-mode deep learning power generation equipment abnormity integrated identification method, a multi-mode deep learning power generation equipment abnormity integrated identification device, computer equipment and a storage medium, and aims to avoid the phenomena of missing report, false report and wrong report in the power generation equipment abnormity detection process and improve the accuracy of predicting the power generation equipment abnormity detection.
Therefore, a first object of the present invention is to provide a power generation equipment abnormality integration recognition method for multi-modal deep learning, comprising:
obtaining multi-modal historical data, preprocessing the data, and taking the preprocessed multi-modal historical data as a training set;
constructing a power generation equipment abnormity detection network model, and training the constructed power generation equipment abnormity detection network model through a training set; the power generation equipment anomaly detection network model comprises a feature extraction module, a space mapping module, a feature fusion module and a result prediction module which are connected in sequence;
and preprocessing real-time multi-modal data, inputting the preprocessed real-time multi-modal data into a trained power generation equipment abnormity detection network model, and outputting a result as a detection result of whether the power generation equipment is abnormal or not.
The method comprises the following steps of obtaining multi-modal historical data and preprocessing the data, wherein the step of obtaining the multi-modal historical data and preprocessing the data comprises the following steps:
multi-modal historical data acquisition: collecting automatic uploading multi-mode historical data obtained by a high-definition camera, a pickup and a sensor which are arranged around power generation equipment, and manually uploading the multi-mode historical data obtained by photographing through a mobile phone;
data cleaning: cleaning the data of the collected multi-modal historical data; wherein, the cleaning mode includes at least: data elimination and data completion;
data separation: and separating data according to bimodal or multimodal mixed data in the collected multimodal historical data, dividing the multimodal historical data into voice data, text data, image data and video data after the data are separated, and marking a detection result.
The power generation equipment anomaly detection network model comprises a feature extraction module, a space mapping module, a feature fusion module and a result prediction module; wherein,
the feature extraction module is a feature extraction neural network and is used for carrying out single-mode feature extraction on voice data, text data, image data and video data obtained after data separation;
the space mapping module is used for mapping the single-mode features to the same semantic space to obtain semantic structure information in the single-mode data features;
the characteristic fusion module is used for carrying out characteristic fusion splicing on the multimode historical data on the single-mode characteristics to obtain multimode characteristic fusion information;
and the result prediction module is used for calculating a prediction result according to the multi-modal feature fusion information to complete the abnormity detection of the power generation equipment.
Wherein, the characteristic extraction neural network is a BilSTM network, a convolution neural network or a deep neural network; the BilSTM network is applied, and the context information of the single-mode data, namely the semantic information between the adjacent single-mode data, is obtained while the characteristics are extracted.
The feature fusion module comprises a bidirectional attention mechanism unit, a self-attention mechanism unit and a first full-connection layer unit; the voice data, the text data, the image data and the video data which are mapped and processed by the space mapping module are sequentially input into the bidirectional attention mechanism unit, the self-attention mechanism unit and the first full-connection layer unit, and the output data are transmitted to the fusion module for fusion.
The method for training the established power generation equipment abnormity detection network model through the training set comprises the following steps:
inputting the preprocessed training set data into a feature extraction neural network of a feature extraction module, and respectively extracting the features of a single mode from voice data, text data, image data and video data in the training set data through a BilSTM network;
after single modal feature extraction is carried out on voice data, text data, image data and video data in training set data, semantic space mapping is carried out on the single modal features of the voice data, the text data, the image data and the video data;
respectively inputting single modal characteristics of the voice data, the text data, the image data and the video data which are output after the semantic space mapping into a multi-modal characteristic fusion mechanism; the multi-modal feature fusion mechanism comprises a first feature fusion module, a fourth feature fusion module, a first fusion module and a second fusion module; the method comprises the steps that single modal characteristics of voice data, text data, image data and video data which are output after semantic space mapping are input into a first-fourth characteristic fusion module respectively, output results of a first characteristic fusion module and a second characteristic fusion module are input into the first fusion module, and output results of the first fusion module, a third characteristic fusion module and the fourth characteristic fusion module are input into a second fusion module;
and the output result of the second fusion module is output to a result prediction module, a prediction result is calculated by utilizing a Softmax function, the result is compared with a marked detection result, and network training is completed by continuously adjusting network functions and parameters until the prediction result is consistent with the marked detection result.
If the abnormity detection result of the power generation equipment is obtained, displaying the abnormity detection result; the display mode at least comprises the following steps: text display, voice broadcast, outbound call terminal, mail, short message prompt, intelligent sound box.
A second object of the present invention is to provide a power generation equipment abnormality integrated recognition device for multimodal deep learning, including:
the data acquisition module is used for acquiring multi-modal historical data, preprocessing the data and taking the preprocessed multi-modal historical data as a training set;
the network construction module is used for constructing a power generation equipment abnormity detection network model and training the constructed power generation equipment abnormity detection network model through a training set; the power generation equipment anomaly detection network model comprises a feature extraction module, a space mapping module, a feature fusion module and a result prediction module which are connected in sequence;
and the abnormality detection module is used for preprocessing the real-time multi-modal data, inputting the preprocessed real-time multi-modal data into the trained power generation equipment abnormality detection network model, and outputting a result as a detection result of whether the power generation equipment is abnormal or not.
A third object of the present invention is to provide a computer device, which includes a memory, a processor and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the method according to the foregoing technical solution.
A fourth object of the invention is to propose a non-transitory computer-readable storage medium on which a computer program is stored, which computer program, when executed by a processor, implements the method of the aforementioned technical solution.
Different from the prior art, the multi-mode deep learning power generation equipment abnormity integrated identification method provided by the invention comprises the steps of constructing a power generation equipment abnormity detection neural network, extracting the characteristics of single-mode data through a characteristic extraction network, mapping the extracted characteristics to the same semantic space, performing characteristic fusion on the semantic characteristics of the single-mode data by adopting a multi-mode fusion strategy to generate multi-mode fusion characteristics, and predicting the power generation equipment abnormity according to the generated multi-mode fusion characteristics. By the method and the device, the phenomena of false alarm and missing alarm in the abnormity detection process of the power generation equipment can be avoided, and the abnormity detection accuracy of the power generation equipment is improved.
Drawings
The foregoing and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
FIG. 1 is a flow chart of a method for recognizing abnormal integration of power generation equipment through multi-modal deep learning according to the present invention.
Fig. 2 is a schematic structural diagram of a power generation equipment abnormality detection network model of the power generation equipment abnormality integrated identification method for multimodal deep learning according to the present invention.
Fig. 3 is a schematic structural diagram of a power generation equipment abnormality integration recognition device for multi-modal deep learning according to the present invention.
Fig. 4 is a schematic structural diagram of a non-transitory computer-readable storage medium according to the present invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are illustrative and intended to be illustrative of the invention and are not to be construed as limiting the invention.
Fig. 1 is a schematic flow chart of a multi-modal deep learning power generation equipment abnormality integrated identification method according to an embodiment of the present invention. The method comprises the following steps:
step 101, obtaining multi-modal historical data, performing data preprocessing, and taking the preprocessed multi-modal historical data as a training set.
The multi-modal data mentioned in the invention mainly refers to four modes of text, image, voice and video. At the present stage, a large number of devices such as cameras, sensors and sound pick-up devices are arranged around power generation equipment for monitoring, but data generated by the devices are checked to judge whether the devices are in failure or not through manual inspection, a large number of data can be deleted directly after being generated, and the monitoring devices cannot effectively complete monitoring tasks while misjudgment is caused through manual inspection, so that data redundancy or data waste is easily caused. The power generation equipment related by the invention comprises but is not limited to thermal power, hydroelectric power, clean energy power generation equipment or a generator set, wherein the clean energy power generation equipment or the generator set comprises but is not limited to photovoltaic power, offshore wind power, nuclear power generation equipment or a generator set. The power generation equipment abnormality integration includes, but is not limited to, abnormality detection, fault diagnosis, and fault warning of the power generation equipment.
Specifically, the multi-modal historical data is obtained by obtaining various monitoring devices, such as a high-definition camera, a sound pickup and a sensor, which are installed for monitoring the corresponding power generation devices from a management unit to which the power generation devices belong; the equipment automatically uploads the monitoring data of the power generation equipment to a monitoring center, which is called as automatic uploading multi-mode historical data in the invention; in addition, image, video or voice data uploaded manually through a mobile phone is called as manually uploading multi-modal historical data in the invention. The collection of the multi-modal historical data is to acquire a large amount of automatically uploaded multi-modal historical data and manually upload the multi-modal historical data.
After data is acquired, data cleaning needs to be performed on the acquired multi-modal historical data. In the process of data cleaning, data screening is firstly carried out, and data irrelevant to the abnormal detection of the power generation equipment are removed; and then judging the data quality of the multi-modal historical data, and cleaning the data by means of a script or a tool according to the low-quality data, such as manually adding noise or completing sentences.
And after cleaning, screening data belonging to more than two modes from the multi-mode data, and separating the data. If the bimodal data belonging to the voice and the video exist, the separation is carried out by means of a tool to obtain the monomodal data. As shown at 101 in fig. 2.
After the data preprocessing is completed, the process proceeds to step 102.
Step 102: and constructing a power generation equipment abnormity detection network model, and training the constructed power generation equipment abnormity detection network model through a training set.
The network structure of the power generation equipment anomaly detection network model constructed by the invention is shown in fig. 2 and comprises a feature extraction module 102, a space mapping module 103, a feature fusion module 104 and a result prediction module 105; wherein,
the feature extraction module 102 is a feature extraction neural network, and is configured to perform single-mode feature extraction on the voice data, the text data, the image data, and the video data obtained by data separation.
The characteristic extraction neural network is a BilSTM network, a convolutional neural network or a deep neural network; the BilSTM network is applied, and the context information of the single-mode data, namely the semantic information between the adjacent single-mode data, is obtained while the characteristics are extracted.
The space mapping module 103 is configured to map the single-mode features to the same semantic space, so as to obtain semantic structure information inside the single-mode data features. Through spatial mapping, the relevance inside the single-mode data features can be effectively represented, and the internal semantic structure information is determined.
The feature fusion module 104 is configured to perform feature fusion and concatenation on the multi-modal historical data on the single-modal features to obtain multi-modal feature fusion information.
The feature fusion module 104 comprises a bidirectional attention mechanism unit 10401, a self-attention mechanism unit 10402 and a first full-connection layer unit 10403; the voice data, the text data, the image data and the video data which are mapped and processed by the space mapping module are sequentially input into the bidirectional attention mechanism unit, the self-attention mechanism unit and the first full-connection layer unit, and the output data are transmitted to the fusion module for fusion.
And the result prediction module 105 is used for calculating a prediction result according to the multi-modal feature fusion information and completing the abnormity detection of the power generation equipment. The result prediction module 105 includes a second fully connected unit 1051 and a classifier unit 1052, which employs a Softmax classifier as shown in fig. 2.
As shown in fig. 2, for four data of different modalities, the network of the present invention is respectively provided with four feature extraction modules 102 and a space mapping module 103 which are connected in sequence, and the four single-modality data obtained after the preprocessing are respectively input into the feature extraction BiLSTM network to output the features of the four single-modality data; then, inputting the characteristics of the four single-mode data into a space mapping module 103 for space mapping; after the spatial mapping features of the four single-mode data are output, the four features are simultaneously input into the four feature fusion modules 104, that is, the spatial mapping features of the four single-mode data are simultaneously input into each feature fusion module 104. Because the collected multi-modal data have different contribution degrees to the abnormal detection of the predictive power generation equipment, in order to enhance the weight of certain modal data in a prediction result, the spatial mapping features of the text and the voice are subjected to time sequence feature fusion, and then the spatial mapping features of the text and the voice are subjected to merging operation with the features of different scales of the image and the video to obtain the final fusion features. Two attention mechanisms are used in the feature fusion module 104: one is that the bidirectional attention method explores the dependency of interactive features between two modes; the second is to explore the correlation between the prediction result and the single mode itself by the attention method.
The step of training the constructed power generation equipment abnormality detection network model through the training set comprises the following steps:
inputting the preprocessed training set data into the feature extraction neural network of the feature extraction module 102, and respectively performing single-mode feature extraction on the voice data, the text data, the image data and the video data in the training set data through a BilSTM network.
After single modal feature extraction is carried out on voice data, text data, image data and video data in training set data, semantic space mapping is carried out on the single modal features of the voice data, the text data, the image data and the video data.
Respectively inputting single modal characteristics of the voice data, the text data, the image data and the video data which are output after the semantic space mapping into a multi-modal characteristic fusion mechanism; the multi-modal feature fusion mechanism comprises a first-fourth feature fusion module 1041-; the single modal features of the voice data, the text data, the image data and the video data output after the semantic space mapping are respectively input into the first-fourth feature fusion module 1041-1044, the output results of the first feature fusion module 1041 and the second feature fusion module 1042 are input into the first fusion module 1045, and the output results of the first fusion module 1045, the third feature fusion module 1043 and the fourth feature fusion module 1044 are input into the second fusion module 1046.
The output result of the second fusion module 1046 is output to the result prediction module 105, the prediction result is calculated by using the Softmax function, the comparison is performed with the detection result of the mark, and the network training is completed by continuously adjusting the network function and the parameters until the prediction result is consistent with the detection result of the mark.
S103: and preprocessing real-time multi-modal data, inputting the preprocessed real-time multi-modal data into a trained power generation equipment abnormity detection network model, and outputting a result as a detection result of whether the power generation equipment is abnormal or not.
If the abnormity detection result of the power generation equipment is obtained, displaying the abnormity detection result; the display mode at least comprises: text display, voice broadcast, outbound call terminal, mail, short message prompt and intelligent sound box. As shown at 107 in fig. 2.
In order to implement the above embodiment, the present invention further provides a power generation equipment abnormality integrated recognition device for multi-modal deep learning, as shown in fig. 3, including:
the data acquisition module 310 is configured to acquire multi-modal historical data, perform data preprocessing, and use the preprocessed multi-modal historical data as a training set;
the network construction module 320 is used for constructing a power generation equipment abnormity detection network model and training the constructed power generation equipment abnormity detection network model through a training set; the power generation equipment anomaly detection network model comprises a feature extraction module, a space mapping module, a feature fusion module and a result prediction module which are connected in sequence;
and the anomaly detection module 330 is configured to input the preprocessed real-time multi-modal data into the trained power generation equipment anomaly detection network model, and output a result as a detection result of whether the power generation equipment is abnormal.
In order to implement the above embodiment, the present invention further provides another computer device, including: the power generation equipment abnormality detection system comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein when the processor executes the computer program, the abnormality detection of the power generation equipment is realized according to the embodiment of the invention.
As shown in fig. 4, the non-transitory computer readable storage medium includes a memory 810 of instructions executable by the processor 820 of the coal mining equipment walking speed estimation device to perform the method, and an interface 830. Alternatively, the storage medium may be a non-transitory computer readable storage medium, for example, the non-transitory computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.
In order to achieve the above-described embodiments, the present invention also proposes a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements power generation equipment abnormality detection as an embodiment of the present invention.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Moreover, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing steps of a custom logic function or process, and alternate implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., a sequential list of executable instructions that may be thought of as being useful for implementing logical functions, may be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that may fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: discrete logic circuits with logic gates for implementing logic functions on data signals, application specific integrated circuits with appropriate combinational logic gates, Programmable Gate Arrays (PGAs), Field Programmable Gate Arrays (FPGAs), etc.
It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware that can be related to instructions of a program, which can be stored in a computer-readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments.
In addition, functional units in the embodiments of the present invention may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.
The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc. Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.
Claims (10)
1. A multi-mode deep learning power generation equipment abnormity integrated identification method is characterized by comprising the following steps:
obtaining multi-modal historical data, preprocessing the data, and taking the preprocessed multi-modal historical data as a training set;
constructing a power generation equipment abnormity detection network model, and training the constructed power generation equipment abnormity detection network model through the training set; the power generation equipment anomaly detection network model comprises a feature extraction module, a space mapping module, a feature fusion module and a result prediction module which are connected in sequence;
and preprocessing real-time multi-modal data, inputting the preprocessed real-time multi-modal data into the trained power generation equipment abnormity detection network model, and outputting a result as a detection result of whether the power generation equipment is abnormal or not.
2. The method for recognizing abnormality integration of power generation equipment based on multimodal deep learning according to claim 1, wherein the step of acquiring multimodal historical data and preprocessing the data comprises the steps of:
multi-modal historical data acquisition: collecting automatic uploading multi-mode historical data obtained by a high-definition camera, a pickup and a sensor which are arranged around power generation equipment, and manually uploading the multi-mode historical data obtained by photographing through a mobile phone;
data cleaning: cleaning the data of the collected multi-modal historical data; wherein, the cleaning mode includes at least: data elimination and data completion;
data separation: and separating data aiming at bimodal or multimodal mixed data in the collected multimodal historical data, dividing the multimodal historical data into voice data, text data, image data and video data after the data are separated, and marking a detection result.
3. The method for integrally identifying the abnormal power generation equipment in the multi-modal deep learning mode according to claim 2, wherein the power generation equipment abnormality detection network model comprises a feature extraction module, a space mapping module, a feature fusion module and a result prediction module; wherein,
the feature extraction module is a feature extraction neural network and is used for carrying out single-mode feature extraction on voice data, text data, image data and video data obtained after data separation;
the space mapping module is used for mapping the single-mode features to the same semantic space to obtain semantic structure information in the single-mode data features;
the feature fusion module is used for performing feature fusion splicing on multi-modal historical data on the features of a single modality to obtain multi-modal feature fusion information;
and the result prediction module is used for calculating a prediction result according to the multi-modal feature fusion information to complete the abnormity detection of the power generation equipment.
4. The method for integrally identifying the abnormality of the multi-modal deep learning power generation device according to claim 3, wherein the feature extraction neural network is a BilSTM network, a convolutional neural network or a deep neural network; the BilSTM network is applied, and the context information of the single-mode data, namely the semantic information between the adjacent single-mode data, is obtained while the characteristics are extracted.
5. The method for integrally identifying abnormity of power generation equipment through multi-modal deep learning according to claim 3, wherein the characteristic fusion module comprises a bidirectional attention mechanism unit, a self-attention mechanism unit and a first full-connection layer unit; the voice data, the text data, the image data and the video data which are mapped and processed by the space mapping module are sequentially input into the bidirectional attention mechanism unit, the self-attention mechanism unit and the first full-connection layer unit, and the output data are transmitted to the fusion module for fusion.
6. The multi-modal deep learning power generation equipment abnormality integrated recognition method according to claim 4, wherein the step of training the constructed power generation equipment abnormality detection network model by the training set comprises:
inputting the preprocessed training set data into a feature extraction neural network of a feature extraction module, and respectively carrying out single-mode feature extraction on voice data, text data, image data and video data in the training set data through a BilSTM network;
after single modal feature extraction is carried out on voice data, text data, image data and video data in training set data, semantic space mapping is carried out on the single modal features of the voice data, the text data, the image data and the video data;
respectively inputting the single modal characteristics of the voice data, the text data, the image data and the video data which are output after the semantic space mapping into a set multi-modal characteristic fusion mechanism; the multi-modal feature fusion mechanism comprises a first feature fusion module, a fourth feature fusion module, a first fusion module and a second fusion module; respectively inputting the single modal characteristics of the voice data, the text data, the image data and the video data which are output after semantic space mapping into a first-fourth characteristic fusion module, inputting the output results of the first and second characteristic fusion modules into the first fusion module, and simultaneously inputting the output results of the first fusion module, the third characteristic fusion module and the fourth characteristic fusion module into the second fusion module;
and the output result of the second fusion module is output to a result prediction module, a prediction result is calculated by utilizing a Softmax function, the result is compared with a marked detection result, and network training is completed by continuously adjusting network functions and parameters until the prediction result is consistent with the marked detection result.
7. The method for integrally recognizing the abnormality of the power generation equipment through the multimodal deep learning according to claim 1, wherein if the abnormality detection result of the power generation equipment is obtained, the abnormality detection result is displayed; the display mode at least comprises: text display, voice broadcast, outbound call terminal, mail, short message prompt, intelligent sound box.
8. The utility model provides a power generation facility unusual integration recognition device of multimode deep learning which characterized in that includes:
the data acquisition module is used for acquiring multi-modal historical data, preprocessing the data and taking the preprocessed multi-modal historical data as a training set;
the network construction module is used for constructing a power generation equipment abnormity detection network model and training the constructed power generation equipment abnormity detection network model through the training set; the power generation equipment anomaly detection network model comprises a feature extraction module, a space mapping module, a feature fusion module and a result prediction module which are connected in sequence;
and the abnormality detection module is used for preprocessing the real-time multi-modal data, inputting the preprocessed real-time multi-modal data into the trained power generation equipment abnormality detection network model, and outputting a result as a detection result of whether the power generation equipment is abnormal or not.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method of any one of claims 1-7 when executing the computer program.
10. A non-transitory computer-readable storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the method of any one of claims 1-7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111400102.9A CN114463594A (en) | 2021-11-19 | 2021-11-19 | Multi-mode deep learning power generation equipment abnormity integrated identification method and equipment |
| PCT/CN2022/071290 WO2023087525A1 (en) | 2021-11-19 | 2022-01-11 | Multi-modal deep learning power generation device anomaly integrated identification method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111400102.9A CN114463594A (en) | 2021-11-19 | 2021-11-19 | Multi-mode deep learning power generation equipment abnormity integrated identification method and equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114463594A true CN114463594A (en) | 2022-05-10 |
Family
ID=81405213
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111400102.9A Pending CN114463594A (en) | 2021-11-19 | 2021-11-19 | Multi-mode deep learning power generation equipment abnormity integrated identification method and equipment |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN114463594A (en) |
| WO (1) | WO2023087525A1 (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115631829A (en) * | 2022-09-30 | 2023-01-20 | 广东亿佛手健康科技有限公司 | Network connection multi-mode detection method and system based on acupoint massage equipment |
| CN116416247A (en) * | 2023-06-08 | 2023-07-11 | 常州微亿智造科技有限公司 | Pre-training-based defect detection method and device |
| CN116881850A (en) * | 2023-09-04 | 2023-10-13 | 山东航天九通车联网有限公司 | Safety early warning system based on multi-mode data fusion |
| CN117407817A (en) * | 2023-10-25 | 2024-01-16 | 国网四川省电力公司绵阳供电公司 | Abnormality monitoring system of power distribution automation machine room |
| WO2024082453A1 (en) * | 2022-10-20 | 2024-04-25 | 国网江苏省电力有限公司南京供电分公司 | Urban energy multi-source data fusion method based on deep learning and storage medium |
| CN118152901A (en) * | 2024-05-13 | 2024-06-07 | 西北工业大学 | Equipment fault prediction method and system based on data driving |
| CN118228200A (en) * | 2024-05-24 | 2024-06-21 | 北京国网信通埃森哲信息技术有限公司 | Multi-mode model-based power equipment abnormality identification method, device and equipment |
Families Citing this family (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116342894B (en) * | 2023-05-29 | 2023-08-08 | 南昌工程学院 | GIS infrared feature recognition system and method based on improved YOLOv5 |
| CN116500335B (en) * | 2023-06-30 | 2023-10-13 | 国网山东省电力公司邹城市供电公司 | Smart power grid electricity larceny detection method and system based on one-dimensional features and two-dimensional features |
| CN116958767B (en) * | 2023-07-14 | 2024-04-30 | 江苏大学 | Intelligent contract security analysis method and system based on multi-mode technology |
| CN116668192B (en) * | 2023-07-26 | 2023-11-10 | 国网山东省电力公司信息通信公司 | Network user behavior anomaly detection method and system |
| CN116978011B (en) * | 2023-08-23 | 2024-03-15 | 广州新华学院 | Image semantic communication method and system for intelligent target recognition |
| CN117312828B (en) * | 2023-09-28 | 2024-06-14 | 光谷技术有限公司 | Public facility monitoring method and system |
| CN117152156B (en) * | 2023-10-31 | 2024-02-13 | 通号通信信息集团有限公司 | Railway anomaly detection method and system based on multi-mode data fusion |
| CN117475593B (en) * | 2023-12-26 | 2024-04-12 | 江苏濠汉信息技术有限公司 | Advanced intelligent early warning method and device for abnormal load of electricity utilization terminal |
| CN117572295B (en) * | 2024-01-12 | 2024-04-12 | 山东和兑智能科技有限公司 | Multi-mode on-line monitoring and early warning method for high-voltage sleeve |
| CN117577270B (en) * | 2024-01-15 | 2024-04-26 | 吉林大学 | Patient intelligent nutrition management method and system |
| CN117579393B (en) * | 2024-01-16 | 2024-03-22 | 国网浙江省电力有限公司 | Information terminal threat monitoring method, device, equipment and storage medium |
| CN117748500B (en) * | 2024-02-19 | 2024-04-30 | 北京智芯微电子科技有限公司 | Photovoltaic power prediction method, device, equipment and medium |
| CN117744012B (en) * | 2024-02-19 | 2024-05-17 | 北京智芯微电子科技有限公司 | Method and device for preventing abnormal electricity consumption based on fusion terminal and electronic equipment |
| CN118038172A (en) * | 2024-03-11 | 2024-05-14 | 广东石油化工学院 | Temperature controller quality detection method and system based on feature enhancement and depth network |
| CN118013444B (en) * | 2024-04-08 | 2024-06-11 | 徐州硕博电子科技有限公司 | Abnormality monitoring method and device for intelligent mine multi-autonomous cluster operation equipment |
| CN118015552A (en) * | 2024-04-09 | 2024-05-10 | 上海勘测设计研究院有限公司 | Method, system and device for identifying working state of in-station equipment based on information fusion |
| CN118035943A (en) * | 2024-04-12 | 2024-05-14 | 国网山东省电力公司淄博供电公司 | Transformer state monitoring method, system, electronic equipment and storage medium |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110574049B (en) * | 2017-05-19 | 2023-05-16 | 谷歌有限责任公司 | Multi-task multi-modal machine learning system |
| CN113057633B (en) * | 2021-03-26 | 2022-11-01 | 华南理工大学 | Multi-modal emotional stress recognition method and device, computer equipment and storage medium |
| CN113255733B (en) * | 2021-04-29 | 2023-04-07 | 西安交通大学 | Unsupervised anomaly detection method under multi-modal data loss |
| CN113255777B (en) * | 2021-05-28 | 2023-01-31 | 郑州轻工业大学 | Equipment fault early warning method and system based on multi-mode sensitive feature selection fusion |
| CN113591902B (en) * | 2021-06-11 | 2022-09-27 | 中国科学院自动化研究所 | Cross-modal understanding and generating method and device based on multi-modal pre-training model |
| CN113590396A (en) * | 2021-07-23 | 2021-11-02 | 南方电网深圳数字电网研究院有限公司 | Method and system for diagnosing defect of primary device, electronic device and storage medium |
-
2021
- 2021-11-19 CN CN202111400102.9A patent/CN114463594A/en active Pending
-
2022
- 2022-01-11 WO PCT/CN2022/071290 patent/WO2023087525A1/en unknown
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115631829A (en) * | 2022-09-30 | 2023-01-20 | 广东亿佛手健康科技有限公司 | Network connection multi-mode detection method and system based on acupoint massage equipment |
| CN115631829B (en) * | 2022-09-30 | 2023-05-30 | 广东亿佛手健康科技有限公司 | Network connection multi-mode detection method and system based on acupoint massage equipment |
| WO2024082453A1 (en) * | 2022-10-20 | 2024-04-25 | 国网江苏省电力有限公司南京供电分公司 | Urban energy multi-source data fusion method based on deep learning and storage medium |
| CN116416247A (en) * | 2023-06-08 | 2023-07-11 | 常州微亿智造科技有限公司 | Pre-training-based defect detection method and device |
| CN116881850A (en) * | 2023-09-04 | 2023-10-13 | 山东航天九通车联网有限公司 | Safety early warning system based on multi-mode data fusion |
| CN116881850B (en) * | 2023-09-04 | 2023-12-08 | 山东航天九通车联网有限公司 | Safety early warning system based on multi-mode data fusion |
| CN117407817A (en) * | 2023-10-25 | 2024-01-16 | 国网四川省电力公司绵阳供电公司 | Abnormality monitoring system of power distribution automation machine room |
| CN117407817B (en) * | 2023-10-25 | 2024-07-09 | 国网四川省电力公司绵阳供电公司 | Abnormality monitoring system of power distribution automation machine room |
| CN118152901A (en) * | 2024-05-13 | 2024-06-07 | 西北工业大学 | Equipment fault prediction method and system based on data driving |
| CN118228200A (en) * | 2024-05-24 | 2024-06-21 | 北京国网信通埃森哲信息技术有限公司 | Multi-mode model-based power equipment abnormality identification method, device and equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2023087525A1 (en) | 2023-05-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114463594A (en) | Multi-mode deep learning power generation equipment abnormity integrated identification method and equipment | |
| CN107220469B (en) | Method and system for estimating state of fan | |
| EP3767551A1 (en) | Inspection system, image recognition system, recognition system, discriminator generation system, and learning data generation device | |
| KR102239377B1 (en) | System and method for safety inspection on bridge facilities based on image and xai | |
| CN112115927B (en) | Intelligent machine room equipment identification method and system based on deep learning | |
| US20230107092A1 (en) | System and method for monitoring wind turbine rotor blades using infrared imaging and machine learning | |
| CN113205039B (en) | Power equipment fault image recognition disaster investigation system and method based on multiple DCNN networks | |
| CN111044149A (en) | Method and device for detecting temperature abnormal point of voltage transformer and readable storage medium | |
| CN113762171A (en) | Method and device for monitoring safety of railway construction site | |
| CN114419556A (en) | Abnormal drainage image identification method and system for drainage pipe network drainage port | |
| CN111710329A (en) | Deep learning-based ship engine voiceprint fault discrimination method and system | |
| CN115471487A (en) | Insulator defect detection model construction and insulator defect detection method and device | |
| CN115640159A (en) | Micro-service fault diagnosis method and system | |
| CN116882758A (en) | Dam safety monitoring physical examination list generation method | |
| CN116842379A (en) | Mechanical bearing residual service life prediction method based on DRSN-CS and BiGRU+MLP models | |
| CN114358371A (en) | Photovoltaic short-term power prediction method and device based on deep learning | |
| CN115035328A (en) | Converter image increment automatic machine learning system and establishment training method thereof | |
| CN107330440A (en) | Sea state computational methods based on image recognition | |
| CN117129815B (en) | Comprehensive detection method and system for multi-degradation insulator based on Internet of things | |
| CN116028893A (en) | Intelligent data monitoring method, system and electronic equipment | |
| CN116127324A (en) | Fault detection method and device for key components in uninterruptible power supply | |
| CN113850773A (en) | Detection method, device, equipment and computer readable storage medium | |
| CN112211793A (en) | Wind turbine generator fault automatic identification method based on video image analysis | |
| CN109584235A (en) | Quality determining method, device, server and the storage medium of plastic foam cutlery box | |
| CN117539674B (en) | Exception handling method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |