CN113297909A - Building damage real-time identification method based on unmanned aerial vehicle vision - Google Patents
Building damage real-time identification method based on unmanned aerial vehicle vision Download PDFInfo
- Publication number
- CN113297909A CN113297909A CN202110448009.9A CN202110448009A CN113297909A CN 113297909 A CN113297909 A CN 113297909A CN 202110448009 A CN202110448009 A CN 202110448009A CN 113297909 A CN113297909 A CN 113297909A
- Authority
- CN
- China
- Prior art keywords
- building damage
- image data
- aerial vehicle
- unmanned aerial
- area
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 21
- 238000012545 processing Methods 0.000 claims description 20
- 238000001514 detection method Methods 0.000 claims description 10
- 230000006870 function Effects 0.000 claims description 10
- 239000011159 matrix material Substances 0.000 claims description 9
- 238000011176 pooling Methods 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims description 5
- 238000013528 artificial neural network Methods 0.000 claims description 3
- 230000005540 biological transmission Effects 0.000 claims description 3
- 238000010586 diagram Methods 0.000 claims description 3
- 230000005284 excitation Effects 0.000 claims description 3
- 238000005070 sampling Methods 0.000 claims description 3
- 238000012216 screening Methods 0.000 claims description 3
- 238000010304 firing Methods 0.000 claims description 2
- 230000005764 inhibitory process Effects 0.000 claims description 2
- 230000001629 suppression Effects 0.000 claims description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 238000011897 real-time detection Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002401 inhibitory effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
- G06V20/176—Urban or other man-made structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/22—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a real-time building damage identification method based on unmanned aerial vehicle vision, which comprises the following steps: the unmanned aerial vehicle camera collects data and writes the data into a memory; image data plug flow; the processor reads the image data; calculating and obtaining the correlation degree of the building damage target; overlapping the damage identification frames and counting; and displaying the plug flow. The method can realize real-time identification of the building damage, and has lower time delay and higher accuracy.
Description
Technical Field
The invention relates to the field of unmanned aerial vehicle technology building application, in particular to a building damage real-time identification method based on unmanned aerial vehicle vision.
Background
With the continuous development of machine learning, the machine learning is gradually applied to various industries. The emergence of deep learning technology makes the target detection research make great progress. However, the current target detection algorithm based on deep learning cannot realize target detection in real time and under a complex background; the traditional target detection method based on segmentation and classifier is not suitable for the unmanned aerial vehicle system any more because of higher time delay, lower recognition speed and accuracy.
Disclosure of Invention
The invention aims to solve the problems and provides a building damage real-time identification method based on unmanned aerial vehicle vision, which comprises the following steps:
step (1), an unmanned aerial vehicle camera collects building damage data and writes the building damage data into a memory;
step (2), transmitting the building damage image data to a server through plug flow;
step (3), the data processing server reads the image data and processes the data;
step (4), slicing operation is carried out on original image data by a Focus structure to form a characteristic diagram;
step (5), the convolutional layer neural network carries out convolution processing on the image data; and using maximum pooling treatment;
step (6), the excitation layer uses the ReLU function to carry out nonlinear mapping on the output of the convolution layer;
and (7) classifying the extracted feature data by using a matrix-vector product by the full connection layer, wherein the matrix-vector product specifically comprises: y is Wx, wherein W is a weight vector and x is a feature vector;
step (8), screening a target frame on the detection result by using nms non-maximum inhibition;
step (9), reading the building damage image data, and performing frame-by-frame identification processing;
step (10), overlapping damage identification frames and counting; and displayed by plug flow.
Further, the pushing the building damage image data to the server in the step (2) specifically includes: and the data transmission selects a mature Nginx streaming media server to receive and transmit the transmitted building damage image data, and the plug flow side transmits the image data to the server by using an rtmp protocol.
Further, the slicing operation of the original image data by the Focus structure in the step (4) specifically includes: the four adjacent blocks in the image are changed from the plane to the function on the characteristic, so that the information is concentrated in the channel space, the input channel is expanded by 4 times, namely the spliced image is changed into 12 channels relative to the original 3-channel mode, and finally, a double-sampling characteristic image under the condition of no information loss is obtained.
Further, the convolution processing and maximum pooling selection in the step (5) specifically include:
where s (i, j) is the convolution value of the image at (i, j), W is the convolution kernel of m n, i.e., W (m, n), X is the matrix of the input, X (i, j) represents the value of matrix X at (i, j),
representing that m X n branch matrixes are respectively selected from the matrix X;
F(x，y)＝max{g(x+i，y+j)，i∈(0，n)，j∈(0，n)}
where n represents the size of the kernel, F (x, y) represents the maximum pixel value in the neighborhood n x n in the image after max-firing, and g (x + i, y + j) represents the pixel value of point (x + i, y + j).
Further, the ReLU function in step (6) specifically includes:
where a is the abscissa value and ReLU (a) is the result of the function of the non-linear mapping.
Further, the specific process of using nms non-maximum suppression to screen the target frame for the detection result in the step (8) is as follows:
wherein, Area of Overlap is the intersection Area of the two target frames, and Area of Union is the Union Area of the two target frames.
Where Ac denotes the Area of the closure region, U is Area of Union, which is the Area of the Union of two object boxes,
the weight of the closure area in the area not belonging to the two boxes is represented, and finally, the weight is subtracted from IoU to obtain the GIoU, and if the GIoU exceeds the set threshold, the GIoU is judged to be the same target box.
Has the advantages that: compared with the prior art, the technical scheme of the invention has the following beneficial technical effects:
the invention can realize mobile scanning of building damage based on unmanned aerial vehicle vision, and uses plug flow to connect the unmanned aerial vehicle and the image data processing server, thereby realizing real-time detection and identification of building damage, and having lower time delay and higher accuracy.
Drawings
FIG. 1 is a flow chart of the method of the present invention.
FIG. 2 is a flow chart of the data processing of the present invention.
Detailed Description
The invention will be further explained with reference to the drawings.
The invention provides a building damage real-time identification method based on unmanned aerial vehicle vision, which is shown in a flow chart of an attached figure 1 and comprises the following specific processes:
the unmanned aerial vehicle camera gathers the building damage data, writes into the memory.
And transmitting the building damage image data to a data processing server through plug flow, wherein the data transmission selects a mature Nginx streaming media server to receive the transmitted building damage image data, and the plug flow side transmits the image data to the server by using an rtmp protocol.
The data processing server reads the image data and performs data processing, as shown in the flow diagram of fig. 2, the specific flow of data processing is as follows:
and (3) slicing treatment: the Focus structure performs slicing operation on the original image data to form a feature map: focus is arranged in a way that four adjacent blocks in an image are changed from a plane to a function on a feature, each image obtains one value in the four adjacent blocks in the original image, and the four images are complementary but have no information loss. Therefore, information is concentrated in a channel space, an input channel is expanded by 4 times, one grid position represents the characteristics of four grids of an original image, namely, a spliced image is changed into 12 channels relative to the original 3-channel mode, and finally a double-sampling feature map under the condition of no information loss is obtained.
Convolution processing: the convolutional layer neural network performs convolution processing on the image data:
where s (i, j) is the convolution value of the image at (i, j), W is the convolution kernel of m n, i.e., W (m, n), X is the matrix of the input, X (i, j) represents the value of matrix X at (i, j),
the branch matrices of m × n are respectively selected from the matrix X.
Performing pooling treatment: selecting the maximum value in the image region as the pooled value for that region using maximum pooling: and performing dimensionality reduction on the data by a max-posing mode so as to simplify calculation. The concrete mode is as follows: let n be the size of the kernel and F (x, y) denote the maximum pixel value in the neighborhood n x n in the image (x, y) as the result after max-posing. g (x + i, y + j) represents a pixel value of the point (x + i, y + j).
F(x，y)＝max{g(x+i，y+j)，i∈(0，n)，j∈(0，n)}
Excitation: the output of the convolutional layer is mapped non-linearly using the ReLU function: assuming x is the abscissa value and relu (x) is the functional result of the non-linear mapping, then:
and (3) feature extraction and classification: the full-connection layer classifies the extracted feature data by using a matrix-vector product, wherein the matrix-vector product specifically comprises: and y is Wx, wherein W is a weight vector and x is a feature vector.
Through the above process, the damage data is subjected to feature extraction, after target detection and classification are carried out, the situation that the same target is repeatedly detected may occur, in order to avoid the situation, the non-maximum value of nms is used for inhibiting the screening of a target frame on a detection result, and the specific process is as follows:
wherein Area of Overlap is the intersection Area of the two target frames, and Area of Union is the Union Area of the two target frames.
Wherein
The weight of the closure area in the area not belonging to the two boxes is represented, and finally, the weight is subtracted from IoU to obtain the GIoU, and if the GIoU exceeds the set threshold, the GIoU is judged to be the same target box.
And reading the building damage image data, and identifying and processing the building damage image data frame by frame. And finally, overlapping the damage identification frame and displaying through plug flow.
And (4) selecting an rtmp protocol provided by Nginx for plug flow display to transmit the image data of the completed overlapping damage identification frame to the video terminal. By configuring the parameters, the video terminal can receive image data via rtmp:// ip: 1935/hls/check code address.
The invention can realize mobile scanning of building damage based on unmanned aerial vehicle vision, and uses plug flow to connect the unmanned aerial vehicle and the image data processing server, thereby realizing real-time detection and identification of building damage, and having lower time delay and higher accuracy.
The technical means disclosed in the invention scheme are not limited to the technical means disclosed in the above embodiments, but also include the technical scheme formed by any combination of the above technical features.
Claims (6)
1. A building damage real-time identification method based on unmanned aerial vehicle vision is characterized by comprising the following steps:
step (1), an unmanned aerial vehicle camera collects building damage data and writes the building damage data into a memory;
step (2), transmitting the building damage image data to a data processing server through plug flow;
step (3), the data processing server reads the image data and processes the data;
step (4), slicing operation is carried out on original image data by a Focus structure to form a characteristic diagram;
step (5), the convolutional layer neural network carries out convolution processing on the image data; and using maximum pooling treatment;
step (6), the excitation layer uses the ReLU function to carry out nonlinear mapping on the output of the convolution layer;
and (7) classifying the extracted feature data by using a matrix-vector product by the full connection layer, wherein the matrix-vector product specifically comprises: y is Wx, wherein W is a weight vector and x is a feature vector;
step (8), screening a target frame on the detection result by using nms non-maximum inhibition;
step (9), reading the building damage image data, and performing frame-by-frame identification processing;
and (10) overlapping the damage identification frames, counting and displaying through plug flow.
2. The real-time building damage identification method based on unmanned aerial vehicle vision according to claim 1, characterized in that: the step (2) of pushing the building damage image data to the data processing server specifically includes: and the data transmission selects a mature Nginx streaming media server to receive the sent building damage image data, and the plug flow transmits the image data to the server by using an rtmp protocol.
3. The real-time building damage identification method based on unmanned aerial vehicle vision according to claim 2, characterized in that: the slicing operation of the original image data by the Focus structure in the step (4) specifically includes: the four adjacent blocks in the image are changed from the plane into the function on the characteristic, so that the information is concentrated into the channel space, the input channel is expanded by 4 times, namely the spliced image is changed into 12 channels relative to the original 3-channel mode, and finally, a double-sampling characteristic image without information loss is obtained.
4. The real-time building damage identification method based on unmanned aerial vehicle vision according to claim 3, characterized in that: the convolution processing and maximum pooling selection in the step (5) specifically comprises:
where s (i, j) is the convolution value of the image at (i, j), W is the convolution kernel of m n, i.e., W (m, n), X is the matrix of the input, X (i, j) represents the value of matrix X at (i, j),
representing that m X n branch matrixes are respectively selected from the matrix X;
F(x，y)＝max{g(x+i，y+j)，i∈(0，n)，j∈(0，n)}
where n denotes the size of the kernel, F (x, y) denotes the maximum pixel value in the neighborhood n x n in the image after max-firing, and g (x + i, y + j) denotes the pixel value of point (x + i, y + j).
5. The method for identifying building damage in real time based on unmanned aerial vehicle vision as claimed in claim 4, wherein the ReLU function in step (6) specifically comprises:
where a is the abscissa value and ReLU (a) is the result of the function of the non-linear mapping.
6. The method for identifying the building damage based on the unmanned aerial vehicle vision in real time as claimed in claim 5, wherein the specific process of using nms non-maximum suppression to screen the detection result for the target frame in the step (8) is as follows:
wherein, Area of Overlap is the intersection Area of the two target frames, and Area of Union is the Union Area of the two target frames;
where Ac denotes the Area of the closure region, U is Area of Union, which is the Area of the Union of two object boxes,
the proportion of the area not belonging to the two boxes in the closure area to the closure area is represented, and the proportion is subtracted from IoU to obtain GIoU, and if the GIoU exceeds a set threshold, the same target box is determined.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110448009.9A CN113297909B (en) | 2021-04-25 | 2021-04-25 | Building damage real-time identification method based on unmanned aerial vehicle vision |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110448009.9A CN113297909B (en) | 2021-04-25 | 2021-04-25 | Building damage real-time identification method based on unmanned aerial vehicle vision |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113297909A true CN113297909A (en) | 2021-08-24 |
| CN113297909B CN113297909B (en) | 2024-05-07 |
Family
ID=77320150
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110448009.9A Active CN113297909B (en) | 2021-04-25 | 2021-04-25 | Building damage real-time identification method based on unmanned aerial vehicle vision |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113297909B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110472272A (en) * | 2019-07-01 | 2019-11-19 | 广东工业大学 | A kind of Structural Damage Identification based on multi-parameter and convolutional neural networks |
| CN110580443A (en) * | 2019-06-19 | 2019-12-17 | 深圳大学 | Low-altitude near-real-time building earthquake damage assessment method |
| KR102157610B1 (en) * | 2019-10-29 | 2020-09-18 | 세종대학교산학협력단 | System and method for automatically detecting structural damage by generating super resolution digital images |
| CN112258495A (en) * | 2020-11-02 | 2021-01-22 | 郑州大学 | Building wood crack identification method based on convolutional neural network |
-
2021
- 2021-04-25 CN CN202110448009.9A patent/CN113297909B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110580443A (en) * | 2019-06-19 | 2019-12-17 | 深圳大学 | Low-altitude near-real-time building earthquake damage assessment method |
| CN110472272A (en) * | 2019-07-01 | 2019-11-19 | 广东工业大学 | A kind of Structural Damage Identification based on multi-parameter and convolutional neural networks |
| KR102157610B1 (en) * | 2019-10-29 | 2020-09-18 | 세종대학교산학협력단 | System and method for automatically detecting structural damage by generating super resolution digital images |
| CN112258495A (en) * | 2020-11-02 | 2021-01-22 | 郑州大学 | Building wood crack identification method based on convolutional neural network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113297909B (en) | 2024-05-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112379231B (en) | Equipment detection method and device based on multispectral image | |
| CN109583483B (en) | Target detection method and system based on convolutional neural network | |
| CN112381784A (en) | Equipment detecting system based on multispectral image | |
| CN113936256A (en) | Image target detection method, device, equipment and storage medium | |
| CN108304754A (en) | The recognition methods of vehicle and device | |
| CN107273870A (en) | The pedestrian position detection method of integrating context information under a kind of monitoring scene | |
| CN114399734A (en) | Forest fire early warning method based on visual information | |
| CN113538457B (en) | Video semantic segmentation method utilizing multi-frequency dynamic hole convolution | |
| CN111582074A (en) | Monitoring video leaf occlusion detection method based on scene depth information perception | |
| CN111768415A (en) | Image instance segmentation method without quantization pooling | |
| CN116186770A (en) | Image desensitizing method, device, electronic equipment and storage medium | |
| CN115205247A (en) | Method, device and equipment for detecting defects of battery pole piece and storage medium | |
| CN110909724A (en) | Multi-target image thumbnail generation method | |
| CN111079864A (en) | Short video classification method and system based on optimized video key frame extraction | |
| CN108345835B (en) | Target identification method based on compound eye imitation perception | |
| CN111815528A (en) | Bad weather image classification enhancement method based on convolution model and feature fusion | |
| CN111369548A (en) | No-reference video quality evaluation method and device based on generation countermeasure network | |
| CN110111269A (en) | Low-light-level imaging algorithm and device based on multiple dimensioned context converging network | |
| CN110647813A (en) | Human face real-time detection and identification method based on unmanned aerial vehicle aerial photography | |
| CN111091122A (en) | Training and detecting method and device for multi-scale feature convolutional neural network | |
| WO2022205329A1 (en) | Object detection method, object detection apparatus, and object detection system | |
| CN112396637A (en) | Dynamic behavior identification method and system based on 3D neural network | |
| CN113297909B (en) | Building damage real-time identification method based on unmanned aerial vehicle vision | |
| CN113920455B (en) | Night video coloring method based on deep neural network | |
| CN116403200A (en) | License plate real-time identification system based on hardware acceleration |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |