CN112800524A - Pavement disease three-dimensional reconstruction method based on deep learning - Google Patents
Pavement disease three-dimensional reconstruction method based on deep learning Download PDFInfo
- Publication number
- CN112800524A CN112800524A CN202110159075.4A CN202110159075A CN112800524A CN 112800524 A CN112800524 A CN 112800524A CN 202110159075 A CN202110159075 A CN 202110159075A CN 112800524 A CN112800524 A CN 112800524A
- Authority
- CN
- China
- Prior art keywords
- disease
- image
- dimensional model
- dimensional
- pavement
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/10—Geometric CAD
- G06F30/13—Architectural design, e.g. computer-aided architectural design [CAAD] related to design of buildings, bridges, landscapes, production plants or roads
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
- G06F18/232—Non-hierarchical techniques
- G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
- G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/20—Design optimisation, verification or simulation
- G06F30/27—Design optimisation, verification or simulation using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computer Hardware Design (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- Mathematical Optimization (AREA)
- Architecture (AREA)
- Probability & Statistics with Applications (AREA)
- Structural Engineering (AREA)
- Computational Mathematics (AREA)
- Mathematical Analysis (AREA)
- Civil Engineering (AREA)
- Pure & Applied Mathematics (AREA)
- Medical Informatics (AREA)
- Computer Graphics (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
- Processing Or Creating Images (AREA)
Abstract
The invention relates to a pavement disease three-dimensional reconstruction method based on deep learning, which uses an unmanned aerial vehicle to carry out oblique photography on a pavement to obtain a plurality of images in different directions, constructs a standardized pavement disease database by a large number of pavement disease images which are acquired in different types and at different angles, realizes the three-dimensional reconstruction of pavement diseases by the deep learning method, and carries out morphological characteristic extraction on the reconstructed diseases. And the three-dimensional model of the disease is obtained through continuous contrast and correction with the CAD model, and the morphological characteristics of the disease can be accurately extracted, and the precision can reach 1.4 mm.
Description
Technical Field
The invention relates to the field of artificial intelligence and pavement disease three-dimensional reconstruction, in particular to a pavement disease three-dimensional reconstruction method based on deep learning.
Background
By the end of 2019, the total highway mileage of China is 501.3 kilometers and is the first place in the world. After the road is built and put into use, a lot of road surface diseases such as cracks, ruts, subsidence, pits, wave congestion, looseness and the like can appear after a period of time under the influence of factors such as vehicle load, climate and the like, and the normal service life of the road can be influenced if the road is not treated in time. Therefore, the three-dimensional reproduction of the pavement diseases is particularly important for pavement maintenance and repair decisions.
At present, the method applied to three-dimensional reconstruction of the road surface mainly depends on detecting vehicles to carry laser, electromagnetic waves, sound waves, cameras and other equipment to acquire information of the road surface. And carrying out three-dimensional reconstruction on the acquired information. A commonly used three-dimensional reconstruction method is a stereoscopic vision method, and mainly includes three ways of directly acquiring range information by using a range finder, inferring three-dimensional information by using one image, and recovering three-dimensional information by using two or more images from different viewpoints. The position deviation between corresponding points of the image is obtained based on the parallax principle by simulating a human visual system, and the three-dimensional information is recovered.
The normal use of the road can be influenced by the collection of the detection vehicle, and the quality of the road information collection can be influenced by different road conditions and vehicle conditions. Meanwhile, the flexibility of the detection vehicle acquisition is insufficient, the detection vehicle is easily limited by traffic control and closed traffic of a highway, only one lane can be acquired at a time, and the acquisition efficiency is low. And the stereoscopic vision method needs to assume that the plane of the space is a positive plane, which is far from the actual situation. In addition to this, matching is ambiguous: for some feature points on one image, there may be several similar feature points in another image. Determination of camera motion parameters and the like are also required.
The scientific, accurate and efficient three-dimensional reconstruction of the pavement diseases needs to be realized: 1. the method comprises the steps of breaking through ground constraint in pavement information acquisition, being more flexible and efficient 2, establishing a pavement disease information database 3 and using a more accurate three-dimensional reconstruction method.
Disclosure of Invention
Aiming at the defects of the prior art, the invention aims to solve the technical problems of avoiding traffic jam and traffic break on the road when road surface disease information is collected, building a road surface disease database, and simultaneously performing three-dimensional reconstruction on road surface diseases by using deep learning and extracting the disease characteristics.
The invention solves the technical problem by adopting the technical scheme that a pavement disease three-dimensional reconstruction method based on deep learning is characterized by comprising the following steps:
constructing a standardized pavement disease image database: the method comprises the steps of obtaining multi-path surface images containing diseases at different shooting angles, segmenting the multi-path surface images containing the diseases at different shooting angles, extracting areas with the road surface diseases to form road surface disease images at different angles, classifying all the road surface disease images by using a K-means clustering algorithm, adding labels to each classified disease image, storing the labels, recording the disease types and the corresponding shooting angles, and obtaining a standardized road surface disease image database.
Constructing a CAD model: measuring each type of pavement diseases on site, collecting size data of the diseases, and modeling by using CAD to obtain CAD models of different types of pavement diseases;
modeling a neural network: training and modeling classified disease images of different shooting angles in a standardized pavement disease image database, predicting three-dimensional coordinates of pixels by using a 3D structure generator, performing coordinate operation by combining image 3D postures of different shooting angles to realize point cloud fusion, and generating a predicted three-dimensional model; randomly generating projection depth maps of k different viewpoints of the predicted three-dimensional model by using a pseudo renderer;
generating a CAD model projection depth map with the same number as the predicted three-dimensional model viewpoints by utilizing the CAD models of the corresponding road surface diseases; error calculation is carried out on the predicted three-dimensional model projection depth map and the CAD model projection depth map, errors are evenly distributed to k different viewpoints through 3D geometric reasoning, the predicted three-dimensional model is optimized, and different types of disease three-dimensional models are obtained;
parameter extraction: adding an external cuboid to the disease three-dimensional model, and extracting morphological parameters of the disease three-dimensional model; and then obtaining real disease form parameters according to the size conversion relation between the shot image and the actual disease.
Compared with the prior art, the reconstruction method of the invention uses an unmanned aerial vehicle to carry out oblique photography on the road surface to obtain a plurality of images in different directions, constructs a standardized road surface disease database by a large number of road surface disease images which are acquired in different types and different angles, realizes three-dimensional reconstruction of the road surface diseases by a deep learning method, and carries out morphological characteristic extraction on the reconstructed diseases. The concrete beneficial effects are that:
1. the method uses a high-efficiency, quick, simple-calculation and accurate-result neural network to realize the prediction of a three-dimensional model by taking 2D convolution operation as a core, takes an unmanned aerial vehicle aerial photography technology as the basis of image acquisition, acquires road surface information from 5 angles, realizes classification and labeling by using an unsupervised learning K-means algorithm, and constructs a brand-new standardized road surface disease database for neural network training. And the three-dimensional model of the disease is obtained through continuous contrast and correction with the CAD model, and the morphological characteristics of the disease can be accurately extracted, and the precision can reach 1.4 mm.
2. According to the method, a 3D structure generator is used for constructing the 2D road surface disease image into a prediction three-dimensional model, then a pseudo renderer and a loss function are used for setting to obtain a disease three-dimensional model, and the method is different from other three-dimensional construction modes for one-to-one construction of a certain object. The 3D structure generator uses 2D convolution operation to predict the 3D model from the multi-angle image, the calculation amount is small, the calculation speed is high, and the shape similarity (prediction density) of the generated model is high. The method solves the problems of large calculation amount, low calculation speed and long time consumption caused by using 3D convolution operation to predict the model in the prior art
3. The invention uses the oblique photography technology of the unmanned aerial vehicle to collect road surface images and related data in the air, thereby avoiding the influence on normal road traffic in the prior detection process, and meanwhile, for the method for detecting the road detection vehicle, the working cost of the unmanned aerial vehicle is far less than that of the road detection vehicle, and the equipment cost is far less than that of the road detection vehicle. Meanwhile, the unmanned aerial vehicle is low in operation threshold and not high in technical requirement, and a large amount of personnel training cost can be saved. In addition, unmanned aerial vehicle oblique photography detection efficiency is high, and required check-out time is short and the operation process is difficult for receiving the topography influence. And unmanned aerial vehicle equipment is light, portable. The unmanned aerial vehicle information acquisition is more flexible, and slices of the whole highway can be acquired; the unmanned aerial vehicle is more efficient in acquisition, the maximum acquisition speed can reach 58km/h, the complete section of the road can be acquired by one-time flight, and all lanes are covered. The unmanned aerial vehicle image acquisition can acquire targets at any angle in the air, the invention acquires road surface disease images from different angles by setting the inclination angle of a camera carried by the unmanned aerial vehicle, so that the data of a database can be conveniently acquired, and the size of the target disease in a photo can be converted out directly according to the parameter of the ground sampling distance of the unmanned aerial vehicle, so that the calculation is simple and convenient.
4. The invention firstly combines images of different diseases collected from different angles into a standardized and unified disease database, wherein the standardized and unified method refers to the following steps: the size of the image is consistent, the shooting angle is consistent, the image database containing the consistent target objects is contained, and the training of deep learning is facilitated for the two-dimensional pavement disease image database specially used for neural network training. And a solid data foundation is laid for three-dimensional reconstruction of the pavement diseases by a stereoscopic vision method.
5. The method utilizes disease images of different angles to predict a three-dimensional model, combines geometric reasoning and 2D projection optimization, and has shape similarity epsiloniAnd the like, are superior to the existing three-dimensional modeling method. The unmanned aerial vehicle is adopted to collect images, camera motion parameters are not required to be determined, and prediction precision can be improved through iteration of a neural network.
Drawings
FIG. 1 is a flow chart of a reconstruction method of the present invention;
FIG. 2 is a schematic diagram of a neural network modeling process in accordance with the present invention;
FIG. 3 is a schematic view of the vertical shooting of the unmanned aerial vehicle of the present invention;
fig. 4 is a schematic diagram of unmanned aerial vehicle oblique shooting according to the invention.
Detailed Description
Specific examples of the present invention are given below. The specific examples are only for illustrating the present invention in further detail and do not limit the scope of protection of the present application.
The invention provides a 4RTK unmanned aerial vehicle using Xinjiang spirit, which carries the following camera parameters: lens FOV84 °; 8.8mm/24mm (35mm format equivalent); the aperture is f/2.8-f/11; with auto-focus (focus distance 1m- ∞), ISO range video: 100-; image resolution was 4864 × 3648(4: 3); 5472X 3648(3:2), and the effective picture width of the photo is 5472X 3648. The maximum flight speed is 58KM/h, DNSS positioning accuracy: the multi-frequency multi-system high-precision RTK GNSS is vertical to 1.5cm +1ppm (RMS); level 1cm +1ppm (RMS). This model unmanned aerial vehicle connected network can use carrier phase difference branch technique (RTK), does not need to measure phase control point, easy operation. The map construction precision meets the internal standard of the aerial photogrammetry of GB/T7930-. The Ground Sampling Distance (GSD) calculation formula is:
GSD＝(H/36.5)cm/pixel
where H is the unmanned aerial vehicle flight height in meters, and the Ground Sampling Distance (GSD) represents the actual ground size represented by a pixel in centimeters per pixel. P4R was used without a remote control and ipad. At a flight height of 5 meters, the Ground Sampling Distance (GSD) is 0.14 cm/pixel. Therefore, the coordinate of the central point of the positive direction area is taken to represent the coordinate value of each pixel point corresponding to the square area with the actual area size of 0.14cm x 0.14cm of each pixel point.
The method comprises the steps of manufacturing a KML file by using the new earth 4 software, drawing a road surface range to be detected by using a drawing surface tool in a toolbar, storing the drawn file as the KML file, storing the KML file into an SD card of the unmanned aerial vehicle according to a format, and detecting the KML file before the unmanned aerial vehicle runs.
The flying height of the unmanned aerial vehicle is set to be 5 meters, the heading overlapping rate is 80, and the side overlapping rate is 80. The method comprises the steps of using a five-way mode of the unmanned aerial vehicle to collect road surface images, wherein the five-way mode is to shoot the road surface from five different angles, and comprises an orthographic flight path, a tripod head inclination angle of-90 degrees, 4 inclined flight paths and a tripod head inclination angle of-45 degrees. The acquired image is based on the World Geodetic System 1984 and on the WGS 84 coordinate System.
The method comprises the steps of carrying out image preprocessing on a pavement image acquired by an image, segmenting and extracting an area containing pavement diseases, and dividing the pavement disease area into five types of cracks, pits, tracks, wave bumps and ruts by using a K-means clustering algorithm. The value of K is 5. And outputting the classified diseases to the same folder, and adding a label for storage to form a standardized pavement disease database. In order to calculate the loss of neural network training, a CAD model (computer aided design model) is established for different types of road surface diseases, the CAD model is a uniformly encrypted set of 3D points, and data used by the CAD model is used for measuring the actual size of the obtained current disease in the field.
The types of the road surface diseases are not limited to the five types, the five types can be regarded as large types, each large type can comprise more subdivided small types, the types of the data in the database are richer and more diverse, and the database can be constructed by considering that a large number of disease images in a large number of road sections are extracted.
The neural network modeling is mainly divided into two parts: the first part is a structure generator for predicting three-dimensional coordinates of pixel points and generating a predicted three-dimensional model through point cloud fusion. And the second part is an optimizer for continuously optimizing the correction model, and a disease three-dimensional model is finally obtained through model prediction and model optimization.
(1) The structure generator is used for predicting the 3D structure of the three-dimensional object at N different viewpoints. The structure generator predicts (x, y, z) images representing 3D surface geometry mainly based on 2D convolution operations. This approach avoids time consuming and memory consuming 3D convolution operations in volume prediction. The 3D coordinates of each pixel position areXi＝[xi,yi,zi]TThe predicted coordinates of any 3D point i in the nth view point may be converted to standard 3D coordinates PiThe conversion relationship is as follows:
the transformation relationship also defines the relationship between the predicted 3D points and the fused set of point clouds in the standard 3D coordinates (the standard 3D coordinate system is the cartesian coordinate system) for point fusion of the predicted coordinates to generate the model point cloud.
Where K is the matrix of the cameras,
wherein f: focal length, in millimeters; dx: width of pixel in X direction, unit millimeter; dY: pixel Y-direction width; (u0, v0) is the coordinates of the origin of the image coordinate system in the pixel coordinate system, and the unit is also the actual position of the pixel, i.e., the principal point (which is the origin of the image coordinate system).
(RN,tN) For rigid transformation matrices at N viewpoints, RNIs a 3 × 3 orthogonal identity matrix (also called rotation matrix), tNIs a translation vector in three dimensions. All transformation matrices have the same dimension, RNIs 3 x 3, but the rotation matrix values at different viewpoints are not equal. The rotation matrix and the translation vector are only related to the camera extrinsic parameters, and the extrinsic parameters change with the change of the rigid body position. N is 1,2, …, N.
The 3D structure generator is a depth generation model and is composed of linear layers and 2D convolutional layers, the 2D convolutional layers comprise multiple layers of convolution, the convolution kernel size of the 2D convolutional layers in the embodiment is 3 x 3, 4 layers of convolution are provided in total, the 4 layers of convolution are sequentially connected in series, the length and the width of a feature diagram are reduced by half after each convolution operation, dimension reduction and feature extraction are achieved, and the linear layers are subjected to point cloud fusion to generate a prediction three-dimensional model. The activation function is a ReLU function. The learning rate was 0.01.
(2) The optimizer optimizes the model. And randomly selecting k new viewpoints for projection on the predicted three-dimensional model generated by the structure generator to generate a projection depth map. A depth map is an image in which the distance (depth) from an image collector to each point in a scene is taken as a pixel value. In order to avoid collision of points in the space in the projection process, a pseudo renderer is used for carrying out pseudo rendering operation on the target image. The pseudo-renderer is a differentiable module to approximate the real rendering, so that dense point clouds can be synthesized into a new depth image. Pseudo-rendering achieves performance closer to real rendering by a higher value of the upsampling factor U. In the invention, U is 50.
Deriving a 2D projection depth map (pseudo-rendered depth map) of new views (randomly selected k views) from a predicted three-dimensional model projection
) Standard 3D coordinate conversion to image coordinates is required. At the kth new viewpoint, the standard 3D coordinates PiConversion to image coordinates
The conversion relationship is as follows:
where K is the camera matrix, (R)k，tk) A three-dimensional rigid transformation matrix for k viewpoints.
For the projected positions of the three-dimensional model points in the 2D projected depth map,
are pixel values of the projected depth image.
CAD modeling is carried out on different types of pavement diseases to obtain a uniformly encrypted set of 3D points, CAD models of different types of pavement diseases are obtained, then CAD models of corresponding types of pavement diseases are utilized to generate CAD model projection depth maps with the same number and same number of three-dimensional model viewpoints, and the CAD model projection depth maps are marked as CAD depth map Zk。
And extracting the CAD model projection depth maps of k different visual angles, wherein the visual angle is consistent with the angle of the predicted three-dimensional model projection depth map, naming and storing, namely the angles and the number of the CAD model projection depth maps are consistent with those of the predicted 3D model projection depth maps.
The loss function L used for model optimization is:
wherein
Pseudo-rendered depth map for the t-th point of all k new viewpoints, i.e. predicted three-dimensional model projection depth map, ZtA depth map is projected for the CAD model.
Quantization index of the model: measured using the average point-by-point 3D Euclidean distance between two 3D models, for predicting the point P in the three-dimensional modeliAnd points P in CAD model SjAnd (3) calculating the distance: piRefers to predicting a point, P, in a three-dimensional modeljRefers to points in the CAD model;
error epsiloniThe index represents the measurement of 3D shape similarity and also represents the surface coverage, i.e. the predicted density;
3D geometric reasoning is carried out on the k new viewpoints, the distances of all the points are minimized, namely the fitting degree of the two models is maximum, namely an error epsiloniAnd uniformly distributing the three-dimensional prediction model on new k viewpoints, and optimizing the three-dimensional prediction model to obtain the three-dimensional pavement disease model constructed by the neural network finally.
And adding an external cuboid to the obtained disease three-dimensional model, and extracting morphological characteristics of the pavement diseases. Including maximum length, maximum width, maximum height difference, volume, and orthographic projection area. The model resolution was 1.4 mm/pixel.
The working principle and the working process of the invention are as follows:
first, the working principle
The unmanned aerial vehicle collects road surface information from five different angles in the air, carries out image preprocessing on collected road surface images, carries out segmentation and extraction on the road surface images, uses a K-means clustering algorithm to classify and store disease images, and constructs a standardized road surface disease database. And performing CAD modeling aiming at different types of road surface diseases, extracting k 2D projection depth maps with different angles of the model, and calculating training loss by using the k 2D projection depth maps and the projection depth maps of the predicted three-dimensional model obtained by neural network training, wherein the projection angles and the number of the predicted three-dimensional model are consistent with those of the CAD model. And predicting the three-dimensional coordinates of pixel points of each image by using a 3D structure generator according to the 5 images at different angles acquired by the unmanned aerial vehicle and a traditional 2D convolution method. The 3D posture of each image is known in advance, and a prediction three-dimensional model is generated through coordinate calculation point cloud fusion. And (3) improving the image resolution of the generated prediction three-dimensional model by using a pseudo renderer, and randomly generating k 2D projection depth maps with different angles. The same number of projection depth maps from the same angle as the CAD model generated calculate the neural network model training loss. And (4) averaging errors to each new viewpoint through 3D geometric reasoning to continuously optimize the 3D model. The flying height of the unmanned aerial vehicle is 5 meters, the ground sampling distance is 1.4 mm/pixel, and diseases with the depth, the length, the width and the height of 1.4mm can be identified. And adding an external cuboid to the disease three-dimensional model according to the conversion ratio (the conversion ratio of the range of 1.4mm x 1.4mm in size of each pixel point), and extracting parameters such as the maximum length, the maximum height difference, the maximum width, the orthographic projection area and the volume of the disease.
Second, the working process
1. Data acquisition: the 4RTK unmanned aerial vehicle of Xinjiang spirit is used, and road pavement image data are collected from 1 orthographic flight path and 4 inclined flight paths respectively in a five-way flight mode at the flight height of 5 m. Wherein the angle of the holder of the orthographic flight path is-90 degrees, and the angle of the inclined flight path is-45 degrees. Heading overlap ratio 80, side-to-side overlap ratio 80. And storing the collected pictures in different directions and angles.
2. Establishing a database: the method comprises the steps of segmenting pavement images with different angles obtained by aerial photography, extracting areas with pavement diseases, classifying the pavement disease images by using a K-means clustering algorithm, and adding labels to each classified class of disease images and storing the classified class of disease images.
3. Modeling a neural network: training and modeling are carried out on classified disease images with different visual angles, three-dimensional coordinates of pixels are predicted by a 3D structure generator, coordinate operation is carried out by combining 3D postures of the images with different angles to realize point cloud fusion, and a predicted three-dimensional model is generated. And the pseudo renderer randomly generates projection depth maps of the three-dimensional models of k different viewpoints for the predicted three-dimensional model.
The method includes the steps of performing CAD modeling on road surface defects (data used for CAD modeling is to measure the defects existing in the road surface in the field, collect relevant size data and perform CAD modeling), and generating projection depth maps of the same view points of k view point projection depth maps generated randomly by a prediction three-dimensional model. The error calculation is performed by predicting the projected depth map of the three-dimensional model (the pseudo renderer is part of the neural network, and its role is to give the projected depth map of the predicted 3D model at different viewing angles) from the projected depth map of the CAD model. And (4) uniformly distributing the errors to k different viewpoints through 3D geometric reasoning, and optimizing the model. And (3) enabling the model obtained by neural network training to be closer to a real model and recording as a disease three-dimensional model.
4. Extracting morphological parameters of the disease model: adding an external cuboid to a disease three-dimensional model obtained by neural network training and optimization, extracting form parameters (maximum length, maximum width, maximum height difference, volume and orthographic projection area) of the three-dimensional model, wherein the maximum length, maximum width, maximum height difference and volume are obtained from the length, width and the like of the external cuboid, and the orthographic projection area is obtained by orthographically projecting the final disease three-dimensional model, and calculating according to the ground sampling distance of 1.4 mm/pixel to obtain the real disease form parameters.
5. In practical use, the road surface diseases to be reconstructed are not required to be measured on site, unmanned aerial vehicle shooting at different angles is directly carried out on the road surface diseases to be reconstructed, the different angles are consistent with the angles when the database is built, at least all shooting angles in the database are included, the shooting angles cannot be less than all shooting angles in the database, and the shooting angles can be more than all shooting angles in the database. Inputting the current road surface disease images of different angles to be reconstructed into the disease three-dimensional model after training and optimization, outputting the disease three-dimensional model corresponding to the road surface disease images, and outputting the actual morphological parameters of the current road surface disease after morphological parameter extraction to obtain the real size of the disease. Here unmanned aerial vehicle will take notes unmanned aerial vehicle ground sampling distance under the current situation when shooting to carry out the distance conversion. The ground sampling distance when the database is constructed can be the same as or different from the ground sampling distance when the database is actually used, and is set according to the actual situation.
Nothing in this specification is said to apply to the prior art.
Claims (7)
1. A pavement disease three-dimensional reconstruction method based on deep learning is characterized by comprising the following steps:
constructing a database: constructing a standardized pavement disease database by using pavement disease images acquired at different shooting angles;
constructing a CAD model: measuring each type of pavement diseases on site, collecting size data of the diseases, and modeling by using CAD to obtain CAD models of different types of pavement diseases;
modeling a neural network: training and modeling classified disease images of different shooting angles in a standard pavement disease image database, predicting three-dimensional coordinates of pixels by using a structure generator, performing coordinate operation by combining image 3D postures of different shooting angles to realize point cloud fusion, and generating a predicted three-dimensional model; randomly generating projection depth maps of k different viewpoints of the predicted three-dimensional model by using a pseudo renderer;
generating a CAD model projection depth map with the same number as the predicted three-dimensional model viewpoints by utilizing the CAD models of the corresponding road surface diseases; error calculation is carried out on the predicted three-dimensional model projection depth map and the CAD model projection depth map, errors are evenly distributed to k different viewpoints through 3D geometric reasoning, the predicted three-dimensional model is optimized, and different types of disease three-dimensional models are obtained;
parameter extraction: adding an external cuboid to the disease three-dimensional model, and extracting morphological parameters of the disease three-dimensional model; and then obtaining real disease form parameters according to the size conversion relation between the shot image and the actual disease.
2. The reconstruction method according to claim 1, wherein the road surface image is shot by adopting an unmanned aerial vehicle aerial photography technology, and the road surface information is acquired from 5 angles: acquiring road pavement image data from 1 orthographic flight path and 4 inclined flight paths respectively at the flying height of 5m, wherein the angle of a tripod head of the orthographic flight path is-90 degrees, the inclination angle of the inclined flight path is-45 degrees, the course overlapping rate is 80, the lateral overlapping rate is 80, and the acquired images are stored in different directions and different angles; recording the size conversion relation between the shot image and the actual disease, namely, recording the size of the actual area of each pixel point to be a square area of 0.14cm by 0.14cm, and taking the coordinate of the center point of the square area to represent the coordinate value of the pixel point.
3. The reconstruction method according to claim 1, wherein the structure generator is configured to predict the 3D structure of the three-dimensional object at N different viewpoints, the structure generator predicting the (x, y, z) image representing the 3D surface geometry based on a 2D convolution operation; the 3D coordinate of each pixel position is Xi＝[xi,yi,zi]TThe predicted coordinates of any 3D point i in the nth view can be converted to standard 3D coordinates PiThe conversion relationship is as follows:
the conversion relation is the relation between the predicted 3D point and the fused set of point clouds in the standard 3D coordinate, and is used for generating model point clouds by point fusion of the predicted coordinate, and the standard 3D coordinate system is a Cartesian coordinate system;
where K is the camera matrix, (R)N,tN) For rigid transformation matrices at N viewpoints, tNIs a three-dimensional translation vector, RNIs a rotation matrix, N ═ 1,2, …, N;
the structure generator is composed of linear layers and 2D convolutional layers, the 2D convolutional layers comprise multiple layers of convolution, the convolution kernel size of the 2D convolutional layers is 3 x 3, and point cloud fusion is performed on the linear layers through halving the size of each convolution operation feature graph to generate a prediction three-dimensional model.
4. The reconstruction method according to claim 1, wherein a pseudo renderer is used for pseudo rendering the target image, so that the dense point clouds are synthesized into a new depth image, and an up-sampling factor U of the pseudo renderer takes a value of 50;
obtaining a 2D projection depth map of a new view angle from the projection of the prediction three-dimensional model, and recording the depth map as a pseudo rendering depth map
The standard 3D coordinates are converted to image coordinates, and at the kth new viewpoint, the standard 3D coordinates PiConversion to image coordinates
The conversion relationship is as follows:
where K is the camera matrix, (R)k，tk) A three-dimensional rigid transformation matrix of k viewpoints;
for the projected positions of the three-dimensional model points in the 2D projected depth map,
are pixel values of the projected depth image.
5. The reconstruction method according to claim 1, wherein the loss function L used in the optimization of the predictive three-dimensional model is:
wherein
Projecting a depth map, Z, for the predicted three-dimensional model of the t-th viewpointtProjecting a depth map for the CAD model;
measured using the average point-by-point 3D Euclidean distance between the predicted three-dimensional model and the CAD model, i.e. for points P in the predicted three-dimensional modeliAnd point P in the CAD modeljAnd (3) performing distance calculation, minimizing the distance of all the points, namely, maximizing the fitting degree of the two models, wherein an error calculation formula is as follows:
error epsiloniIndicating that 3D shape similarity is measured.
6. The reconstruction method according to claim 1, wherein the road surface defect types include cracks, pits, ruts, wave hugs, ruts; the morphological parameters of the disease three-dimensional model comprise maximum length, maximum width, maximum height difference, volume and orthographic projection area.
7. The reconstruction method according to claim 1, wherein the construction process of the standardized pavement disease image database is: the method comprises the steps of obtaining multi-path surface images containing diseases at different shooting angles, segmenting the multi-path surface images containing the diseases at different shooting angles, extracting areas with the road surface diseases to form road surface disease images at different angles, classifying all the road surface disease images by using a K-means clustering algorithm, adding labels to each classified disease image, storing the labels, recording the disease types and the corresponding shooting angles, and obtaining a standardized road surface disease image database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110159075.4A CN112800524A (en) | 2021-02-05 | 2021-02-05 | Pavement disease three-dimensional reconstruction method based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110159075.4A CN112800524A (en) | 2021-02-05 | 2021-02-05 | Pavement disease three-dimensional reconstruction method based on deep learning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112800524A true CN112800524A (en) | 2021-05-14 |
Family
ID=75814340
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110159075.4A Pending CN112800524A (en) | 2021-02-05 | 2021-02-05 | Pavement disease three-dimensional reconstruction method based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112800524A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113538487A (en) * | 2021-07-14 | 2021-10-22 | 安徽炬视科技有限公司 | Virtual three-dimensional perimeter management and control algorithm based on multi-camera three-dimensional reconstruction |
| CN113655477A (en) * | 2021-06-11 | 2021-11-16 | 成都圭目机器人有限公司 | Method for automatically detecting geological diseases of land radar by adopting shallow layer |
| CN114049618A (en) * | 2022-01-12 | 2022-02-15 | 河北工业大学 | Graph-point-graph transformation-based pavement three-dimensional disease PCI calculation method |
| CN115082616A (en) * | 2022-05-18 | 2022-09-20 | 北京大学深圳医院 | Method for generating 3D point cloud model by ultrasonic image based on deep learning |
| CN115937289A (en) * | 2022-11-24 | 2023-04-07 | 深圳市城市交通规划设计研究中心股份有限公司 | Rut depth calculation method based on three-dimensional reconstruction of road rut diseases |
| CN116485887A (en) * | 2023-01-16 | 2023-07-25 | 湖北普罗格科技股份有限公司 | Unsupervised 3D carton detection method and system |
| CN116664773A (en) * | 2023-06-02 | 2023-08-29 | 北京元跃科技有限公司 | Method and system for generating 3D model by multiple paintings based on deep learning |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104778684A (en) * | 2015-03-06 | 2015-07-15 | 江苏大学 | Method and system thereof for automatically measuring, representing and classifying heterogeneous defects on surface of steel |
| CN107392232A (en) * | 2017-06-23 | 2017-11-24 | 中南大学 | A kind of flotation producing condition classification method and system |
| CN109461180A (en) * | 2018-09-25 | 2019-03-12 | 北京理工大学 | A kind of method for reconstructing three-dimensional scene based on deep learning |
| CN111462329A (en) * | 2020-03-24 | 2020-07-28 | 南京航空航天大学 | Three-dimensional reconstruction method of unmanned aerial vehicle aerial image based on deep learning |
| CN112070756A (en) * | 2020-09-15 | 2020-12-11 | 河北工业大学 | Three-dimensional road surface disease measuring method based on unmanned aerial vehicle oblique photography |
| CN112258529A (en) * | 2020-11-02 | 2021-01-22 | 郑州大学 | Pavement crack pixel level detection method based on example segmentation algorithm |
-
2021
- 2021-02-05 CN CN202110159075.4A patent/CN112800524A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104778684A (en) * | 2015-03-06 | 2015-07-15 | 江苏大学 | Method and system thereof for automatically measuring, representing and classifying heterogeneous defects on surface of steel |
| CN107392232A (en) * | 2017-06-23 | 2017-11-24 | 中南大学 | A kind of flotation producing condition classification method and system |
| CN109461180A (en) * | 2018-09-25 | 2019-03-12 | 北京理工大学 | A kind of method for reconstructing three-dimensional scene based on deep learning |
| CN111462329A (en) * | 2020-03-24 | 2020-07-28 | 南京航空航天大学 | Three-dimensional reconstruction method of unmanned aerial vehicle aerial image based on deep learning |
| CN112070756A (en) * | 2020-09-15 | 2020-12-11 | 河北工业大学 | Three-dimensional road surface disease measuring method based on unmanned aerial vehicle oblique photography |
| CN112258529A (en) * | 2020-11-02 | 2021-01-22 | 郑州大学 | Pavement crack pixel level detection method based on example segmentation algorithm |
Non-Patent Citations (1)
| Title |
|---|
| CHEN-HSUAN LIN, CHEN KONG, SIMON LUCEY: "Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction", 《THE THIRTY-SECOND AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE (AAAI-18)》 * |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113655477A (en) * | 2021-06-11 | 2021-11-16 | 成都圭目机器人有限公司 | Method for automatically detecting geological diseases of land radar by adopting shallow layer |
| CN113655477B (en) * | 2021-06-11 | 2023-09-01 | 成都圭目机器人有限公司 | Method for automatically detecting geological diseases by adopting shallow layer ground radar |
| CN113538487A (en) * | 2021-07-14 | 2021-10-22 | 安徽炬视科技有限公司 | Virtual three-dimensional perimeter management and control algorithm based on multi-camera three-dimensional reconstruction |
| CN114049618A (en) * | 2022-01-12 | 2022-02-15 | 河北工业大学 | Graph-point-graph transformation-based pavement three-dimensional disease PCI calculation method |
| CN114049618B (en) * | 2022-01-12 | 2022-03-22 | 河北工业大学 | Graph-point-graph transformation-based pavement three-dimensional disease PCI calculation method |
| CN115082616A (en) * | 2022-05-18 | 2022-09-20 | 北京大学深圳医院 | Method for generating 3D point cloud model by ultrasonic image based on deep learning |
| CN115937289A (en) * | 2022-11-24 | 2023-04-07 | 深圳市城市交通规划设计研究中心股份有限公司 | Rut depth calculation method based on three-dimensional reconstruction of road rut diseases |
| CN115937289B (en) * | 2022-11-24 | 2023-10-20 | 深圳市城市交通规划设计研究中心股份有限公司 | Rut depth calculation method based on three-dimensional reconstruction of pavement rut disease |
| CN116485887A (en) * | 2023-01-16 | 2023-07-25 | 湖北普罗格科技股份有限公司 | Unsupervised 3D carton detection method and system |
| CN116485887B (en) * | 2023-01-16 | 2024-02-02 | 湖北普罗格科技股份有限公司 | Unsupervised 3D carton detection method and system |
| CN116664773A (en) * | 2023-06-02 | 2023-08-29 | 北京元跃科技有限公司 | Method and system for generating 3D model by multiple paintings based on deep learning |
| CN116664773B (en) * | 2023-06-02 | 2024-01-16 | 北京元跃科技有限公司 | Method and system for generating 3D model by multiple paintings based on deep learning |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112800524A (en) | Pavement disease three-dimensional reconstruction method based on deep learning | |
| CN111928862B (en) | Method for on-line construction of semantic map by fusion of laser radar and visual sensor | |
| CN114842438B (en) | Terrain detection method, system and readable storage medium for automatic driving automobile | |
| CN110648389A (en) | 3D reconstruction method and system for city street view based on cooperation of unmanned aerial vehicle and edge vehicle | |
| US8855442B2 (en) | Image registration of multimodal data using 3D-GeoArcs | |
| CN105096386A (en) | Method for automatically generating geographic maps for large-range complex urban environment | |
| CN105160702A (en) | Stereoscopic image dense matching method and system based on LiDAR point cloud assistance | |
| CN111141264B (en) | Unmanned aerial vehicle-based urban three-dimensional mapping method and system | |
| KR20220064524A (en) | Method and system for visual localization | |
| CN112833892B (en) | Semantic mapping method based on track alignment | |
| WO2023002093A1 (en) | Systems and methods for determining road traversability using real time data and a trained model | |
| CN115019208A (en) | Road surface three-dimensional reconstruction method and system for dynamic traffic scene | |
| CN111612829B (en) | High-precision map construction method, system, terminal and storage medium | |
| WO2022064242A1 (en) | The method of automatic 3d designing of constructions and colonies in an smart system using a combination of machine scanning and imaging and machine learning and reconstruction of 3d model through deep learning and with the help of machine learning methods | |
| CN116337068A (en) | Robot synchronous semantic positioning and mapping method and system based on humanoid thought | |
| CN113240755B (en) | City scene composition method and system based on street view image and vehicle-mounted laser fusion | |
| CA3122868A1 (en) | Method and system for generating an environment model for positioning | |
| CN111710039B (en) | High-precision map construction method, system, terminal and storage medium | |
| Boschenriedter et al. | Multi-session visual roadway mapping | |
| Gu et al. | Surveying and mapping of large-scale 3D digital topographic map based on oblique photography technology | |
| Vishnyakov et al. | Semantic scene understanding for the autonomous platform | |
| KR20230026916A (en) | 3d mapping method with time series information using drone | |
| CN113874681B (en) | Evaluation method and system for point cloud map quality | |
| Luo | Research on Reliable and Safe Occupancy Grid Prediction in Underground Parking Lots | |
| Buller et al. | Autonomous Vehicle Sensor Suite Data with Ground Truth Trajectories for Algorithm Development and Evaluation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210514 |