CN112184731A - Multi-view stereo depth estimation method based on antagonism training - Google Patents

Multi-view stereo depth estimation method based on antagonism training Download PDF

Info

Publication number
CN112184731A
CN112184731A CN202011037998.4A CN202011037998A CN112184731A CN 112184731 A CN112184731 A CN 112184731A CN 202011037998 A CN202011037998 A CN 202011037998A CN 112184731 A CN112184731 A CN 112184731A
Authority
CN
China
Prior art keywords
depth map
module
network
depth
training
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202011037998.4A
Other languages
Chinese (zh)
Other versions
CN112184731B (en
Inventor
王亮
范德巧
李建书
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing University of Technology
Original Assignee
Beijing University of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing University of Technology filed Critical Beijing University of Technology
Priority to CN202011037998.4A priority Critical patent/CN112184731B/en
Publication of CN112184731A publication Critical patent/CN112184731A/en
Application granted granted Critical
Publication of CN112184731B publication Critical patent/CN112184731B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20112—Image segmentation details
    • G06T2207/20132—Image cropping

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a multi-view stereo depth estimation method based on antagonism training. In the antagonism training network, a mapping between a network learning image and a corresponding depth map is generated, and a judgment network learns to distinguish whether the depth map comes from a generation module or a reference depth. During training, the whole network is trained by combining the generated loss function and the cross entropy countervailing loss function. The invention improves the deep learning ability of multi-view stereo depth estimation through antagonism training, and collects the spatial and temporal context information in the image depth direction by generating an antagonism network GAN, thereby allowing the network to combine more global information. The antagonism training network of the invention utilizes the antagonism training of the generation module and the discrimination module, adopts the gradient punishment as a soft constrained antagonism loss function, improves the training process of the original generation antagonism network, obviously reduces the memory occupation and the running time during the network training and testing, and improves the multi-view three-dimensional depth prediction precision.

Description

Multi-view stereo depth estimation method based on antagonism training
Technical Field
The invention relates to the fields of object detection, three-dimensional reconstruction and the like of computer vision, in particular to a multi-view stereo depth estimation method based on antagonism training.
Background
Depth estimation from stereo images is a core problem for many stereo vision tasks and has applications in many fields, such as 3D reconstruction, unmanned driving, object detection, robotic navigation and virtual reality, augmented reality, etc. Depth estimation is a computer vision task that aims at estimating depth from 2D images. The task inputs RGB images shot by a plurality of visual angles in the same scene, and outputs a depth map of the scene after processing. The depth map contains information on the subject distance in the image as viewed from the view.
A typical depth estimation algorithm comprises 4 steps: feature extraction, feature matching, depth calculation and depth refinement. Each step plays a crucial role for the overall performance of the depth estimation. Since generative warfare networks exhibit strong feature expression in various visual tasks, generative warfare networks have been applied to depth estimation to improve depth estimation accuracy and significantly surpass conventional approaches. Generation of a countermeasure network was first proposed by Goodfellow et al, who trained two neural networks (generators and discriminators) simultaneously. (I.Goodfellow, J.Pouget-Abadie, M.Mirza, B.Xu, D.Warde-Farley, S.Ozar, A.Courville, and Y.Bengio.Generateiveadaptive networks in Advances in Neural Information Processing Systems 27(NIPS 2014), pages 2672-2680, 2014.) the distribution output by the generators is similar to the distribution of the real data, and the discriminator will distinguish whether the distribution is from the generators or the real data, being a countermeasure training process of mutual game. Pu c. and Song r. et al use an end-to-end architecture similar to the generation of countermeasure networks to learn complex disparity relationships between pixels to improve depth fusion accuracy. (Pu C, Song R, Tylecek R, et al, SDF-MAN: Semi-Supervised discrimination Fusion with Multi-Scale additive Networks [ J ]. Remote Sensing,2019,11(5):487.) they believe that training can be simplified by the concept of antagonism. Following this idea, several approaches have emerged to improve computational efficiency or depth accuracy. However, these methods still have some limitations. In particular, existing network operations are very memory consuming and require significant computational processing power, which cannot be handled for high resolution scenarios.
Disclosure of Invention
The invention mainly adopts a depth learning method to process the input multi-view stereo image so as to obtain a continuous and accurate depth map. Firstly, a 2-dimensional convolution neural network is utilized to extract the characteristics of an input stereo image, and then a coding network is utilized to code the extracted characteristic vectors. Then, the generating module is used for carrying out upsampling to generate an initial depth map. And then, inputting the initial depth map and the reference depth map of the reference image into a discrimination module to discriminate authenticity. And finally, further refining the depth estimation by utilizing a depth residual error learning network to obtain accurate depth estimation.
In order to achieve the above object, the present invention provides the following solutions:
a method of multi-view stereo depth estimation based on antagonism training, the method comprising:
step 1: processing data;
step 2: constructing a depth estimation network;
and step 3: training a network model;
and 4, step 4: and performing multi-view stereo depth estimation by using the trained depth estimation network model.
The data processing specifically comprises the following steps:
step 1: data set: for each group of stereoscopic images taken at multiple viewing angles in a data set, without special description, the lower image of one data set is taken as a reference image and the upper image is taken as a corresponding adjacent image. All stereo images are rectified, i.e. there is only a shift in the horizontal direction and no shift in the vertical direction.
Step 2: pretreatment: and randomly cutting each input stereo image in the data set, cutting the cutting size according to the specific situation of a server, and then performing normalization operation on the image to enable the image color value range to be between-1 and 1.
The method for constructing the depth estimation network specifically comprises the following modules:
module 1: initial feature extraction module
In the training stage, the initial feature extraction module is used for extracting features of the input stereo image group, and the stereo image group is adopted in the training stage because the running memory of the graphics card GPU can be reduced. And a testing stage, wherein the testing stage is used for extracting the characteristics of the input stereo image to be estimated, the specific input is N input stereo images to be estimated, and the output is N unary characteristics. The 2-dimensional convolutional neural network is composed of 8 convolutional layers, and each convolutional layer sequentially performs downsampling on the input N three-dimensional images; the 8 convolutional layers for feature extraction are, except for the last layer, each convolutional layer is followed by a residual block structure composed of a Batch Normalization (BN) layer and a modified linear unit (ReLU), convolution kernels of the residual block structure are all 3x3, feature dimensions are all 32, and the step length is 1; after the convolution operation, the output of the 2-dimensional convolution neural network is N unary feature vectors with the size of H/4 xW/4 xF, wherein H, W respectively represents the height and width of an original input image, and F represents a feature dimension;
and (3) module 2: coding module
The encoding module consists of a convolution maximum pooling layer with 5 layers of convolution kernels of 2 multiplied by 2, in the training stage, N unary feature vectors of a reference image and an adjacent image obtained from the feature extraction module are input, and N one-dimensional hidden feature codes z are output; in the testing stage, the feature vector of an input stereo image to be estimated is input, and the coding module projects the unary feature vector to a hidden space Z for coding to generate a feature code Z;
and a module 3: generation module
The generation module is composed of a deconvolution layer with 7 layers of convolution kernels of 2 multiplied by 2 and is used for carrying out upsampling to generate an initial depth map, the input of the initial depth map is N characteristic codes z, and the output of the initial depth map is N single-channel gray level depth maps; the 7 layers of deconvolution layers of the generating module are, except for the output layer, a Batch Normalization (BN) layer and a modified linear unit (ReLU) after each layer; after the deconvolution operation, the output of the deconvolution neural network is N initial depth maps, the dimensionality of the initial depth maps is H multiplied by W multiplied by 1, wherein H, W respectively represents the height and the width of an original input image;
and (4) module: discrimination module
The judging module consists of 6 layers of convolution layers, the convolution kernel is 5 multiplied by 5, the step length is 2, and the judging module is used for judging the authenticity of the initial depth map; the input is the initial depth map generated by the generation module and the reference depth map of the reference image obtained by labeling, and the output is the judgment result of the initial depth map generated by the generation module; the discrimination module judges the EM distance between the initial depth map distribution generated by the generation module and the reference depth map distribution of the reference image
Figure BDA0002705705220000041
Judging the authenticity of the initial depth map; Π (P) in the above EM distancer,Pg) For the set of all possible joint distributions formed by combining the reference depth map distribution and the initial depth map distribution, for each possible joint distribution gamma, sampling (x, y) -gamma to obtain a real sample x and a generated sample y, calculating the distance | | | | x-y | | | | of the pair of samples, and calculating the expected value E of the samples to the distance under the joint distribution(x,y)～γ[||x-y||]. Can be applied to this expectation E in all possible joint distributions(x,y)～γ[||x-y||]The lower bound, taken, is defined as the EM distance. The judging module judges the initial depth map generated by the generating module G and the reference depth map of the reference image, and if the initial depth map and the reference depth map meet the EM distance condition, the expected value E of the sample pair distance under the combined distribution is(x,y)～γ[||x-y||]If the lower bound can be obtained, the judgment is true, otherwise, the judgment is false. The 6 convolution layers of the discrimination module are divided into an input layer, and each layer is followed by a Batch Normalization (BN) layer and a leakage correction linear unit (leakage ReLU); the discrimination module is only used in the training stage.
And a module 5: depth map refinement module
The depth map refining module further refines depth estimation by using a depth residual error learning network, wherein the input of the depth map refining module is an initial depth map, and the output of the depth map refining module is a final depth map; the deep thinning operation process comprises the following steps: firstly, an initial depth map performs upsampling by utilizing bilinear interpolation; then, the output result passes through a convolution layer with convolution kernel of 3x3 and channel of 32 and then passes through 6 residual blocks with expansion rates of 1, 2, 4, 8, 1 and 1; each residual block structure is BN-conv-BN-ReLU-conv-BN, wherein BN, conv and ReLU refer to batch normalization, convolution layer and modified linear unit respectively. Then, the output of the residual block is sent to a convolution layer with the dimensionality of 1 and the convolution kernel of 3x3, and the output result is the final depth map; the final depth map output by the depth refinement module has dimensions H × W × 1, where H, W represents the height and width of the original input image, respectively.
The training network model specifically comprises the following steps:
step 1: and inputting the multi-view stereo images of the training data set into a model of the depth estimation network for forward propagation training, wherein the learning parameters of the model comprise weight and bias, and the random initialization parameters train the network model from the beginning.
Step 2: training the whole depth estimation network model by using an overall loss function, wherein the overall loss function is as follows:
Figure BDA0002705705220000051
wherein L isgenerationThe generative loss function is used to train the generative network,
Figure BDA0002705705220000052
is a cross-entropy countering loss function used to train a discriminant network, beta1And beta2Are the weighting parameters that adjust the generation penalty and the cross-entropy countering penalty.
The generation loss function is:
Figure BDA0002705705220000053
and M is an unshielded mask of effective pixel points of the characteristic points of the reference image and the adjacent image acquired during the characteristic extraction. The generative loss function is used to train the generative network.
The generation loss function includes an L1 distance between the image and the gradient, a Structural Similarity (SSIM), and a depth smoothing term, where the depth smoothing term is to improve the smoothness of the initially generated depth map, and the three terms are defined as follows:
Figure BDA0002705705220000054
wherein, I'j→iFor adjacent pictures IjWith adjacent picture IiThe mapping relationship between the two; lambda [ alpha ]1,λ2To adjust the percentage parameter of the gradient;
Figure BDA0002705705220000061
the loss stability can be improved and is a stable factor;
Figure BDA0002705705220000062
is a gradient operator.
Figure BDA0002705705220000063
Wherein S (-) represents the structural similarity SSIM, lambda3To adjust the percentage parameter of structural similarity.
Figure BDA0002705705220000064
Where N is the total number of all image feature pixels, diAs an image IiDepth of (a)1，α2A percentage parameter for smoothness adjustment;
in the originally generated confrontation network model, the training targets of the generation module G and the discrimination module D are as follows:
Figure BDA0002705705220000065
wherein, PrealFor the reference depth map distribution, PrefinerThe initial depth map distribution generated for the generating module G. In the process of generating training in the original generation countermeasure network, the training is easy to crash due to weight clipping. The invention proposes a gradient penalty-based loss function as a soft constraint, improving the training process. Thus, the cross-entropy countervailing loss function is:
Figure BDA0002705705220000066
where theta is the penalty factor and where theta is the penalty factor,
Figure BDA0002705705220000067
to represent
Figure BDA0002705705220000068
Belonging to the initial depth map distribution P generated by the generating module GrefinerRandom sampling of (1); x to PrealIndicating x belongs to the reference depth map distribution PrealRandom sampling of (1);
Figure BDA0002705705220000069
to represent
Figure BDA00027057052200000610
Joint distribution belonging to reference depth map distribution and initial depth map
Figure BDA00027057052200000611
Random sampling of (1); di(-) represents the weight of the discrimination network D;
Figure BDA00027057052200000612
representing a gradient penalty of the discrimination network D; the cross-entropy countermeasures loss function is used to train the discriminant network.
And step 3: and (5) repeating the step (1) and the step (2), and continuously iterating and training the parameters of the network model to obtain an optimal depth estimation network model.
And 4, step 4: and (3) inputting the initial depth map obtained by the final countermeasure training learning in the steps 1, 2 and 3 into a deep refinement network for residual error learning to obtain a final depth map.
And performing multi-view stereo depth estimation by using the trained depth estimation network model.
Has the advantages that:
the invention provides a multi-view stereo depth estimation method based on antagonism training, which comprises 5 steps including initial feature extraction, feature coding, initial depth map calculation, antagonism training and depth map refinement, wherein each step is designed in detail, and meanwhile, the 5 steps are integrated into a network, so that multi-view stereo depth estimation can be realized end to end. In the antagonism training network, the depth estimation method of the invention utilizes the antagonism training of the generation module and the discrimination module, adopts the gradient punishment as a soft constrained antagonism loss function, improves the original generation antagonism network training process, obviously reduces the memory occupation and the running time during the network training and testing, and improves the multi-view three-dimensional depth prediction precision.
Drawings
FIG. 1 is a network flow chart of a method for estimating depth of a multi-view stereo based on antagonism training provided by the present invention;
fig. 2(a) is a network work flow chart in a training phase, fig. 2(b) is a network work flow chart, fig. 2(c) is a structure chart of a discrimination module, fig. 2(d) is a structure chart of a depth map refinement module, and fig. 2(e) is a system work flow chart in a testing phase.
FIG. 3 is a reference image and its neighboring images to be estimated in the Tanks & samples dataset according to an embodiment of the present invention, where 3(a) is the reference image and 3(b) and 3(c) are the neighboring images;
FIG. 4 is a final depth map of an embodiment stereo image in a Tanks & Temples dataset obtained using the method of the invention;
Detailed Description
The invention aims to provide a multi-view stereo depth estimation method based on antagonism training, which can realize multi-view stereo depth estimation end to end without any post-processing process and can obviously reduce memory occupation and running time during training/testing.
The present invention will be described in detail below with reference to the attached drawings, and it should be noted that the described embodiments are only intended to facilitate understanding of the present invention, and do not have any limiting effect thereon.
Fig. 1 is a flowchart of a method for estimating a depth of a multi-view stereo based on antagonism training according to the present invention. Fig. 2 is a workflow diagram of different stages of the method for estimating depth of a multi-view stereo based on antagonism training and a schematic structural diagram of each module. The multi-view stereo depth estimation method based on antagonism training provided by the invention specifically comprises the following steps:
step 1: processing data; and randomly cutting the image containing the real parallax value, wherein the cutting size is 640 multiplied by 512, and normalizing the cut image to enable the range of the image pixel value to be between-1 and 1. One image is selected as a reference image, and the other images are adjacent images, as shown in fig. 3, fig. 3(a) is the reference image, fig. 3(b) and 3(c) are the adjacent images, and the reference image and the adjacent images thereof form a group of multi-view stereo images. The training sample stereo image is a DTU data set, and the test image is a Tanks & Temples data set.
Step 2: constructing a depth estimation network; first, an initial feature representation of the image is extracted. A feature representation is typically used instead of computing using the raw pixel intensities. Inspired by the descriptor, the feature representation is more robust to the ambiguity of the illuminated surface, so the input image stereo image first extracts the depth feature representation by 8 convolutional layers. In order to realize better feature matching, the extracted feature vectors are input into a coding module for coding generation to generate feature codes. The simultaneous coding structure is proposed, which significantly reduces memory usage and run time during training/testing. And next, inputting the feature codes into a generation module for up-sampling operation to generate an initial depth map. And then, the discrimination module is used for discriminating the authenticity of the initial depth map generated by the generation module and the reference depth map of the reference image. In the deep refinement stage, a deep residual error learning network is utilized to further refine the depth estimation and generate a final depth map.
And step 3: training a network model: firstly, inputting a preprocessed training data set DTU multi-view stereo image into a model of a depth estimation network for forward propagation training, wherein learning parameters of the model comprise weight and bias. Then, utilize
Figure BDA0002705705220000091
Generating a loss function training a generating network, where1，λ2，λ3Set to be between 0.1 and 0.9, alpha1，α2The setting is between 0.5 and 0.9; by using
Figure BDA0002705705220000092
The cross-entropy countermeasures the loss function as a soft constraint of the gradient penalty, and the penalty factor theta is set to be between 0.0001 and 0.0005. Integral loss function beta1，β2The setting is between 0.8 and 0.9. In the network framework, the generation quality is judged by utilizing a discrimination network, and the parameters of the generation network and the discrimination network are alternately trained until convergence. And finally, updating the learning parameters of the iterative model for multiple times according to the gradient to obtain an optimal depth estimation network model.
And 4, step 4: estimating the depth;
the depth estimation network model is obtained by step 3, and the actual scene is now tested by using the data set Tanks & Temples multi-view stereo images. Fig. 3 is a set of stereo images to be estimated according to an embodiment of the present invention. In which fig. 3(a) is a reference image, and 3(b) and (c) are neighboring images. In this embodiment, the stereo image of the embodiment to be estimated is extracted from the Tanks & Temples dataset. Referring to fig. 1 and fig. 2, the depth estimation is performed by using stereo images in embodiments of the Tanks & Temples data set based on a multi-view stereo depth estimation method of antagonism training (the 3-order tensor dimension is H × W × F, the 4-order tensor dimension is H × W × D × F, H, W respectively represents the height and width of an original input image, D represents the maximum possible disparity value, and is 192 by default, and F represents a feature dimension):
1) randomly cutting the stereo image of the embodiment in the Tanks & Temples data set to an image block with the size of 640 multiplied by 512, then carrying out normalization processing on the image block to enable the pixel value range of the image to be between-1 and 1, and inputting the stereo image into a trained depth estimation network after finishing the preprocessing stage.
2) As shown in fig. 2, feature extraction is performed on the input stereoscopic image of the embodiment. First, feature extraction is performed on a stereo image by using a 2-dimensional convolutional neural network, and downsampling is performed twice, so that the output feature map dimension is 160 × 128 × 32 at this time.
3) And inputting the extracted initial characteristic vector into an encoding module for encoding. The initial feature vector is subjected to a coding process including 5 convolutional maximum pooling layers with 2 × 2 convolutional kernels, and then the feature code with the output size of 100 is output.
4) And generating an initial depth map. Inputting the feature code into a generating network, and outputting an initial depth map with feature map dimensions of 640 × 512 × 1 through an upsampling operation of an deconvolution layer with 7 layers of convolution kernels being 2 × 2.
5) And (5) deep thinning. And inputting the initial depth map into a depth residual error learning network for refining to obtain a final depth map.
FIG. 4 is a final depth map of a stereo image of an embodiment in a Tanks & Temples dataset obtained using the method of the present invention. By generating a countermeasure network to collect spatial and temporal context information in the image depth direction, the network is allowed to combine more global information, and the multi-view stereo depth estimation accuracy is improved. The entire Tanks & Temples dataset image (1920 x 1080) was processed up to 5Hz, and the running speed during testing was significantly increased compared to existing depth estimation networks.
The above description is only an embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can understand that the modifications and substitutions within the technical scope of the present invention are included in the scope of the present invention, and therefore, the scope of the present invention should be subject to the protection scope of the claims.

Claims (9)

1. A multi-view stereo depth estimation method based on antagonism training is characterized by comprising the following steps:
step 1: and constructing a data set and preprocessing the data set, wherein the data set is RGB images shot by multiple visual angles in the same scene, and comprises a reference image and adjacent images thereof, and the reference image and the adjacent images thereof are used as a group of input images.
Step 2: constructing a depth estimation network, wherein the depth estimation network comprises a feature extraction module, a coding module, a generation module, a discrimination module and a depth map refinement module, the feature extraction module is used for extracting features of an input stereo image pair, and the coding module is used for projecting a unary feature vector to a hidden space for coding to generate a feature code z, so that feature representation is more stable; the generating module is used for generating an initial depth map of the image according to the input hidden feature codes; the feature extraction module, the coding module and the generation module form a generation network together; the judging module is only used during training, and judges whether the initial depth map is true or not by using a reference depth map of a reference image, wherein the reference depth map is obtained by labeling the reference image; the depth map refining module is used for generating a final depth map;
and step 3: model training: firstly, inputting a preprocessed training data set multi-view stereo image into a generation network of a depth estimation network for forward propagation calculation to obtain an initial depth map of an adjacent image; then, inputting the output initial depth map of the adjacent image and the reference depth map of the reference image into a discrimination network, and performing back propagation by using a batch gradient descent method; in the process of mutual game confrontation training of the generation network and the discrimination module, the learning parameters of the iterative model are updated for multiple times according to the gradient to obtain an optimal depth estimation network model, wherein the learning parameters of the model comprise weight and bias; finally, inputting the initial depth map obtained by the final confrontation training learning into a depth refinement network for residual learning to obtain a final depth map; during training, training the whole network model by using a whole loss function;
and 4, step 4: performing depth estimation on a multi-view stereo image to be estimated by using the trained depth estimation network model, and specifically, inputting the multi-view stereo image to be estimated into a feature extraction module for feature extraction; then, generating a hidden feature code through a coding module; then generating a preliminary depth map through a generating module; and finally, thinning the initial depth map through a depth map thinning module to obtain a final depth map so as to finish the multi-view depth estimation.
2. The multi-view depth estimation method according to claim 1, wherein the preprocessing in step 1 refers to: each input stereo image in the data set is randomly cropped and then normalized to have an image color value range between-1, 1.
3. The method of multi-view depth estimation according to claim 1, wherein the feature extraction module is a 2-dimensional convolutional neural network; the 2-dimensional convolutional neural network is composed of 8 layers of convolutional layers, the convolutional kernel is 5 multiplied by 5, the step length is 2, and each layer of convolutional layer sequentially carries out downsampling on the input N three-dimensional images; the 8 convolutional layers with the extracted characteristics except the last layer are sequentially combined by a Batch Normalization (BN) layer and a modified linear unit (ReLU) after each convolutional layer, the convolution kernels of the residual block structures are all 3 multiplied by 3, the characteristic dimensions are all 32, and the step length is 1; after the convolution operation, the output of the 2-dimensional convolution neural network is N unary feature vectors with the size of H/4 xW/4 xF, wherein H, W represents the height and width of the original input image respectively, and F represents the feature dimension.
4. The multi-view depth estimation method of claim 1, wherein the coding module is composed of 5 convolutional max pooling layers with convolutional kernel 2 x 2, the input is N unary feature vectors, and the output is N one-dimensional feature codes z; and the coding module projects the unary feature vector to a hidden space Z for coding and dimension reduction to generate a feature code Z.
5. The multi-view depth estimation method of claim 1, wherein the generation module is composed of a deconvolution layer with 2 x 2 convolution kernels for up-sampling to generate an initial depth map, the input of which is N implicit feature codes z, and the output of which is N single-channel grayscale depth maps; the 7 layers of deconvolution layers of the generating module are sequentially a Batch Normalization (BN) layer and a modified linear unit (ReLU) after each layer except an output layer; after the deconvolution operation, the output of the deconvolution neural network is N initial depth maps with dimensions H × W × 1, where H, W respectively represent the height and width of the original input image.
6. The multi-view depth estimation method according to claim 1, wherein the discriminating module is composed of 6 convolutional layers, the convolutional layer is 5 x 5, the step size is 2, and is used for discriminating the authenticity of the depth map; the input is an initial depth map generated by a generating module and a reference depth map of a reference image, and the output is a judgment result of the depth map; the 6 convolution layers of the discrimination module are divided into an input layer, and each layer is followed by a Batch Normalization (BN) layer and a leakage correction linear unit (leakage ReLU); the discrimination module is only used in the training stage.
7. The multi-view depth estimation method according to claim 6, wherein the discrimination module determines an EM distance between the initial depth map distribution generated by the generation module and a reference depth map distribution of the reference image
Figure FDA0002705705210000031
Judging the authenticity of the initial depth map; Π (P) in the above EM distancer,Pg) For the set of all possible joint distributions formed by combining the reference depth map distribution and the initial depth map distribution, for each possible joint distribution gamma, sampling (x, y) -gamma to obtain a real sample x and a generated sample y, calculating the distance | | | | x-y | | | | of the pair of samples, and calculating the expected value E of the samples to the distance under the joint distribution(x,y)～γ[||x-y||](ii) a Can be applied to this expectation E in all possible joint distributions(x,y)～γ[||x-y||]Lower bound takenDefined as the EM distance; the judging module judges the initial depth map generated by the generating module G and the reference depth map of the reference image, and if the initial depth map and the reference depth map meet the EM distance condition, the expected value E of the sample pair distance under the combined distribution is(x,y)～γ[||x-y||]And if the lower bound can be obtained, judging the depth map to be true, inputting the initial depth map obtained at the moment into the depth map thinning module, otherwise, judging the depth map to be false, and regenerating the initial depth map by the generating module G.
8. The multi-view depth estimation method of claim 1, wherein the depth map refinement module employs a depth residual learning network for further refinement of the depth estimation, the input being an initial depth map and the output being a final depth map; firstly, an initial depth map performs upsampling by utilizing bilinear interpolation; then, the output result passes through a convolution layer with convolution kernel of 3x3 and channel of 32 and then passes through 6 residual blocks with expansion rates of 1, 2, 4, 8, 1 and 1; each residual block structure is BN-conv-BN-ReLU-conv-BN, wherein BN, conv and ReLU refer to batch normalization, convolution layer and modified linear unit respectively; then, the output of the residual block is sent to a convolution layer with the dimensionality of 1 and the convolution kernel of 3x3, and the output result is the final depth map; the final depth map output by the depth refinement module has dimensions H × W × 1, where H, W represents the height and width of the original input image, respectively.
9. The method according to claim 1, wherein the overall loss function in step 3 is specifically as follows:
Figure FDA0002705705210000041
wherein
Figure FDA0002705705210000045
Is the generation of the loss function or functions,
Figure FDA0002705705210000046
is a cross-entropy countering loss function, beta1And beta2Is a percentage parameter for adjusting the generating loss function and the cross entropy counteracting loss function;
the generation loss function is:
Figure FDA0002705705210000042
wherein M is an unshielded mask of effective pixel points of the characteristic points of the reference image and the adjacent image acquired during the characteristic extraction;
the above generation penalty includes L1 distance between the image and the gradient, Structural Similarity (SSIM), and depth smoothing terms, where the depth smoothing terms are to improve the smoothness of the initially generated depth map, and these three terms are defined as follows:
Figure FDA0002705705210000043
wherein, I'j→iFor adjacent pictures IjWith adjacent picture IiThe mapping relationship between the two; lambda [ alpha ]1,λ2To adjust the percentage parameter of the gradient;
Figure FDA0002705705210000047
the loss stability can be improved and is a stable factor;
Figure FDA0002705705210000048
is a gradient operator.
Figure FDA0002705705210000044
Wherein S (-) represents the structural similarity SSIM, lambda3To adjust the percentage parameter of the structural similarity;
Figure FDA0002705705210000051
where N is the total number of all image feature pixels, diAs an image IiDepth of (a)1、α2A percentage parameter for smoothness adjustment;
the cross-entropy countering loss function is:
Figure FDA0002705705210000052
where theta is the penalty factor and where theta is the penalty factor,
Figure FDA0002705705210000053
to represent
Figure FDA0002705705210000054
Belonging to the initial depth map distribution P generated by the generating module GrefinerRandom sampling of (1); x to PrealIndicating x belongs to the reference depth map distribution PrealRandom sampling of (1);
Figure FDA0002705705210000055
to represent
Figure FDA0002705705210000058
Joint distribution belonging to reference depth map distribution and initial depth map
Figure FDA0002705705210000057
Random sampling of (1); di(-) represents the weight of the discrimination network D;
Figure FDA0002705705210000056
representing a gradient penalty of the discrimination network D;
the original generation countermeasure network easily causes training collapse due to weight clipping in the generation training process. The invention provides a gradient penalty-based confrontation loss function as a soft constraint, and improves the training process.
CN202011037998.4A 2020-09-28 2020-09-28 Multi-view stereoscopic depth estimation method based on contrast training Active CN112184731B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011037998.4A CN112184731B (en) 2020-09-28 2020-09-28 Multi-view stereoscopic depth estimation method based on contrast training

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011037998.4A CN112184731B (en) 2020-09-28 2020-09-28 Multi-view stereoscopic depth estimation method based on contrast training

Publications (2)

Publication Number Publication Date
CN112184731A true CN112184731A (en) 2021-01-05
CN112184731B CN112184731B (en) 2024-05-28

Family

ID=73944338

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011037998.4A Active CN112184731B (en) 2020-09-28 2020-09-28 Multi-view stereoscopic depth estimation method based on contrast training

Country Status (1)

Country Link
CN (1) CN112184731B (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114998683A (en) * 2022-06-01 2022-09-02 北京理工大学 Attention mechanism-based ToF multipath interference removing method
WO2022207438A1 (en) * 2021-03-29 2022-10-06 Conti Temic Microelectronic Gmbh Method and system for estimating depth information
CN116757068A (en) * 2023-06-06 2023-09-15 青岛理工大学 Prediction method for CFRP self-piercing riveting forming process based on deep learning

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108389171A (en) * 2018-03-08 2018-08-10 深圳市唯特视科技有限公司 A kind of light field deblurring and depth estimation method based on Combined estimator fuzzy variable
CN108564611A (en) * 2018-03-09 2018-09-21 天津大学 A kind of monocular image depth estimation method generating confrontation network based on condition
US20190325597A1 (en) * 2018-04-24 2019-10-24 Ford Global Technologies, Llc Simultaneous Localization And Mapping Constraints In Generative Adversarial Networks For Monocular Depth Estimation
CN110533712A (en) * 2019-08-26 2019-12-03 北京工业大学 A kind of binocular solid matching process based on convolutional neural networks
CN110738697A (en) * 2019-10-10 2020-01-31 福州大学 Monocular depth estimation method based on deep learning
CN110992217A (en) * 2019-11-15 2020-04-10 广东工业大学 Method and device for expressing and searching multi-view features of design patent

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108389171A (en) * 2018-03-08 2018-08-10 深圳市唯特视科技有限公司 A kind of light field deblurring and depth estimation method based on Combined estimator fuzzy variable
CN108564611A (en) * 2018-03-09 2018-09-21 天津大学 A kind of monocular image depth estimation method generating confrontation network based on condition
US20190325597A1 (en) * 2018-04-24 2019-10-24 Ford Global Technologies, Llc Simultaneous Localization And Mapping Constraints In Generative Adversarial Networks For Monocular Depth Estimation
CN110533712A (en) * 2019-08-26 2019-12-03 北京工业大学 A kind of binocular solid matching process based on convolutional neural networks
CN110738697A (en) * 2019-10-10 2020-01-31 福州大学 Monocular depth estimation method based on deep learning
CN110992217A (en) * 2019-11-15 2020-04-10 广东工业大学 Method and device for expressing and searching multi-view features of design patent

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
叶春凯;万旺根;: "基于特征金字塔网络的多视图深度估计", 电子测量技术, no. 11, 8 June 2020 (2020-06-08) *
李天成;何嘉;: "一种基于生成对抗网络的图像修复算法", 计算机应用与软件, no. 12, 12 December 2019 (2019-12-12) *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022207438A1 (en) * 2021-03-29 2022-10-06 Conti Temic Microelectronic Gmbh Method and system for estimating depth information
CN114998683A (en) * 2022-06-01 2022-09-02 北京理工大学 Attention mechanism-based ToF multipath interference removing method
CN114998683B (en) * 2022-06-01 2024-05-31 北京理工大学 Attention mechanism-based ToF multipath interference removal method
CN116757068A (en) * 2023-06-06 2023-09-15 青岛理工大学 Prediction method for CFRP self-piercing riveting forming process based on deep learning
CN116757068B (en) * 2023-06-06 2024-01-09 青岛理工大学 Prediction method for CFRP self-piercing riveting forming process based on deep learning

Also Published As

Publication number Publication date
CN112184731B (en) 2024-05-28

Similar Documents

Publication Publication Date Title
CN110533712B (en) Binocular stereo matching method based on convolutional neural network
CN110443842B (en) Depth map prediction method based on visual angle fusion
US10353271B2 (en) Depth estimation method for monocular image based on multi-scale CNN and continuous CRF
CN110490919B (en) Monocular vision depth estimation method based on deep neural network
CN108537871B (en) Information processing apparatus and information processing method
CN111612807B (en) Small target image segmentation method based on scale and edge information
CN112184731B (en) Multi-view stereoscopic depth estimation method based on contrast training
CN110381268B (en) Method, device, storage medium and electronic equipment for generating video
CN114565655B (en) Depth estimation method and device based on pyramid segmentation attention
CN108171249B (en) RGBD data-based local descriptor learning method
CN112541865A (en) Underwater image enhancement method based on generation countermeasure network
CN113034563A (en) Self-supervision type monocular depth estimation method based on feature sharing
CN114996814A (en) Furniture design system based on deep learning and three-dimensional reconstruction
CN116279592A (en) Method for dividing travelable area of unmanned logistics vehicle
CN116563682A (en) Attention scheme and strip convolution semantic line detection method based on depth Hough network
CN114972882A (en) Wear surface damage depth estimation method and system based on multi-attention machine system
CN114677349A (en) Image segmentation method and system for edge information enhancement and attention guidance of encoding and decoding
CN110889868A (en) Monocular image depth estimation method combining gradient and texture features
CN114693744A (en) Optical flow unsupervised estimation method based on improved cycle generation countermeasure network
Hou et al. Joint learning of image deblurring and depth estimation through adversarial multi-task network
CN117315153A (en) Human body reconstruction and rendering method and device for cooperative light field and occupied field
CN116681976A (en) Progressive feature fusion method for infrared small target detection
CN106056599B (en) A kind of object recognition algorithm and device based on Object Depth data
CN108460383A (en) Saliency refined method based on neural network and image segmentation
CN115330935A (en) Three-dimensional reconstruction method and system based on deep learning

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant