CN111445465B - Method and equipment for detecting and removing snow or rain belt of light field image based on deep learning - Google Patents
Method and equipment for detecting and removing snow or rain belt of light field image based on deep learning Download PDFInfo
- Publication number
- CN111445465B CN111445465B CN202010243333.2A CN202010243333A CN111445465B CN 111445465 B CN111445465 B CN 111445465B CN 202010243333 A CN202010243333 A CN 202010243333A CN 111445465 B CN111445465 B CN 111445465B
- Authority
- CN
- China
- Prior art keywords
- light field
- field image
- network
- rain
- snowflake
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000013135 deep learning Methods 0.000 title claims abstract description 33
- 241000533950 Leucojum Species 0.000 claims abstract description 105
- 230000006870 function Effects 0.000 claims abstract description 37
- 238000011156 evaluation Methods 0.000 claims abstract description 26
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims description 20
- 238000001514 detection method Methods 0.000 claims description 14
- 238000011176 pooling Methods 0.000 claims description 11
- 230000000873 masking effect Effects 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 15
- 238000003062 neural network model Methods 0.000 abstract description 2
- 230000005284 excitation Effects 0.000 description 8
- 238000012549 training Methods 0.000 description 7
- 230000008034 disappearance Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 230000002708 enhancing effect Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 210000002569 neuron Anatomy 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 230000001965 increasing effect Effects 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000000386 microscopy Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000003014 reinforcing effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
- G06T7/62—Analysis of geometric attributes of area, perimeter, diameter or volume
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Quality & Reliability (AREA)
- Health & Medical Sciences (AREA)
- Geometry (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a method and a system for detecting and removing snow or rain drops of a light field image based on deep learning, wherein the method comprises the following steps: detecting the position and the size of snowflakes or rain belts in the acquired first light field image by using a 3D residual error network; removing the detected snowflakes or rain bands by adopting a 3D U type network to obtain a second light field image; acquiring a second light field image and a third light field image, and optimizing a discriminator and an objective function by judging the authenticity of the second light field image and the third light field image; evaluating the quality of the generated second light field image through the peak signal-to-noise ratio and the structural similarity index, and if the evaluation result does not meet the requirement, executing the steps S11, S13 and S15 again until the evaluation result meets the requirement; according to the scheme, snowflake or rain belt data are directly detected, the characteristics of the light field image with rich three-dimensional structure information are utilized to detect and remove the snowflake or the rain belt, the middle hidden parameters of the neural network model are automatically and iteratively updated, and the snowflake or the rain belt removing effect is good.
Description
Technical Field
The invention relates to the technical field of computer image processing, in particular to a method and equipment for detecting and removing snow or rain drops in a light field image based on deep learning.
Background
With the tremendous development of the field of computer science under the support of high-precision technology, the research of the field of computer vision has also made a tremendous progress. In the field of computer vision, light field imaging techniques are very widely used. Compared with the traditional camera, the light field camera can acquire four-dimensional information of a scene through single exposure, wherein the four-dimensional information comprises two-dimensional space information and two-dimensional angle information, so that more abundant image information can be acquired in the image reconstruction process, and in addition, the problems of defocusing of images in special occasions, excessive background targets and the like can be solved through a digital refocusing technology; realizing perspective monitoring through a synthetic aperture technology; after being fused with the microscopy, the multi-view large-depth-of-field microscopic image and the reconstructed three-dimensional stereogram can be obtained. With the intensive research of the light field technology, light field image processing is gradually focused by expert students at home and abroad, and the industry represented by automatic driving is used as an important technical means for three-dimensional scene perception. The research focus of light field image processing is in aspects of depth estimation, super resolution, image restoration and the like.
In the field of computer vision, due to the complexity of illumination, shielding and the shape and color of the snowflake or rain belt, a snowflake or rain belt removing algorithm on a single image can only detect and remove the snowflake or rain belt according to the color and shape information in the image, and has great limitation and poor snowflake or rain belt removing effect.
In summary, the prior art scheme lacks a snowflake or rain belt removing scheme with good snowflake or rain belt removing effect.
Disclosure of Invention
The embodiment of the invention provides a method and a system for detecting and removing snow or rain strips of a light field image based on deep learning, which are used for solving the technical problem that a snow or rain strip removing scheme with a good snow or rain strip removing effect is lacked in the prior art.
In a first aspect, according to an embodiment of the present application, there is provided a method for detecting and removing snow or rain drops in a light field image based on deep learning, including:
step S11, a detector detects the position and the size of snowflakes or rain strips in the acquired first light field image by using a 3D residual error network;
step S13, a generator adopts a 3D U type network, and snow or rain strips in the detected first light field image are removed through self-learning to obtain a second light field image;
step S15, the discriminator acquires a second light field image and a third light field image, and optimizes an objective function of the discriminator by judging the authenticity of the second light field image and the third light field image, wherein the third light field image is a light field image without snowflakes or rain bands under the same scene as the first light field image, and is also called as a true value of the second light field image;
step S17, evaluating the quality of the generated second light field image through peak signal-to-noise ratio and structural similarity, and if the evaluation result does not meet the requirement, executing the steps S11, S13 and S15 again until the evaluation result meets the requirement;
the detecting, by using a 3D residual error network, a position and a size of a snowflake or a rain belt in the obtained first light field image includes: obtaining a mask of snowflake or raindrop data of the first light field image by using a 3D residual error network;
the masking of snowflake or rain data of the first light field image by using the 3D residual error network comprises the following steps: setting a residual block according to a 3D residual network; setting parameters of a first convolution layer and a second convolution layer, wherein each residual block uses the first convolution layer and the second convolution layer respectively; after the first convolution layer and the second convolution layer are used, adding standardized operation and a long-short-time memory network to obtain a mask of snowflake or raindrop data of the first light field image;
the step of removing the detected snow or rain belt in the first light field image by adopting the 3D U type network to obtain a second light field image comprises the following steps: removing snow or rain bands in the first light field image by using a 3D U type network according to the mask to obtain a second light field image;
removing snow or rain bands in the first light field image by using a 3D U type network to obtain a second light field image, wherein the method comprises the following steps of: dividing a type 3D U network into an encoder network and a decoder network according to the position relation; encoding and decoding snowflake or rain belt data by adopting an encoder network and a decoder network respectively; and adopting pooling layer downsampling, and setting an output function to obtain a second light field image.
In one embodiment, before the step S11, the method further includes:
the first light field image and the third light field image are acquired.
In one embodiment, the objective function of the arbiter is:
wherein V is T Is the true value of the 3D snowless EPI volume block of the virtual scene,
to add 3DEPI volumes of snowflake or rain-band data distribution masks,d represents the arbiter and G represents the generator.
In a second aspect, according to an embodiment of the present invention, there is provided a light field image snowflake or rain belt detection and removal system based on deep learning, including:
the detection module is used for detecting the position and the size of snowflakes or rain belts in the acquired first light field image by utilizing the 3D residual error network;
the removing module is used for removing snow or rain strips in the detected first light field image by adopting a 3D U type network through self-learning by the generator to obtain a second light field image;
the judging module is used for acquiring a second light field image and a third light field image, optimizing an objective function of the judging device by judging the authenticity of the second light field image and the third light field image, wherein the third light field image is a light field image without snowflake or rain belt under the same scene with the first light field image;
the evaluation module evaluates the quality of the generated second light field image through peak signal-to-noise ratio and structural similarity, and if the evaluation result does not meet the requirement, the steps S11, S13 and S15 are executed again until the evaluation result meets the requirement;
the detecting, by using a 3D residual error network, a position and a size of a snowflake or a rain belt in the obtained first light field image includes: obtaining a mask of snowflake or raindrop data of the first light field image by using a 3D residual error network;
the masking of snowflake or rain data of the first light field image by using the 3D residual error network comprises the following steps: setting a residual block according to a 3D residual network; setting parameters of a first convolution layer and a second convolution layer, wherein each residual block uses the first convolution layer and the second convolution layer respectively; after the first convolution layer and the second convolution layer are used, adding standardized operation and a long-short-time memory network to obtain a mask of snowflake or raindrop data of the first light field image;
the step of removing the detected snow or rain belt in the first light field image by adopting the 3D U type network to obtain a second light field image comprises the following steps: removing snow or rain bands in the first light field image by using a 3D U type network according to the mask to obtain a second light field image;
removing snow or rain bands in the first light field image by using a 3D U type network to obtain a second light field image, wherein the method comprises the following steps of: dividing a type 3D U network into an encoder network and a decoder network according to the position relation; encoding and decoding snowflake or rain belt data by adopting an encoder network and a decoder network respectively; and adopting pooling layer downsampling, and setting an output function to obtain a second light field image.
In a third aspect, according to an embodiment of the present invention, there is provided a light field image snowflake or rain strip detection and removal apparatus based on deep learning, including:
the detector is used for detecting the position and the size of snowflakes or rain belts in the acquired first light field image by using the 3D residual error network;
the generator adopts a 3D U type network to remove snow or rain strips in the detected first light field image to obtain a second light field image;
the second light field image and the third light field image are acquired, the authenticity of the second light field image and the third light field image is judged to be distinguished to optimize the objective function of the discriminator, and the third light field image is a light field image without snowflake or rain belt under the same scene with the first light field image;
the evaluator evaluates the quality of the generated second light field image through peak signal-to-noise ratio and structural similarity, and if the evaluation result does not meet the requirement, the detector, the generator and the discriminator are controlled to work again until the evaluation result meets the requirement;
the detecting, by using a 3D residual error network, a position and a size of a snowflake or a rain belt in the obtained first light field image includes: obtaining a mask of snowflake or raindrop data of the first light field image by using a 3D residual error network;
the masking of snowflake or rain data of the first light field image by using the 3D residual error network comprises the following steps: setting a residual block according to a 3D residual network; setting parameters of a first convolution layer and a second convolution layer, wherein each residual block uses the first convolution layer and the second convolution layer respectively; after the first convolution layer and the second convolution layer are used, adding standardized operation and a long-short-time memory network to obtain a mask of snowflake or raindrop data of the first light field image;
the step of removing the detected snow or rain belt in the first light field image by adopting the 3D U type network to obtain a second light field image comprises the following steps: removing snow or rain bands in the first light field image by using a 3D U type network according to the mask to obtain a second light field image;
removing snow or rain bands in the first light field image by using a 3D U type network to obtain a second light field image, wherein the method comprises the following steps of: dividing a type 3D U network into an encoder network and a decoder network according to the position relation; encoding and decoding snowflake or rain belt data by adopting an encoder network and a decoder network respectively; and adopting pooling layer downsampling, and setting an output function to obtain a second light field image.
In a fourth aspect, according to an embodiment of the present invention, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method as described in any of the above.
According to the deep learning-based method and system for detecting and removing the snow or the rain belt of the light field image, snow or rain belt data of the first light field image are directly detected, the snow or the rain belt data are detected and removed by utilizing the characteristic that the light field image has abundant three-dimensional structure information, and the snow or rain belt data removing effect is good; according to the scheme, the deep learning self-learning method is adopted to remove snowflakes or rain strips, intermediate parameters are not required to be estimated, the deep learning-based light field image snowflakes or rain strips can be directly adopted to detect and remove the system to achieve end-to-end snowflakes or rain strips, and the snowflakes or rain strips removing effect is good; meanwhile, in the detector and the generator, batch standardization operation and a long-short-time memory network are added behind each convolution layer, and the batch standardization layer is used for regularizing the model, accelerating the training of the model and avoiding gradient disappearance; the long-short-term memory network is used for enhancing the relation between adjacent view points, and the shielded snowflake or rain zone area is better processed through the information of the adjacent view points, so that the texture information with more complex background can be recovered.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention and do not constitute a limitation on the invention. In the drawings:
FIG. 1 is a flow chart of a method for detecting and removing snow or rain drops in a light field image based on deep learning according to an embodiment of the present invention;
FIG. 2 is a flow chart of another method for detecting and removing snow or rain drops in a light field image based on deep learning according to an embodiment of the present invention;
FIG. 3 is a flowchart of a method for detecting and removing snow or rain drops in a light field image based on deep learning according to an embodiment of the present invention;
FIG. 4 is a schematic structural diagram of a detection and removal system for snow or rain strips in a light field image based on deep learning according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a device for detecting and removing snow or rain drops in a light field image based on deep learning according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In a first aspect, according to an embodiment of the present invention, a method for detecting and removing snowflakes or rain bands in a light field image based on deep learning is provided, as shown in fig. 1, and may include the following steps:
step S11, a detector detects the position and the size of snowflakes or rain strips in the acquired first light field image by using a 3D residual error network;
in the embodiment of the present application, the first light field image may be a light field image acquired by the light field camera under the virtual scene light field. The position and the size of the snowflake or the rain belt in the acquired first light field image are detected, two continuous convolutions can be used, and batch standardization operation and a long-short-time memory network are added after each convolution, so that the position and the size of the snowflake or the rain belt are detected.
Step S13, a generator adopts a 3D U type network to remove snow or rain strips in the detected first light field image, so as to obtain a second light field image;
in the embodiment of the present application, the snowflake or rain band detected in step S11 is removed, and a type 3D U network may be adopted, and two continuous convolutions are also used, and a batch normalization operation and a long-short-term memory network are added after each convolution.
Step S15, the second light field image and the third light field image are obtained by the discriminator, and the objective function of the discriminator is optimized by judging the authenticity of the second light field image and the third light field image;
in the embodiment of the present application, the step of distinguishing the authenticity of the second light field image from the third light field image may use a plurality of convolution layers, and in the last convolution layer, the generated output is compared with the third light field image to obtain a corresponding output parameter.
Step S17, evaluating the quality of the generated second light field image through peak signal-to-noise ratio and structural similarity, and if the evaluation result does not meet the requirement, executing the steps S11, S13 and S15 again until the evaluation result meets the requirement;
the third light field image is a light field image without snowflake or rain band in the same scene as the first light field image, and can also be called as a true value of the second light field image.
In the embodiment of the present application, a self-learning depth algorithm is adopted, and if the evaluation result in the step S17 does not meet the preset requirement, the steps S11, S13, S15 are automatically and repeatedly executed until the evaluation result meets the preset requirement.
In the embodiment of the application, after the second light field image and the third light field image are acquired by the discriminator by adopting the self-learning depth algorithm, the authenticity of the second light field image and the third light field image is discriminated, and the generated snow-free image (second light field image) and the real snow-free image (third light field image) respectively output a value of 0 to 1 through the discriminator, wherein the closer the value is to 1, the closer the second light field image and the third light field image are, namely the better the snow or rain strip removing effect is. The quality of the generated snow-free image is quantitatively evaluated using the peak signal-to-noise ratio (PSNR) and the Structural Similarity (SSIM), and if the evaluation result does not satisfy the requirement of the objective function, steps S11, S13, S15 are performed again.
In this embodiment, as shown in fig. 2, before the step S11, the method further includes:
step S10, a first light field image and a third light field image are acquired. The specific method comprises the following steps: firstly, constructing a light field data set, constructing a virtual light field camera by using a blender to render the virtual scene data set, dividing data into preset (such as 256 multiplied by 256) sizes to obtain a final virtual scene light field data set, and collecting a virtual scene light field image to obtain a first light field image; and acquiring a real scene light field data set by using the light field camera to obtain a third light field image. The third light field image may also be a snowless image in the synthetic data set, and the first light field image may also be a light field image in a real scene. In the embodiment of the application, a virtual scene is constructed, a virtual scene light field data set is acquired and used for training and testing snowflakes or rain belts based on the deep neural network model and provided by the application, and the synthesized light field image can be replaced by the light field image of the real scene.
In the embodiment of the present application, in step S11, the detecting, by using the 3D residual network, the position and the size of the snowflake or the rain belt in the obtained first light field image includes:
and obtaining a mask of snowflake or raindrop data of the first light field image by using the 3D residual error network. The method specifically comprises the following steps:
1) Setting a residual block according to a 3D residual network;
2) Setting parameters of a first convolution layer and a second convolution layer, wherein each residual block uses the first convolution layer and the second convolution layer respectively;
after the first convolution layer and the second convolution layer are used, batch standardization operation and a long-short-time memory network are added, and then a mask of snowflake or raindrop data of the first light field image is obtained.
In particular, the residual blocks may be set according to preset requirements, then each residual block of the 3D residual network is set to use two consecutive convolutional layers, the convolution kernel size of each convolution layer may be set to 3 x 3, setting the excitation function of the first convolution layer as Relu, and setting the excitation function of the second convolution layer as Sigmoid; and adding batch standardization operation and a long-short-time memory network after the first convolution layer and after the second convolution layer respectively. The batch normalization operation is used for regularizing the model, accelerating the training of the model and avoiding the disappearance of gradients; the relationship between adjacent viewpoints is enhanced by using a convolution long-short time memory network, and for images, the corresponding characteristics are difficult to extract by using a common long-short time memory network, because the images are three-dimensional, the deficiency of the long-short time memory network can be exactly made up by convolution operation, and the shielded snowflake or rain zone area can be better processed by the information of the adjacent viewpoints, so that the texture information with more complex background can be recovered, and the specific formula is as follows:
wherein, the convolution is represented by,
representing matrix-corresponding element multiplication, where i, f, C, o, H, X are three-dimensional tensors. X is the input, here representing the 3D EPI volume of one line of sub-view of the light field, f representing the forgetting gate, which would read H t-1 And X t Outputting a value between 0 and 1 to each of the neuron states C t-1 In (2), 1 means complete retention, 0 means complete rejection. Wherein H is t-1 Representing the output of the last neuron, X t Representing the input of the current neuron. i.e t C t An input gate is formed, an output gate determines how much new information to be added to the network, i t Representative of which information needs to be updated, o t H t An output gate is formed to determine what value is output last, sigma represents a Sigmoid function, W is a corresponding coefficient, and b is a deviation.
The loss function in step S11 is:
wherein,,
a row of subpoint 3DEPI volume blocks, V, which are fixed light field image t-axis M True value of 3D mask volume block of snowflake or rain band of virtual scene, F m Is a detector of the present invention, using the L2 loss as a loss function. L (L) M The 3D EPI volume representing the generated snowflake or rain belt distribution mask and the L2 distance, L, of the 3D EPI of the real snowflake or rain belt distribution mask M The smaller the representation the closer the generated result is to the true value.
In the embodiment of the present application, referring to fig. 3, the step S13 may specifically include:
step S131, dividing the 3D U type network into an encoder network and a decoder network according to the operation principle; for example, the encoder may be located on the left and the decoder network may be located on the right.
Step S132, coding and decoding snowflake or rain belt data by adopting a coder network and a decoder network respectively; each convolution layer of the encoding section reduces the width and height of the 3D EPI volume by one half, the number of channels is doubled, and the decoder section restores the feature map.
And step S133, adopting pooling layer downsampling, and setting an output function to obtain a second light field image. For example, each layer of convolution kernel is set to a size of 3 x 3, downsampling by the pooling layer, the excitation function is Relu, the excitation function of the last layer is Sigmoid, and obtaining a second light field image with the value range of 0 to 1, and adding batch standardization operation and a long-short-time memory network behind each convolution layer in the same way. In step S13, the loss function is:
wherein the method comprises the steps of
3DEPI volume block added with snowflake or rain-band data distribution mask, V T Is the 3D snowless EPI volume block truth value for a virtual scene, fs is the generator of the present invention, using the L2 penalty as the penalty function. L (L) S Represents the volume of the generated snowless 3D EPI and the L2 distance of the real snowless 3D EPI, L S The smaller the representation the closer the generated result is to the true value. According to the scheme, the authenticity of the generated snow-free image is judged by utilizing the judging network, so that the aim of obtaining a more real snow-free image is fulfilled.
In the embodiment of the application, the discriminator is used for judging the authenticity of the snow-free image, and the adopted function model is as follows:
wherein V is T Is the true value of the 3D snowless EPI volume block of the virtual scene,
and (3) generating a snowless 3D EPI volume block, wherein D in the formula (3) is a discriminator and related parameters thereof, and G is a generator and related parameters thereof.
Once a satisfactory detector, generator and discriminator are obtained through learning training, in actual testing, only the steps performed by the detector and generator are required to be performed, so that a second light field image without snow (rain streaks) as a target can be obtained from the first light field image. The arbiter need not be executed.
The following is an illustration of one specific embodiment:
firstly, constructing a virtual light field camera by using a blender to render a virtual scene data set, dividing light field data into 256 multiplied by 256 to obtain a final virtual scene light field data set, and collecting a first light field image; continuously acquiring a real scene light field data set by using a light field camera to obtain a third light field image;
then, the first light field image is processed as a stack of 3D EPI volume blocks, four dimensional parameters (depth, weight, height, channel) of the volume blocks are obtained, where depth is the number of sub-views of a row of light field fixed t, set to 9, weight is the width of the sub-view image, set to 256, height is the height of the sub-view image, set to 256, channel is the number of channels of the image, here RGB three channels.
Continuing, performing convolution operation on the light field data in the four-dimensional data form, performing batch standardization, and reinforcing the relation between adjacent viewpoints through a long-short-term memory network to obtain a mask for snowflake or rain-band data distribution in the first light field image; and adding the obtained mask to the original image 3DEPI volume block through connection operation, reducing the width and the height of the first light field image to be one half of the original one through convolution of each layer of the coding part, increasing the channel number to be two times of the original one, restoring the first light field image by the decoder part, finally obtaining a second light field image with the value range of 0 to 1, and finally evaluating the second light field image.
According to the deep learning-based light field image snow or rain belt detection and removal system provided by the embodiment of the invention, snow or rain belt data of a first light field image are directly detected, the snow or rain belt data are detected and removed by utilizing the characteristic that the light field image has abundant three-dimensional structure information, and the snow or rain belt data removal effect is relatively good; according to the scheme, the snow or rain belt data are removed by adopting a deep learning self-learning method, intermediate parameters are not required to be estimated, and the snow or rain belt data removal from end to end can be directly realized by adopting a light field image snow or rain belt detection and removal system based on deep learning, so that a better snow or rain belt removal effect is achieved; meanwhile, in the detector and the generator, batch standardization operation and a long-short-time memory network are added behind each convolution layer, and the batch standardization layer is used for regularizing the model, accelerating the training of the model and avoiding gradient disappearance; the long-short-term memory network is used for enhancing the relation between adjacent view points, and the shielded snowflake or rain zone area is better processed through the information of the adjacent view points, so that the texture information with more complex background can be recovered.
Fig. 4 is a schematic structural diagram of a detection and removal system for snow or rain strips of a light field image based on deep learning according to an embodiment of the present invention. The system as shown in fig. 4 may include:
the detection module 41 is configured to detect a position and a size of a snowflake or a rain belt in the acquired first light field image by using the 3D residual error network;
the removing module 42 is configured to remove detected snowflakes or rain bands in the first light field image by using a 3D U type network, so as to obtain a second light field image;
the judging module 43 is configured to obtain a second light field image and a third light field image, and optimize an objective function of the discriminator by judging whether the second light field image and the third light field image are true or false, where the third light field image is a light field image without snowflake or rain band under the same scene as the first light field image, and is also called as a true value of the second light field image;
an evaluation module 44, configured to evaluate the quality of the generated second light field image by using a peak signal-to-noise ratio (PSNR) and a Structural Similarity (SSIM), and if the evaluation result does not meet the requirement, execute steps S11, S13, S15 again until the evaluation result meets the requirement;
the detecting, by using a 3D residual error network, a position and a size of a snowflake or a rain belt in the obtained first light field image includes: obtaining a mask of snowflake or raindrop data of the first light field image by using a 3D residual error network;
the masking of snowflake or rain data of the first light field image by using the 3D residual error network comprises the following steps: setting a residual block according to a 3D residual network; setting parameters of a first convolution layer and a second convolution layer, wherein each residual block uses the first convolution layer and the second convolution layer respectively; after the first convolution layer and the second convolution layer are used, adding standardized operation and a long-short-time memory network to obtain a mask of snowflake or raindrop data of the first light field image;
the step of removing the detected snow or rain belt in the first light field image by adopting the 3D U type network to obtain a second light field image comprises the following steps: removing snow or rain bands in the first light field image by using a 3D U type network according to the mask to obtain a second light field image;
removing snow or rain bands in the first light field image by using a 3D U type network to obtain a second light field image, wherein the method comprises the following steps of: dividing a type 3D U network into an encoder network and a decoder network according to the position relation; encoding and decoding snowflake or rain belt data by adopting an encoder network and a decoder network respectively; and adopting pooling layer downsampling, and setting an output function to obtain a second light field image.
The detection and removal system for the snow or rain belt of the light field image based on the deep learning provided by the embodiment of the invention can realize each process in the method embodiments of fig. 1 to 3, and in order to avoid repetition, the description is omitted here.
According to the deep learning-based light field image snow or rain belt detection and removal system provided by the embodiment of the invention, snow or rain belt data of a first light field image are directly detected, the snow or rain belt data are detected and removed by utilizing the characteristic that the light field image has abundant three-dimensional structure information, and the snow or rain belt data removal effect is relatively good; according to the scheme, the snow or rain belt data are removed by adopting a deep learning self-learning method, intermediate parameters are not required to be estimated, and the snow or rain belt data removal from end to end can be directly realized by adopting a light field image snow or rain belt detection and removal system based on deep learning, so that a better snow or rain belt removal effect is achieved; meanwhile, in the detector and the generator, batch standardization operation and a long-short-time memory network are added behind each convolution layer, and the batch standardization layer is used for regularizing the model, accelerating the training of the model and avoiding gradient disappearance; the long-short-term memory network is used for enhancing the relation between adjacent view points, and the shielded snowflake or rain zone area is better processed through the information of the adjacent view points, so that the texture information with more complex background can be recovered.
Corresponding to the method and system for detecting and removing light field image snowflakes or raindrops based on deep learning provided by the embodiment of the present invention, the embodiment of the present invention provides a device for detecting and removing light field image snowflakes or raindrops based on deep learning, as shown in fig. 5, including a detector 51, a generator 52, a discriminator 53 and an evaluator 54, wherein:
a detector 51 for detecting the position and size of snow or rain in the first acquired light field image by using the 3D residual error network; the detector can obtain a mask of snowflake or rain data of the first light field image by using the 3D residual error network;
the generator 52 adopts a 3D U type network to remove snow or rain bands in the detected first light field image to obtain a second light field image; the generator can take snowflake or rain strip data mask output by the detector as one input and the first light field image as the other input, the generator network can be divided into a left encoder network and a right decoder network, each layer of convolution of the encoder network reduces the width and the height of the first light field image to be one half of the original, the number of channels is increased to be two times of the original, and the decoder part restores the first light field image. The convolution kernel size of each convolution layer is 3 multiplied by 3, the excitation function is Relu through the downsampling of the pooling layer, the excitation function of the last layer is Sigmoid, and the snowflake-free or rain-band image with the value range of 0 to 1 is obtained, and similarly, batch standardization operation and a long-time and short-time memory network are added behind each convolution layer.
A discriminator 53 that acquires the second light field image and the third light field image, and optimizes an objective function of the discriminator by judging whether the second light field image and the third light field image are true or false;
the evaluator 54 evaluates the quality of the generated second light field image through the peak signal-to-noise ratio and the structural similarity, and if the evaluation result does not meet the requirement, the steps S11, S13, S15 are executed again until the evaluation result meets the requirement;
in the embodiment of the invention, the discriminator and the generator can be trained alternately until the discriminator cannot distinguish between the generated snowless pattern and the real snowless pattern; the third light field image is a light field image without snowflakes or rain bands in the same scene with the first light field image.
In an embodiment of the present invention, the detecting, by using a 3D residual network, a position and a size of a snowflake or a rain belt in an acquired first light field image includes: obtaining a mask of snowflake or raindrop data of the first light field image by using a 3D residual error network;
the masking of snowflake or rain data of the first light field image by using the 3D residual error network comprises the following steps: setting a residual block according to a 3D residual network; setting parameters of a first convolution layer and a second convolution layer, wherein each residual block uses the first convolution layer and the second convolution layer respectively; after the first convolution layer and the second convolution layer are used, adding standardized operation and a long-short-time memory network to obtain a mask of snowflake or raindrop data of the first light field image;
the step of removing the detected snow or rain belt in the first light field image by adopting the 3D U type network to obtain a second light field image comprises the following steps: removing snow or rain bands in the first light field image by using a 3D U type network according to the mask to obtain a second light field image;
removing snow or rain bands in the first light field image by using a 3D U type network to obtain a second light field image, wherein the method comprises the following steps of: dividing a type 3D U network into an encoder network and a decoder network according to the position relation; encoding and decoding snowflake or rain belt data by adopting an encoder network and a decoder network respectively; and adopting pooling layer downsampling, and setting an output function to obtain a second light field image.
The arbiter may comprise 6 convolution layers, with a convolution kernel of 3 x 3, the last layer using a fully connected layer to compare the generated output with the original snow-free image, evaluating the detector-generator network for quality, and thus making corresponding adjustments to the network parameters. The excitation function is set to Relu and the excitation function of the last layer is Sigmoid.
The loss function of the arbiter is:
where D is the arbiter and G is the generator.
The device for detecting and removing the snow or rain belt of the light field image based on the deep learning provided by the embodiment of the invention has the following advantages:
(1) According to the invention, the light field image is used as input, and the snow or rain belt is detected and removed by utilizing the characteristic that the light field image has abundant three-dimensional structure information, so that the effect is better than the snow or rain belt removal effect of a single image.
(2) The invention constructs a network model composed of the detector, the generator and the discriminator based on deep learning, directly converts the image with snowflakes or rain bands into the image without snowflakes or rain bands through the network model, realizes the end-to-end image snowflakes or rain bands removal without estimating intermediate parameters, and can obtain good snowflakes or rain bands removal effect.
(3) A batch of standardized operation and a long-short-time memory network are added behind each convolution layer of the detector and the generator, and the batch of standardized layers are used for regularizing the model, accelerating the training of the model and avoiding gradient disappearance; the long-short-term memory network is used for enhancing the relation between adjacent view points, and the shielded snowflake or rain zone area is better processed through the information of the adjacent view points, so that the texture information with more complex background can be recovered.
(4) The discriminator compares the generated image part with the original snow-free image to judge the authenticity of the output image, thereby being beneficial to optimizing the network model and generating a more real target image.
The method and the system for detecting and removing the snow or the rain belt of the light field image based on the deep learning provided by the embodiment of the invention are not limited to being only used for detecting and removing the snow or the rain belt in the light field image, and can be also used for detecting and removing the light field image containing attachments in the capturing process. Therefore, the technical solution of only changing the object attached to the light field image obtained in the capturing process still belongs to the protection scope of the present invention without any creative effort.
The embodiment of the invention also provides a computer readable storage medium, on which a computer program is stored, which when executed by a processor, implements the processes of the above method embodiment, and can achieve the same technical effects, and in order to avoid repetition, the description is omitted here. Wherein the computer readable storage medium is selected from Read-Only Memory (ROM), random access Memory (Random Access Memory, RAM), magnetic disk or optical disk.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising instructions for causing a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method according to the embodiments of the present invention.
The embodiments of the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those having ordinary skill in the art without departing from the spirit of the present invention and the scope of the claims, which are to be protected by the present invention.
Claims (6)
1. A method for detecting and removing snow or rain belt of a light field image based on deep learning is characterized by comprising the following steps:
step S11, a detector detects the position and the size of snowflakes or rain strips in the acquired first light field image by using a 3D residual error network;
step S13, a generator adopts a 3D U type network, and snow or rain strips in the detected first light field image are removed through self-learning to obtain a second light field image;
step S15, the discriminator acquires a second light field image and a third light field image, and optimizes an objective function of the discriminator by judging the authenticity of the second light field image and the third light field image, wherein the third light field image is a light field image without snowflake or rain band under the same scene with the first light field image;
step S17, evaluating the quality of the generated second light field image through peak signal-to-noise ratio and structural similarity, and if the evaluation result does not meet the requirement, executing the steps S11, S13 and S15 again until the evaluation result meets the requirement;
the detecting, by using a 3D residual error network, a position and a size of a snowflake or a rain belt in the obtained first light field image includes: obtaining a mask of snowflake or raindrop data of the first light field image by using a 3D residual error network;
the masking of snowflake or rain data of the first light field image by using the 3D residual error network comprises the following steps: setting a residual block according to a 3D residual network; setting parameters of a first convolution layer and a second convolution layer, wherein each residual block uses the first convolution layer and the second convolution layer respectively; after the first convolution layer and the second convolution layer are used, adding standardized operation and a long-short-time memory network to obtain a mask of snowflake or raindrop data of the first light field image;
the step of removing the detected snow or rain belt in the first light field image by adopting the 3D U type network to obtain a second light field image comprises the following steps: removing snow or rain bands in the first light field image by using a 3D U type network according to the mask to obtain a second light field image;
removing snow or rain bands in the first light field image by using a 3D U type network to obtain a second light field image, wherein the method comprises the following steps of: dividing a type 3D U network into an encoder network and a decoder network according to the position relation; encoding and decoding snowflake or rain belt data by adopting an encoder network and a decoder network respectively; and adopting pooling layer downsampling, and setting an output function to obtain a second light field image.
2. The method according to claim 1, characterized in that prior to said step S11, it further comprises:
the first light field image and the third light field image are acquired.
3. The method of claim 1, wherein the objective function of the arbiter is:
wherein V is T Is the true value of the 3D snowless EPI volume block of the virtual scene,
and (3) generating a snowless 3DEPI volume block, wherein D in the formula is a discriminator and related parameters thereof, and G is a generator and related parameters thereof.
4. A deep learning based light field image snowflake or rain belt detection and removal system, comprising:
the detection module is used for detecting the position and the size of snowflakes or rain belts in the acquired first light field image by utilizing the 3D residual error network;
the removing module is used for removing snow or rain strips in the detected first light field image by adopting a 3D U type network through self-learning by the generator to obtain a second light field image;
the judging module is used for acquiring a second light field image and a third light field image, optimizing an objective function of the judging device by judging the authenticity of the second light field image and the third light field image, wherein the third light field image is a light field image without snowflake or rain belt under the same scene with the first light field image;
the evaluation module evaluates the quality of the generated second light field image through peak signal-to-noise ratio and structural similarity, and if the evaluation result does not meet the requirement, the steps S11, S13 and S15 are executed again until the evaluation result meets the requirement;
the detecting, by using a 3D residual error network, a position and a size of a snowflake or a rain belt in the obtained first light field image includes: obtaining a mask of snowflake or raindrop data of the first light field image by using a 3D residual error network;
the masking of snowflake or rain data of the first light field image by using the 3D residual error network comprises the following steps: setting a residual block according to a 3D residual network; setting parameters of a first convolution layer and a second convolution layer, wherein each residual block uses the first convolution layer and the second convolution layer respectively; after the first convolution layer and the second convolution layer are used, adding standardized operation and a long-short-time memory network to obtain a mask of snowflake or raindrop data of the first light field image;
the step of removing the detected snow or rain belt in the first light field image by adopting the 3D U type network to obtain a second light field image comprises the following steps: removing snow or rain bands in the first light field image by using a 3D U type network according to the mask to obtain a second light field image;
removing snow or rain bands in the first light field image by using a 3D U type network to obtain a second light field image, wherein the method comprises the following steps of: dividing a type 3D U network into an encoder network and a decoder network according to the position relation; encoding and decoding snowflake or rain belt data by adopting an encoder network and a decoder network respectively; and adopting pooling layer downsampling, and setting an output function to obtain a second light field image.
5. A deep learning based light field image snowflake or rain belt detection and removal device, comprising:
the detector is used for detecting the position and the size of snowflakes or rain belts in the acquired first light field image by using the 3D residual error network;
the generator adopts a 3D U type network to remove snow or rain strips in the detected first light field image to obtain a second light field image;
the second light field image and the third light field image are acquired, the authenticity of the second light field image and the third light field image is judged to be distinguished to optimize the objective function of the discriminator, and the third light field image is a light field image without snowflake or rain belt under the same scene with the first light field image;
the evaluator evaluates the quality of the generated second light field image through peak signal-to-noise ratio and structural similarity, and if the evaluation result does not meet the requirement, the detector, the generator and the discriminator are controlled to work again until the evaluation result meets the requirement;
the detecting, by using a 3D residual error network, a position and a size of a snowflake or a rain belt in the obtained first light field image includes: obtaining a mask of snowflake or raindrop data of the first light field image by using a 3D residual error network;
the masking of snowflake or rain data of the first light field image by using the 3D residual error network comprises the following steps: setting a residual block according to a 3D residual network; setting parameters of a first convolution layer and a second convolution layer, wherein each residual block uses the first convolution layer and the second convolution layer respectively; after the first convolution layer and the second convolution layer are used, adding standardized operation and a long-short-time memory network to obtain a mask of snowflake or raindrop data of the first light field image;
the step of removing the detected snow or rain belt in the first light field image by adopting the 3D U type network to obtain a second light field image comprises the following steps: removing snow or rain bands in the first light field image by using a 3D U type network according to the mask to obtain a second light field image;
removing snow or rain bands in the first light field image by using a 3D U type network to obtain a second light field image, wherein the method comprises the following steps of: dividing a type 3D U network into an encoder network and a decoder network according to the position relation; encoding and decoding snowflake or rain belt data by adopting an encoder network and a decoder network respectively; and adopting pooling layer downsampling, and setting an output function to obtain a second light field image.
6. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, implements the steps of the method according to any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010243333.2A CN111445465B (en) | 2020-03-31 | 2020-03-31 | Method and equipment for detecting and removing snow or rain belt of light field image based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010243333.2A CN111445465B (en) | 2020-03-31 | 2020-03-31 | Method and equipment for detecting and removing snow or rain belt of light field image based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111445465A CN111445465A (en) | 2020-07-24 |
| CN111445465B true CN111445465B (en) | 2023-06-16 |
Family
ID=71652575
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010243333.2A Active CN111445465B (en) | 2020-03-31 | 2020-03-31 | Method and equipment for detecting and removing snow or rain belt of light field image based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111445465B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112288738B (en) * | 2020-11-18 | 2022-07-29 | 新疆大学 | Single image snowflake removing method and device based on deep learning and storage medium |
| CN112767274A (en) * | 2021-01-25 | 2021-05-07 | 江南大学 | Light field image rain stripe detection and removal method based on transfer learning |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107993260A (en) * | 2017-12-14 | 2018-05-04 | 浙江工商大学 | A kind of light field image depth estimation method based on mixed type convolutional neural networks |
| CN109102475B (en) * | 2018-08-13 | 2021-03-09 | 苏州飞搜科技有限公司 | Image rain removing method and device |
| CN109344818B (en) * | 2018-09-28 | 2020-04-14 | 合肥工业大学 | Light field significant target detection method based on deep convolutional network |
| KR20190142216A (en) * | 2019-06-07 | 2019-12-26 | 케이에스아이 주식회사 | Vehicle sensing system using deep learning recognition of marking |
| CN110728640B (en) * | 2019-10-12 | 2023-07-18 | 合肥工业大学 | Fine rain removing method for double-channel single image |
-
2020
- 2020-03-31 CN CN202010243333.2A patent/CN111445465B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN111445465A (en) | 2020-07-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111754446A (en) | Image fusion method, system and storage medium based on generation countermeasure network | |
| CN111709895A (en) | Image blind deblurring method and system based on attention mechanism | |
| CN111160249A (en) | Multi-class target detection method of optical remote sensing image based on cross-scale feature fusion | |
| CN113312983B (en) | Semantic segmentation method, system, device and medium based on multi-mode data fusion | |
| CN111445465B (en) | Method and equipment for detecting and removing snow or rain belt of light field image based on deep learning | |
| CN113781510B (en) | Edge detection method and device and electronic equipment | |
| CN106447640B (en) | Multi-focus image fusing method and device based on dictionary learning, rotation guiding filtering | |
| CN111260775B (en) | Three-dimensional reconstruction method and device based on multi-scale perception of occlusion information | |
| CN111507426A (en) | No-reference image quality grading evaluation method and device based on visual fusion characteristics | |
| Chen et al. | Multi-scale adaptive dehazing network | |
| CN110728640A (en) | Double-channel single-image fine rain removing method | |
| CN110866926A (en) | Infrared remote sensing image rapid and fine sea-land segmentation method | |
| CN112348762A (en) | Single image rain removing method for generating confrontation network based on multi-scale fusion | |
| CN116279592A (en) | Method for dividing travelable area of unmanned logistics vehicle | |
| Wang et al. | An efficient method for image dehazing | |
| CN116309178A (en) | Visible light image denoising method based on self-adaptive attention mechanism network | |
| CN115909268A (en) | Dynamic obstacle detection method and device | |
| CN115439738A (en) | Underwater target detection method based on self-supervision cooperative reconstruction | |
| Wang et al. | Haze removal algorithm based on single-images with chromatic properties | |
| CN115239672A (en) | Defect detection method and device, equipment and storage medium | |
| Saleem et al. | A non-reference evaluation of underwater image enhancement methods using a new underwater image dataset | |
| Babu et al. | An efficient image dahazing using Googlenet based convolution neural networks | |
| CN109064444B (en) | Track slab disease detection method based on significance analysis | |
| CN114462486A (en) | Training method of image processing model, image processing method and related device | |
| CN117036442A (en) | Robust monocular depth completion method, system and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |