CN111445465A - Light field image snowflake or rain strip detection and removal method and device based on deep learning - Google Patents
Light field image snowflake or rain strip detection and removal method and device based on deep learning Download PDFInfo
- Publication number
- CN111445465A CN111445465A CN202010243333.2A CN202010243333A CN111445465A CN 111445465 A CN111445465 A CN 111445465A CN 202010243333 A CN202010243333 A CN 202010243333A CN 111445465 A CN111445465 A CN 111445465A
- Authority
- CN
- China
- Prior art keywords
- light field
- field image
- rain
- snowflakes
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 241000533950 Leucojum Species 0.000 title claims abstract description 98
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000013135 deep learning Methods 0.000 title claims abstract description 33
- 238000001514 detection method Methods 0.000 title claims description 21
- 230000006870 function Effects 0.000 claims abstract description 28
- 238000011156 evaluation Methods 0.000 claims abstract description 26
- 230000015654 memory Effects 0.000 claims description 15
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 6
- 238000011176 pooling Methods 0.000 claims description 4
- 238000005070 sampling Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 15
- 238000003062 neural network model Methods 0.000 abstract 1
- 238000012545 processing Methods 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 6
- 230000005284 excitation Effects 0.000 description 6
- 238000012549 training Methods 0.000 description 6
- 230000008034 disappearance Effects 0.000 description 5
- 230000007787 long-term memory Effects 0.000 description 5
- 238000005728 strengthening Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 3
- 210000002569 neuron Anatomy 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000003014 reinforcing effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
- G06T7/62—Analysis of geometric attributes of area, perimeter, diameter or volume
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Quality & Reliability (AREA)
- Health & Medical Sciences (AREA)
- Geometry (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a method and a system for detecting and removing snowflakes or rain belts of light field images based on deep learning, wherein the method comprises the following steps: detecting the position and the size of the snowflake or the rain strip in the acquired first light field image by using a 3D residual error network; removing the detected snowflakes or rain belts by adopting a 3D U type network to obtain a second light field image; acquiring a second light field image and a third light field image, and optimizing a discriminator and an objective function by judging the authenticity of the second light field image and the third light field image; evaluating the quality of the generated second light field image through the peak signal-to-noise ratio and the structural similarity index, and if the evaluation result does not meet the requirement, executing the steps S11, S13 and S15 again until the evaluation result meets the requirement; the scheme directly detects the data of the snowflakes or the rain strips, detects and removes the snowflakes or the rain strips by utilizing the characteristic that the light field image has rich three-dimensional structure information, automatically and iteratively updates the intermediate implicit parameters of the neural network model, and has good snowflake or rain strip removing effect.
Description
Technical Field
The invention relates to the technical field of computer image processing, in particular to a method and equipment for detecting and removing snowflakes or rain strips of a light field image based on deep learning.
Background
With the tremendous development of the field of computer science under the support of high-precision technology, the research in the field of computer vision also gets a great progress. In the field of computer vision, the light field imaging technology is widely applied. Compared with the traditional camera, the light field camera can acquire four-dimensional information of a scene, including two-dimensional space information and two-dimensional angle information, through single exposure, so that richer image information can be acquired in the image reconstruction process, and in addition, the problems of out-of-focus of images, excessive background targets and the like in special occasions can be solved through a digital refocusing technology; realizing 'perspective' monitoring by a synthetic aperture technology; after the method is fused with the microscopic technology, a multi-view large-depth-of-field microscopic image and a reconstructed three-dimensional stereogram can be obtained. With the intensive research on light field technology, light field image processing is gradually concerned by experts and scholars at home and abroad, and is used as an important technical means for three-dimensional scene perception in the industry represented by automatic driving. The research of the light field image processing focuses on aspects of depth estimation, super-resolution, image restoration and the like.
In the field of computer vision, due to the complexity of illumination, shading and the shape and color of the snowflake or the rain belt, a snowflake or rain belt removing algorithm on a single image can only detect and remove the snowflake or the rain belt according to the color and shape information in the image, so that the method has great limitation and poor snowflake or rain belt removing effect.
In summary, a scheme for removing snow or rain belts with a good snow or rain belt removing effect is lacked in the prior art.
Disclosure of Invention
The embodiment of the invention provides a method and a system for detecting and removing snowflakes or rain belts in a light field image based on deep learning, and aims to solve the technical problem that a scheme for removing the snowflakes or the rain belts with a good snowflakes or rain belt removing effect is lacked in the prior art.
In a first aspect, a method for detecting and removing snowflakes or rain strips in a light field image based on deep learning is provided according to an embodiment of the present application, and includes:
s11, detecting the position and the size of the snowflake or the rain belt in the acquired first light field image by the detector by using a 3D residual error network;
s13, removing the detected snowflakes or rain belts in the first light field image by self-learning by using a 3D U type network by a generator to obtain a second light field image;
step S15, the discriminator acquires the second light field image and the third light field image, and optimizes the objective function of the discriminator by judging and distinguishing the authenticity of the second light field image and the third light field image;
step S17, evaluating the quality of the generated second light field image through the peak signal-to-noise ratio and the structural similarity, and if the evaluation result does not meet the requirement, executing the steps S11, S13 and S15 again until the evaluation result meets the requirement;
wherein the third light field image is a light field image without snowflakes or rain strips in the same scene as the first light field image, and is also called a true value of the second light field image.
In one embodiment, before the step S11, the method further includes:
the first light field image and the third light field image are acquired.
In one embodiment, the detecting the position and size of the snowflakes or rain strips in the acquired first light field image by using the 3D residual network includes:
and obtaining a mask of the snow or rain strip data of the first light field image by using the 3D residual error network.
In one embodiment, the obtaining a mask of snowflake or rainstrip data of the first light-field image using the 3D residual network comprises:
setting a residual block according to the 3D residual network;
setting parameters of a first convolution layer and a second convolution layer, wherein each residual block uses the first convolution layer and the second convolution layer respectively;
after the first convolution layer and the second convolution layer are used, a standardization operation and a long-time and short-time memory network are added, and a mask of snow or rain strip data of the first light field image is obtained.
In one embodiment, the removing the detected snowflakes or rain strips from the first light-field image by using a 3D U type network to obtain a second light-field image includes:
from the mask, snow or rain flakes in the first light field image are removed using a 3D U type network, resulting in a second light field image.
In one embodiment, said removing snowflakes or rain strips from the first light field image using a 3D U type network to obtain a second light field image comprises:
dividing a 3D U type network into an encoder network and a decoder network according to the position relation;
respectively adopting an encoder network and a decoder network to encode and decode snowflake or rain belt data;
and adopting pooling layer down-sampling to set an output function to obtain a second light field image.
In one embodiment, the objective function of the discriminator is:
wherein, VTIs the true value of the 3D snow-free EPI volume block for the virtual scene,
for the 3DEPI volume block with the added snowflake or raindrop data distribution mask, D represents the discriminator and G represents the generator.
In a second aspect, a snow or rain strip detection and removal system for a light field image based on deep learning is provided according to an embodiment of the present invention, including:
the detection module is used for detecting the position and the size of the snowflakes or the rain strips in the acquired first light field image by using the 3D residual error network;
the removing module is used for removing the detected snowflakes or rain belts in the first light field image by the self-learning by adopting a 3D U type network to obtain a second light field image;
the discrimination module is used for acquiring the second light field image and the third light field image and optimizing the objective function of the discriminator by judging and distinguishing the authenticity of the second light field image and the third light field image;
the evaluation module is used for evaluating the quality of the generated second light field image through the peak signal-to-noise ratio and the structural similarity, and if the evaluation result does not meet the requirement, the steps S11, S13 and S15 are executed again until the evaluation result meets the requirement;
and the third light field image is a light field image without snowflakes or rain strips under the same scene as the first light field image.
In a third aspect, a light field image snow or rain strip detection and removal device based on deep learning is provided according to an embodiment of the present invention, including:
the detector is used for detecting the position and the size of the snowflakes or the rain strips in the acquired first light field image by using the 3D residual error network;
the generator is used for removing the detected snowflakes or rain belts in the first light field image by adopting a 3D U type network to obtain a second light field image;
the discriminator acquires the second light field image and the third light field image, and optimizes the objective function of the discriminator by judging and distinguishing the truth of the second light field image and the third light field image;
the evaluator evaluates the quality of the generated second light field image through the peak signal-to-noise ratio and the structural similarity, and controls the detector, the generator and the discriminator to work again if the evaluation result does not meet the requirement until the evaluation result meets the requirement;
and the third light field image is a light field image without snowflakes or rain strips under the same scene as the first light field image.
In a fourth aspect, according to an embodiment of the invention, a computer-readable storage medium is provided, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.
According to the method and the system for detecting and removing the snowflakes or the rain belts of the light field image based on the deep learning, the snowflakes or the rain belts of the first light field image are directly detected, the snowflakes or the rain belts are detected and removed by utilizing the characteristic that the light field image has rich three-dimensional structure information, and the snowflakes or the rain belts are better in data removal effect; according to the scheme, the snow or rain belt data are removed by adopting a deep learning self-learning method, intermediate parameters do not need to be estimated, the end-to-end snow or rain belt data removal can be realized by directly adopting a deep learning-based light field image snow or rain belt detection and removal system, and the snow or rain belt removal effect is better; meanwhile, in the detector and the generator, after each convolution layer, a batch of standardized operation and a long-time and short-time memory network are added, and the batch of standardized layers are used for regularizing the model, accelerating the training of the model and avoiding gradient disappearance; the long-time memory network is used for strengthening the relation between adjacent viewpoints, and the information of the adjacent viewpoints is used for better processing the shielded snowflake or rain zone area, so that the more complex texture information of the background can be recovered.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention and not to limit the invention. In the drawings:
FIG. 1 is a flow chart of a deep learning-based light field image snowflake or rainstrip detection and removal method according to an embodiment of the present invention;
FIG. 2 is a flow chart of another method for detecting and removing snow or rain flakes from a light field image based on deep learning according to an embodiment of the present invention;
FIG. 3 is a flowchart of a snow or rain strip detection and removal method for a light field image based on deep learning according to another embodiment of the present invention;
FIG. 4 is a schematic structural diagram of a snow or rain strip detection and removal system for a light field image based on deep learning according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a snow or rain strip detection and removal device for a light field image based on deep learning according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In a first aspect, a method for detecting and removing snowflakes or rain strips in a light field image based on deep learning is provided according to an embodiment of the present invention, as shown in fig. 1, which may include the following steps:
s11, detecting the position and the size of the snowflake or the rain belt in the acquired first light field image by the detector by using a 3D residual error network;
in an embodiment of the present application, the first light field image may be a light field image acquired by a light field camera under a light field of a virtual scene. And detecting the position and the size of the snowflakes or the rain strips in the acquired first light field image, wherein two continuous convolutions can be used, and batch standardization operation and a long-time and short-time memory network are added after each convolution, so that the position and the size of the snowflakes or the rain strips are detected.
S13, removing the detected snowflakes or rain belts in the first light field image by the generator by adopting a 3D U type network to obtain a second light field image;
in the present embodiment, the snow or rain flakes detected in step S11 are removed, and a 3D U type network is used, and similarly two successive convolutions are used, with a batch of normalization operations and a long-term memory network added after each convolution.
Step S15, the discriminator acquires the second light field image and the third light field image, and optimizes the objective function of the discriminator by judging and distinguishing the authenticity of the second light field image and the third light field image;
in this embodiment, the step of distinguishing the authenticity of the second light field image and the third light field image may adopt a plurality of convolution layers, and the generated output is compared with the third light field image at the last convolution layer to obtain the corresponding output parameter.
Step S17, evaluating the quality of the generated second light field image through the peak signal-to-noise ratio and the structural similarity, and if the evaluation result does not meet the requirement, executing the steps S11, S13 and S15 again until the evaluation result meets the requirement;
the third light field image is a light field image without snowflakes or rain strips in the same scene as the first light field image, and can also be called a true value of the second light field image.
In the embodiment of the application, a self-learning depth algorithm is adopted, and if the evaluation result of the step S17 does not meet the preset requirement, the steps S11, S13 and S15 are automatically and repeatedly executed until the evaluation result meets the preset requirement.
In the embodiment of the application, a self-learning depth algorithm is adopted, after the discriminator acquires the second light field image and the third light field image, the authenticity of the second light field image and the third light field image can be discriminated, the generated snow-free image (the second light field image) and the real snow-free image (the third light field image) can respectively output a value from 0 to 1 through the discriminator, the closer the value is to 1, the closer the second light field image and the third light field image are, namely, the better the snow or rain strip removing effect is. The quality of the generated snow-free image is quantitatively evaluated using a peak signal-to-noise ratio (PSNR) and a Structural Similarity (SSIM), and if the evaluation result does not satisfy the requirement of the objective function, steps S11, S13, S15 are performed again.
In the embodiment of the present application, referring to fig. 2, before the step S11, the method further includes:
the method comprises the steps of S10, acquiring a first light field image and a third light field image, constructing a light field data set, constructing a virtual light field camera by using a blender to render a virtual scene data set, dividing the data into preset sizes (such as 256 × 256) to obtain a final virtual scene light field data set, acquiring a virtual scene light field image to obtain a first light field image, and acquiring a real scene light field data set by using a light field camera to obtain a third light field image.
In this embodiment of the application, in step S11, the detecting, by using a 3D residual network, the position and size of the snowflakes or the rain strips in the acquired first light field image includes:
and obtaining a mask of the snow or rain strip data of the first light field image by using the 3D residual error network. The method specifically comprises the following steps:
1) setting a residual block according to the 3D residual network;
2) setting parameters of a first convolution layer and a second convolution layer, wherein each residual block uses the first convolution layer and the second convolution layer respectively;
after the first convolution layer and the second convolution layer are used, batch standardization operation and a long-time and short-time memory network are added, and a mask of snow or rain belt data of the first light field image is obtained.
Specifically, the residual block can be set according to preset requirements, then each residual block of the 3D residual network is set to use two continuous convolution layers, the convolution kernel size of each convolution layer can be set to be 3 × 3 × 3, the excitation function of the first convolution layer is set to be Relu, and the excitation function of the second convolution layer is set to be Sigmoid, and a standardized operation and a long-term memory network are added behind the first convolution layer and the second convolution layer respectively, wherein the standardized operation is used for regularizing the model, training of the model is accelerated, and gradient disappearance is avoided, the long-term memory network is used for reinforcing the relation between adjacent viewpoints, for an image, a common long-term memory network is difficult to extract corresponding features, because the image is three-dimensional, and the convolutional operation can just make up for the shortages of the long-term memory network, and hidden snow or rain belt regions are better processed through the information of the adjacent viewpoints, so that texture information with a more complex background can be recovered, and the formula is as follows:
wherein, represents the convolution of the data,
which means multiplication of corresponding elements of the matrix, where i, f, C, o, H, X are all three-dimensional tensors. X is the input, here representing a light field-line sub-viewpoint 3D EPI volume, f represents the forgetting gate, which reads Ht-1And XtOutputting a value between 0 and 1 to each of the at least one neuron states Ct-1The number in (1) indicates complete retention, and 0 indicates complete rejection. Wherein Ht-1Representing the output of the last neuron, XtThe input to the current neuron is indicated. i.e. itCtForming input gates, output gates determining how much new information to add to the network, itRepresenting a decision which information needs to be updated, otHtAnd forming an output gate, and determining what value is finally output, wherein sigma represents a Sigmoid function, W is a corresponding coefficient, and b is a deviation.
The loss function in step S11 is:
wherein,
is a row of sub-point 3DEPI volume blocks, V, of the t-axis of the fixed light field imageMIs a true value, F, of a snowflake or rain strip 3D mask volume block of a virtual scenemIs the detector of the present invention, using L2 loss as a function of loss LML2 distance, L, of 3D EPI volume representing generated snowflake or rain strip distribution mask and 3D EPI of real snowflake or rain strip distribution maskMThe smaller the representation, the closer the generated result is to the true value.
In this embodiment, referring to fig. 3, the step S13 may specifically include:
step S131, dividing a 3D U type network into an encoder network and a decoder network according to an operation principle; for example, the encoder is located on the left and the decoder network may be located on the right.
S132, respectively adopting an encoder network and a decoder network to encode and decode snowflake or rain strip data; each convolutional layer of the encoding part reduces the width and height of the 3D EPI volume to half of the original, respectively, increases the number of channels to twice of the original, and the decoder part restores the feature map.
If the size of each convolution kernel is set to be 3 × 3 × 3, the second light field image with the value range of 0 to 1 is obtained by the pooling layer down-sampling, the excitation function is Relu, the excitation function of the last layer is Sigmoid, and similarly, a standardized operation and a long-time and short-time memory network are added behind each convolution layer, wherein in the step S13, the loss function is as follows:
wherein
3DEPI volume block V added with snowflake or rain strip data distribution maskTIs the true value of the 3D snow-free EPI volume for a virtual scene, Fs is the generator of the present invention, using L2 losses as the loss function LSRepresentsThe L2 distance, L, of the resulting snow-free 3DEPI volume and the true snow-free 3D EPI is shownSThe smaller the representation, the closer the generated result is to the true value. According to the scheme, the judgment network is used for judging the authenticity of the snow-free image generated by distinguishing so as to achieve the aim of obtaining a more real snow-free image.
In the embodiment of the present application, the discriminator is used for judging whether the snow-free image is true or false, and the adopted function model is as follows:
wherein, VTIs the true value of the 3D snow-free EPI volume block for the virtual scene,
the snow-free 3D EPI volume is generated, wherein D in formula (3) is the discriminator and relevant parameters thereof, and G is the generator and relevant parameters thereof.
Once a satisfactory detector, generator and discriminator are obtained through learning training, in an actual test, only the steps performed by the detector and the generator need to be performed, so that a snow-free (rainstreak-free) second light field image as a target can be obtained from the first light field image. The arbiter need not be executed any more.
As follows, a specific example is illustrated:
firstly, a virtual light field camera is constructed by using a blender to render a virtual scene data set, the light field data is divided into 256 × 256 sizes to obtain a final virtual scene light field data set, and a first light field image is acquired;
then, the first light field image is processed as a stack of 3D EPI volume blocks, and four-dimensional parameters (depth, weight, height, channel) of the volume blocks are obtained, where depth is the number of sub-viewpoints of one line of the light field fixed t, set to 9, weight is the width of the sub-viewpoint image, set to 256, height is the height of the sub-viewpoint image, set to 256, and channel is the number of channels of the image, here, RGB three channels.
Continuously carrying out batch standardization on the light field data in the four-dimensional data form after convolution operation, and then strengthening the relation between adjacent viewpoints through a long-time and short-time memory network to obtain a mask of snowflake or rain belt data distribution in the first light field image; and adding the obtained mask to the original image 3DEPI volume block through connection operation, reducing the width and height of the first light field image to one half of the original width and height by convolution of each layer of the coding part, increasing the number of channels to two times of the original width and height, restoring the first light field image by the decoder part to finally obtain a second light field image with a value range of 0 to 1, and finally evaluating the second light field image.
The light field image snowflake or rainstrip detection and removal system based on deep learning provided by the embodiment of the invention directly detects the snowflake or rainstrip data of the first light field image, detects and removes the snowflake or rainstrip data by utilizing the characteristic that the light field image has rich three-dimensional structure information, and has a good snowflake or rainstrip data removal effect; according to the scheme, the snow or rain belt data are removed by adopting a deep learning self-learning method, intermediate parameters do not need to be estimated, the end-to-end snow or rain belt data removal can be realized by directly adopting a deep learning-based light field image snow or rain belt detection and removal system, and the snow or rain belt removal effect is better; meanwhile, in the detector and the generator, after each convolution layer, a batch of standardized operation and a long-time and short-time memory network are added, and the batch of standardized layers are used for regularizing the model, accelerating the training of the model and avoiding gradient disappearance; the long-time memory network is used for strengthening the relation between adjacent viewpoints, and the information of the adjacent viewpoints is used for better processing the shielded snowflake or rain zone area, so that the more complex texture information of the background can be recovered.
Fig. 4 is a schematic structural diagram of a snow or rain strip detection and removal system for a light field image based on deep learning according to an embodiment of the present invention. The system as shown in fig. 4 may include:
a detection module 41, configured to detect a position and a size of a snowflake or a rain strip in the acquired first light field image by using a 3D residual network;
a removing module 42, configured to remove the detected snowflakes or rain strips in the first light field image by using a 3D U-type network, so as to obtain a second light field image;
a distinguishing module 43, configured to obtain the second light field image and the third light field image, and optimize an objective function of the discriminator by determining whether the second light field image and the third light field image are true or false
The evaluation module 44 evaluates the quality of the generated second light field image through peak signal-to-noise ratio (PSNR) and Structural Similarity (SSIM), and if the evaluation result does not meet the requirement, performs steps S11, S13, and S15 again until the evaluation result meets the requirement;
wherein the third light field image is a light field image without snowflakes or rain strips in the same scene as the first light field image, and is also called a true value of the second light field image.
The light field image snowflake or rain strip detection and removal system based on deep learning provided by the embodiment of the invention can realize each process in the method embodiments of fig. 1 to 3, and is not repeated here for avoiding repetition.
The light field image snowflake or raindrop detection and removal system based on deep learning provided by the embodiment of the invention directly detects the snowflake or raindrop data of the first light field image, detects and removes the snowflake or raindrop data by utilizing the characteristic that the light field image has rich three-dimensional structure information, and has a good snowflake or raindrop data removal effect; according to the scheme, the snow or rain belt data are removed by adopting a deep learning self-learning method, intermediate parameters do not need to be estimated, the end-to-end snow or rain belt data removal can be realized by directly adopting a deep learning-based light field image snow or rain belt detection and removal system, and the snow or rain belt removal effect is better; meanwhile, in the detector and the generator, after each convolution layer, a batch of standardized operation and a long-time and short-time memory network are added, and the batch of standardized layers are used for regularizing the model, accelerating the training of the model and avoiding gradient disappearance; the long-time memory network is used for strengthening the relation between adjacent viewpoints, and the information of the adjacent viewpoints is used for better processing the shielded snowflake or rain zone area, so that the more complex texture information of the background can be recovered.
Corresponding to the method and system for detecting and removing the snowflakes or the rain belts of the light field image based on the deep learning provided by the embodiment of the present invention, the embodiment of the present invention provides a device for detecting and removing the snowflakes or the rain belts of the light field image based on the deep learning, as shown in fig. 5, including a detector 51, a generator 52, a discriminator 53 and an evaluator 54, wherein:
a detector 51 for detecting the position and size of a snowflake or a rain strip in a first acquired light field image by using a 3D residual error network; the detector may use a 3D residual network to obtain a mask of snowflake or rainstrip data of the first light field image;
the generator 52 removes the detected snowflakes or rain bands in the first light field image by adopting a 3D U type network to obtain a second light field image, the generator can take a snowflake or rain band data mask output by the detector as one input, the first light field image as the other input, the generator network can be divided into a left encoder network and a right decoder network, the width and the height of the first light field image are reduced to one half of the original width and the height of the first light field image by convolution of each layer of the encoder network, the number of channels is increased to two times, the first light field image is reduced by the decoder part, the convolution kernel size of each layer is 3 × 3 × 3, the laminated image without the snowflakes or the rain bands is obtained by downsampling through layers of pooling, the excitation function is Relu, the excitation function of the last layer is Sigmoid, and the similar laminated image without the snowflakes or the rain bands with the value range of 0-1 is obtained, and batch standardization operation and the long-.
A discriminator 53 for acquiring the second light field image and the third light field image, and optimizing an objective function of the discriminator by judging whether the second light field image and the third light field image are true or false;
the evaluator 54 evaluates the quality of the generated second light field image through the peak signal-to-noise ratio and the structural similarity, and if the evaluation result does not meet the requirement, the steps S11, S13 and S15 are executed again until the evaluation result meets the requirement;
in the embodiment of the invention, the discriminator and the generator can be alternately trained until the discriminator can not distinguish the generated snow-free image from the real snow-free image; and the third light field image is a light field image without snowflakes or rain strips under the same scene as the first light field image.
The arbiter may contain 6 convolutional layers with convolutional kernels of 3 × 3 × 3, the last layer compares the resulting output with the original snow-free image using a fully connected layer, evaluates the detector-generator network for goodness and badness, and adjusts the network parameters accordingly.
The penalty function for the discriminator is:
where D is the discriminator and G is the generator.
The light field image snowflake or rain strip detection and removal device based on deep learning provided by the embodiment of the invention has the following advantages:
(1) the invention uses the light field image as input, detects and removes the snowflakes or the rain strips by using the characteristic that the light field image has abundant three-dimensional structure information, and has better effect than the snowflakes or the rain strips removing effect of a single image.
(2) The invention constructs a network model consisting of a detector, a generator and a discriminator based on deep learning, directly converts an image with snowflakes or a rain belt into an image without snowflakes or a rain belt through the network model, realizes end-to-end image snowflakes or rain belt removal, does not need to estimate intermediate parameters, and can obtain good snowflakes or rain belt removal effect.
(3) Adding batch standardization operation and a long-time memory network behind each convolution layer of the detector and the generator, wherein the batch standardization layer is used for regularizing the model, accelerating the training of the model and avoiding gradient disappearance; the long-time memory network is used for strengthening the relation between adjacent viewpoints, and the information of the adjacent viewpoints is used for better processing the shielded snowflake or rain zone area, so that the more complex texture information of the background can be recovered.
(4) The discriminator compares the generated image part with the original snow-free image to judge the authenticity of the output image, which is helpful for optimizing the network model and generating a more real target image.
It is noted that the method and system for detecting and removing snow or rain strips in a light field image based on deep learning provided by the embodiments of the present invention are not limited to be used only for detecting and removing snow or rain strips in a light field image, and can also be used for detecting and removing light field images containing attachments during capturing. Therefore, the technical solution of only changing the object attached to the light field image acquired during the capturing process without any creative labor still belongs to the protection scope of the present invention.
The embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program implements the processes of the method embodiments, and can achieve the same technical effects, and in order to avoid repetition, the details are not repeated here. The computer-readable storage medium may be a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.
While the present invention has been described with reference to the embodiments shown in the drawings, the present invention is not limited to the embodiments, which are illustrative and not restrictive, and it will be apparent to those skilled in the art that various changes and modifications can be made therein without departing from the spirit and scope of the invention as defined in the appended claims.
Claims (10)
1. A method for detecting and removing snowflakes or rain belts of light field images based on deep learning is characterized by comprising the following steps:
s11, detecting the position and the size of the snowflake or the rain belt in the acquired first light field image by the detector by using a 3D residual error network;
s13, removing the detected snowflakes or rain belts in the first light field image by self-learning by using a 3D U type network by a generator to obtain a second light field image;
step S15, the discriminator acquires the second light field image and the third light field image, and optimizes the objective function of the discriminator by judging and distinguishing the authenticity of the second light field image and the third light field image;
step S17, evaluating the quality of the generated second light field image through the peak signal-to-noise ratio and the structural similarity, and if the evaluation result does not meet the requirement, executing the steps S11, S13 and S15 again until the evaluation result meets the requirement;
and the third light field image is a light field image without snowflakes or rain strips under the same scene as the first light field image.
2. The method according to claim 1, wherein before the step S11, further comprising:
the first light field image and the third light field image are acquired.
3. The method according to claim 1, wherein the detecting the position and size of the snowflakes or rain strips in the acquired first light field image by using the 3D residual network comprises:
and obtaining a mask of the snow or rain strip data of the first light field image by using the 3D residual error network.
4. The method of claim 3, wherein said deriving a mask of snowflake or rainribbon data for the first light-field image using a 3D residual network comprises:
setting a residual block according to the 3D residual network;
setting parameters of a first convolution layer and a second convolution layer, wherein each residual block uses the first convolution layer and the second convolution layer respectively;
after the first convolution layer and the second convolution layer are used, a standardization operation and a long-time and short-time memory network are added, and a mask of snow or rain strip data of the first light field image is obtained.
5. The method of claim 4, wherein said removing detected snow or rain flakes from the first light field image using a 3D U type network to obtain a second light field image comprises:
from the mask, snow or rain flakes in the first light field image are removed using a 3D U type network, resulting in a second light field image.
6. The method of claim 5 wherein said removing snowflakes or rain flakes from the first light field image using a 3D U type network to obtain a second light field image comprises:
dividing a 3D U type network into an encoder network and a decoder network according to the position relation;
respectively adopting an encoder network and a decoder network to encode and decode snowflake or rain belt data;
and adopting pooling layer down-sampling to set an output function to obtain a second light field image.
7. The method of claim 1, wherein the objective function of the discriminator is:
wherein, VTIs a true value, V, of the 3D snow-free EPI volume block of the virtual scenetm*The snow-free 3D EPI volume block is generated, wherein D in the formula is a discriminator and related parameters thereof, and G is the generator and related parameters thereof.
8. A light field image snowflake or rain strip detection and removal system based on deep learning is characterized by comprising:
the detection module is used for detecting the position and the size of the snowflakes or the rain strips in the acquired first light field image by using the 3D residual error network;
the removing module is used for removing the detected snowflakes or rain belts in the first light field image by the self-learning by adopting a 3D U type network to obtain a second light field image;
the discrimination module is used for acquiring the second light field image and the third light field image and optimizing the objective function of the discriminator by judging and distinguishing the authenticity of the second light field image and the third light field image;
the evaluation module is used for evaluating the quality of the generated second light field image through the peak signal-to-noise ratio and the structural similarity, and if the evaluation result does not meet the requirement, the steps S11, S13 and S15 are executed again until the evaluation result meets the requirement;
and the third light field image is a light field image without snowflakes or rain strips under the same scene as the first light field image.
9. A light field image snowflake or rain strip detection and removal device based on deep learning is characterized by comprising:
the detector is used for detecting the position and the size of the snowflakes or the rain strips in the acquired first light field image by using the 3D residual error network;
the generator is used for removing the detected snowflakes or rain belts in the first light field image by adopting a 3D U type network to obtain a second light field image;
the discriminator acquires the second light field image and the third light field image, and optimizes the objective function of the discriminator by judging and distinguishing the truth of the second light field image and the third light field image;
the evaluator evaluates the quality of the generated second light field image through the peak signal-to-noise ratio and the structural similarity, and controls the detector, the generator and the discriminator to work again if the evaluation result does not meet the requirement until the evaluation result meets the requirement;
and the third light field image is a light field image without snowflakes or rain strips under the same scene as the first light field image.
10. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010243333.2A CN111445465B (en) | 2020-03-31 | 2020-03-31 | Method and equipment for detecting and removing snow or rain belt of light field image based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010243333.2A CN111445465B (en) | 2020-03-31 | 2020-03-31 | Method and equipment for detecting and removing snow or rain belt of light field image based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111445465A true CN111445465A (en) | 2020-07-24 |
| CN111445465B CN111445465B (en) | 2023-06-16 |
Family
ID=71652575
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010243333.2A Active CN111445465B (en) | 2020-03-31 | 2020-03-31 | Method and equipment for detecting and removing snow or rain belt of light field image based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111445465B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112288738A (en) * | 2020-11-18 | 2021-01-29 | 新疆大学 | Single image snowflake removing method and device based on deep learning and storage medium |
| CN112767274A (en) * | 2021-01-25 | 2021-05-07 | 江南大学 | Light field image rain stripe detection and removal method based on transfer learning |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107993260A (en) * | 2017-12-14 | 2018-05-04 | 浙江工商大学 | A kind of light field image depth estimation method based on mixed type convolutional neural networks |
| CN109102475A (en) * | 2018-08-13 | 2018-12-28 | 北京飞搜科技有限公司 | A kind of image rain removing method and device |
| CN109344818A (en) * | 2018-09-28 | 2019-02-15 | 合肥工业大学 | A kind of light field well-marked target detection method based on depth convolutional network |
| KR20190142216A (en) * | 2019-06-07 | 2019-12-26 | 케이에스아이 주식회사 | Vehicle sensing system using deep learning recognition of marking |
| CN110728640A (en) * | 2019-10-12 | 2020-01-24 | 合肥工业大学 | Double-channel single-image fine rain removing method |
-
2020
- 2020-03-31 CN CN202010243333.2A patent/CN111445465B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107993260A (en) * | 2017-12-14 | 2018-05-04 | 浙江工商大学 | A kind of light field image depth estimation method based on mixed type convolutional neural networks |
| CN109102475A (en) * | 2018-08-13 | 2018-12-28 | 北京飞搜科技有限公司 | A kind of image rain removing method and device |
| CN109344818A (en) * | 2018-09-28 | 2019-02-15 | 合肥工业大学 | A kind of light field well-marked target detection method based on depth convolutional network |
| KR20190142216A (en) * | 2019-06-07 | 2019-12-26 | 케이에스아이 주식회사 | Vehicle sensing system using deep learning recognition of marking |
| CN110728640A (en) * | 2019-10-12 | 2020-01-24 | 合肥工业大学 | Double-channel single-image fine rain removing method |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112288738A (en) * | 2020-11-18 | 2021-01-29 | 新疆大学 | Single image snowflake removing method and device based on deep learning and storage medium |
| CN112767274A (en) * | 2021-01-25 | 2021-05-07 | 江南大学 | Light field image rain stripe detection and removal method based on transfer learning |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111445465B (en) | 2023-06-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108986050B (en) | Image and video enhancement method based on multi-branch convolutional neural network | |
| CN111754446A (en) | Image fusion method, system and storage medium based on generation countermeasure network | |
| CN110807757B (en) | Image quality evaluation method and device based on artificial intelligence and computer equipment | |
| CN112861690B (en) | Multi-method fused remote sensing image change detection method and system | |
| CN113298837B (en) | Image edge extraction method and device, storage medium and equipment | |
| CN110580704A (en) | ET cell image automatic segmentation method and system based on convolutional neural network | |
| US7450762B2 (en) | Method and arrangement for determining an object contour | |
| CN108921942B (en) | Method and device for 2D (two-dimensional) conversion of image into 3D (three-dimensional) | |
| CN113781510B (en) | Edge detection method and device and electronic equipment | |
| CN111445465B (en) | Method and equipment for detecting and removing snow or rain belt of light field image based on deep learning | |
| CN110570402B (en) | Binocular salient object detection method based on boundary perception neural network | |
| CN111798469A (en) | Digital image small data set semantic segmentation method based on deep convolutional neural network | |
| CN111027347A (en) | Video identification method and device and computer equipment | |
| CN114220126A (en) | Target detection system and acquisition method | |
| CN112418087A (en) | Underwater video fish identification method based on neural network | |
| CN110503049B (en) | Satellite video vehicle number estimation method based on generation countermeasure network | |
| CN112348762A (en) | Single image rain removing method for generating confrontation network based on multi-scale fusion | |
| CN116279592A (en) | Method for dividing travelable area of unmanned logistics vehicle | |
| CN116309178A (en) | Visible light image denoising method based on self-adaptive attention mechanism network | |
| CN112329662B (en) | Multi-view saliency estimation method based on unsupervised learning | |
| Song et al. | A single image dehazing method based on end-to-end cpad-net network in deep learning environment | |
| CN112365451A (en) | Method, device and equipment for determining image quality grade and computer readable medium | |
| CN116363064A (en) | Defect identification method and device integrating target detection model and image segmentation model | |
| CN115829895A (en) | Multi-focus image fusion method and system based on structural similarity and region segmentation | |
| CN115375579A (en) | Offshore image defogging method based on multi-branch pyramid large-kernel convolutional network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |