CN116524199B - Image rain removing method and device based on PReNet progressive network - Google Patents
Image rain removing method and device based on PReNet progressive network Download PDFInfo
- Publication number
- CN116524199B CN116524199B CN202310438990.6A CN202310438990A CN116524199B CN 116524199 B CN116524199 B CN 116524199B CN 202310438990 A CN202310438990 A CN 202310438990A CN 116524199 B CN116524199 B CN 116524199B
- Authority
- CN
- China
- Prior art keywords
- image
- rain
- prenet
- progressive
- output
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 230000000750 progressive effect Effects 0.000 title claims abstract description 36
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims abstract description 30
- 238000001556 precipitation Methods 0.000 claims abstract description 23
- 238000012549 training Methods 0.000 claims abstract description 12
- 230000004927 fusion Effects 0.000 claims abstract description 4
- 230000006870 function Effects 0.000 claims description 23
- 238000012545 processing Methods 0.000 claims description 23
- 230000000694 effects Effects 0.000 claims description 10
- 238000010586 diagram Methods 0.000 claims description 9
- 238000005457 optimization Methods 0.000 claims description 7
- 238000009877 rendering Methods 0.000 claims description 6
- 238000012360 testing method Methods 0.000 claims description 6
- 238000000605 extraction Methods 0.000 claims description 5
- 238000012937 correction Methods 0.000 claims description 3
- 238000009826 distribution Methods 0.000 claims description 3
- 230000007774 longterm Effects 0.000 claims description 3
- 229920001084 poly(chloroprene) Polymers 0.000 claims description 3
- 238000011176 pooling Methods 0.000 claims description 3
- 238000002360 preparation method Methods 0.000 claims description 3
- 238000005070 sampling Methods 0.000 claims description 3
- 238000013441 quality evaluation Methods 0.000 claims description 2
- 230000008034 disappearance Effects 0.000 claims 1
- 238000001514 detection method Methods 0.000 abstract description 16
- 238000005516 engineering process Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000003860 storage Methods 0.000 description 3
- 238000011897 real-time detection Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000003708 edge detection Methods 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 239000004575 stone Substances 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/30—Noise filtering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V20/582—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads of traffic signs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Evolutionary Computation (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Processing (AREA)
Abstract
The invention discloses an image rain removing method and device based on a PReNet progressive network, comprising the following steps: s1, acquiring a truly rendered image to be rain removed and corresponding precipitation amount information; s2, gradually extracting characteristic information of the image in a mode of gradually increasing resolution; s3, extracting characteristic information in the image by adopting a progressive residual block as a core module; s4, fusing the characteristic information with different resolutions by using a characteristic fusion network; s5, finishing rain removal; s6, outputting a rain removing image of 1-N stages; s7, training a model by adopting a polynomial regression algorithm. The image rain removing method and device based on the PReNet progressive network not only improves the distinguishing performance of traffic signs in rainy days, but also improves the speed and accuracy of the whole traffic target detection and recognition system.
Description
Technical Field
The invention relates to the technical field of automatic driving and control of vehicles, in particular to an image rain removing method and device based on a PReNet progressive network.
Background
With the development of artificial intelligence technology, various technologies applied to public transportation scenes in the field of automatic driving of vehicles are rapidly developed. The target identification under the traffic scene is a key technology for acquiring effective information of traffic image data, and is an important foundation stone for making correct decisions by an intelligent power-assisted traffic related system and platform. Target detection and recognition under normal ambient light good conditions has found good experimental and production scenario applications. However, under extremely severe conditions, such as rainy, snowy, and dust-free environments, the quality of an image obtained by conventional machine vision is rapidly degraded, and the hue, saturation, and brightness deviate more than those in a clear environment, and cannot meet the high-precision standard of a detection target required in the field of automatic driving. Taking the most common heavy rain weather as an example, the quality contrast of pictures shot by a traditional camera is not high, and the images are blurred. The dense rain lines under complex road conditions can seriously influence the extraction of small targets, so that the decision and execution level of an automatic driving program are influenced, and the safety of personal driving and public transportation is endangered. Meanwhile, the traditional computer vision direction image segmentation and edge detection cannot meet the high-precision traffic target detection and identification tasks.
In recent years, students at home and abroad surround target detection in severe weather environments, and a series of image restoration and target detection methods based on a deep learning network are proposed. At present, there are two algorithms related to adding bad weather target detection, one is to train two sub-networks of image enhancement and target detection from end to end, and this method has a problem that it is difficult to balance between the image enhancement task and the target detection task. The other is to pre-process the image before the target detection so as to reduce the influence caused by severe weather conditions.
By utilizing the idea of image enhancement, the whole or partial characteristics of the image are purposefully emphasized, the original hue, brightness, contrast and the like of the image are improved to be destroyed by raindrop noise, the difference between different objects in the image is enlarged, and the visual effect of the image is improved.
For the general condition, when the PReNet recursion optimization times t=T=6, the rain removing effect is better. But as the number of recursions increases, the longer the individual images optimize, the performance decreases. Meanwhile, considering the requirement of automatic driving real-time detection, after the image denoising, the detection and the identification of traffic signs are required, and the requirement of the automatic driving real-time detection is extremely high on the capability of processing the image in real time.
Therefore, considering the influence of the precipitation amount under different environments on the image quality, the precipitation amount of the vehicle in the current environment is acquired by using the piezoelectric rainfall sensor and the Arduino interface and updated in real time, and the optimal cycle optimization times are applied according to the precipitation gradient by matching with the PReNet, so that the denoising time of a single frame image is reduced as much as possible while the image is ensured to meet the repair of a high-precision low-loss function.
Meanwhile, the function of transmitting the current precipitation correction parameter by the auxiliary sensor is achieved by using the existing map weather interface, so that the risk that the sensor is affected by interference under special conditions is reduced, and the detection efficiency of the whole system is enhanced.
Therefore, the invention provides a method and a device for image rain removal applied to an automatic driving environment based on the cooperation of a progressive image rain removal network and a rainfall sensor, so as to improve the distinguishing performance of traffic signs in rainy days.
Disclosure of Invention
The invention aims to provide an image rain removing method and device based on a PReNet progressive network, which not only improve the distinguishing performance of traffic signs in rainy days, but also improve the speed and accuracy of the whole traffic target detection and identification system.
In order to achieve the above purpose, the invention provides an image rain removing method and device based on a PReNet progressive network, comprising the following steps: s1, acquiring a truly rendered image to be rain removed and corresponding precipitation amount information; s2, gradually extracting characteristic information of the image in a mode of gradually increasing resolution; s3, extracting characteristic information in the image by adopting a progressive residual block as a core module; s4, fusing the characteristic information with different resolutions by using a characteristic fusion network; s5, finishing rain removal; s6, outputting a rain removing image of 1-N stages; s7, training a model by adopting a polynomial regression algorithm.
Preferably, in step S1, a preparation data set and a test set are first obtained, and a special data set KITTI for urban traffic in a clear environment is adopted, the data set is subjected to rainfall rendering by a related algorithm, an image to be processed is obtained by overlapping simulated rain lines and rain drops on a clear image, for the same image, real image rendering is simulated under different precipitation conditions, the clear images with and without rain correspond one by one, and all the images are in accordance with 8:2 to divide the data set and the test set.
Preferably, in step S2, a plurality of progressive residual blocks are used, each residual block including an upsampling module and a residual module, where the upsampling module upsamples the low resolution feature map to a high resolution by bilinear interpolation, and the residual module enhances feature expression by learning residual connections, and the output feature map of each residual block is input to the next residual block, so as to gradually increase the resolution and expression of the feature map.
Preferably, in step S3, progressive residual blocks (Progressive Residual Block, PRB) are used to extract advanced features of the image, the PRB structure contains multiple residual blocks, each residual block is composed of two convolution layers and one jump connection, the PRB structure increases the depth and complexity of the features step by step through multiple stages, so that advanced features in the image are extracted effectively, and furthermore, the jump connection in the PRB structure can help prevent gradient vanishing and accelerate model convergence.
Preferably, in step S4, the FFN structure includes a plurality of convolution layers and pooling layers, so that feature maps with different resolutions can be fused, thereby improving the perceptibility of the model to information with different scales.
Preferably, in step S5, the Neoprene gradually increases the resolution of the network by means of multi-stage expansion, so as to better recover the detail information of the image, specifically, the pnenet is divided into T stages, each stage uses an input image with different resolution, and different residual learning modules are adopted, which can be specifically explained as the following parts: tensor join and convolution layer processing: tensor connection is carried out on the output rain removal image of the previous stage and the original rain image y, and then the output rain removal image of the previous stage and the original rain image y are subjected to joint processing through a convolution layer (Conv) and a correction linear unit (Rectified Linear Units, reLU); LSTM module processing: the feature map output by the Encoder (Encoder) is processed to capture long-term dependencies in the sequence data. The input is a sequence of feature maps output by the encoder, each feature map representing a feature representation of the input image at different scales; residual block processing: the input and output feature diagram of the last stage is registered on the feature diagram after weight distribution so as to further improve the network performance and the anti-interference capability; output convolution layer processing: the characteristic extraction network of the rain line is subjected to a convolution processing of 3*3, namely 3-channel processing. In the final stage, the processed output value is assigned to the corresponding pixel position of the output image through the output convolution operation, and a modified linear unit (ReLU) is added to strengthen the contrast of the image.
Preferably, in step S6, since the pixels of the rain-removed image and the original image obtained by the convolution operation are equal, the following two Loss functions (Loss) are adopted:
one is mean square error (Mean Squared Error, MSE):
wherein m and n represent the width and channel of the output image, I, j represents the image after rain removal, K I, j represents the original image parameter, and simultaneously, a Structural Similarity Index (SSIM) is used, which is an image quality evaluation index, comprehensively considering brightness and contrast, if the value of SSIM is more prone to 1, the higher the similarity degree of two images is proved, and r (SSIM) =1-L (SSIM) is used in consideration of the rain removal degree of the image,
the formula of the loss function L (SSIM) is as follows:
compared with the traditional PSNR loss function, the SSIM more reflects the structural information before and after the image distortion, so that the L=lambda is finally adopted 1 L MSE +λ 2 L r(SSIM) Loss function as modelA number.
Preferably, in step S7, the data set is actually rendered and processed for the KITTI, the actual model is trained by a polynomial regression algorithm in supervised learning, and the actual weighted Loss function is Loss for the above requirements total ＝λ 1 L MSE +λ 2 (1-L SSIM ) For such a loss function, the lower the data is in theory, the better the rain removing effect of the image processing is, so the specific formula for the minimum cycle optimization times is as follows:
T＝min(L total ≤L max )；
on the premise of meeting the minimum loss function, the optimal recursion times T meeting the rain removing effect are obtained.
Therefore, the image rain removing method and device based on the PReNet progressive network with the structure not only improves the distinguishing performance of traffic signs in rainy days, but also improves the speed and accuracy of the whole traffic target detecting and identifying system.
The technical scheme of the invention is further described in detail through the drawings and the embodiments.
Drawings
Fig. 1 is a schematic flow chart of an embodiment of an image rain removing method and device based on a PReNet progressive network;
fig. 2 is a schematic structural diagram of an image rain removing method and an embodiment of a device based on a PReNet progressive network according to the present invention;
FIG. 3 is a schematic diagram of LSTM module of a PReNet progressive network based image rain removal method and apparatus according to the present invention;
fig. 4 is a schematic diagram of a method and a device for removing rain in a T-stage according to the present invention.
Detailed Description
The technical scheme of the invention is further described below through the attached drawings and the embodiments.
Unless defined otherwise, technical or scientific terms used herein should be given the ordinary meaning as understood by one of ordinary skill in the art to which this invention belongs. The terms "first," "second," and the like, as used herein, do not denote any order, quantity, or importance, but rather are used to distinguish one element from another. The word "comprising" or "comprises", and the like, means that elements or items preceding the word are included in the element or item listed after the word and equivalents thereof, but does not exclude other elements or items. The terms "connected" or "connected," and the like, are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "upper", "lower", "left", "right", etc. are used merely to indicate relative positional relationships, which may also be changed when the absolute position of the object to be described is changed.
As shown in fig. 1-4, the invention provides an image rain removing method and device based on a PReNet progressive network, comprising the following steps:
s1, acquiring a truly rendered image to be rain removed and corresponding precipitation amount information;
firstly, acquiring a preparation data set and a test set, adopting a special data set KITTI for urban traffic in a clear environment, performing related algorithm rainfall rendering on the data set, obtaining an image to be processed by superposing simulated rain lines and rain drops on a clear image, simulating real image rendering under different precipitation conditions for the same image, wherein the clear images with and without rain correspond one by one, and all the images are as follows: 2 to divide the data set and the test set.
S2, gradually extracting characteristic information of the image in a mode of gradually increasing resolution;
feature extraction to step up resolution: in particular, a plurality of progressive residual blocks are employed, each comprising an upsampling module and a residual module. The up-sampling module up-samples the low-resolution feature map to high resolution by adopting a bilinear interpolation method, and the residual error module enhances the feature expression capability by learning residual error connection. The output feature map of each residual block will be input into the next residual block, thereby gradually increasing the resolution and expressive power of the feature map. The process can be expressed by the following formula:
y＝H(x)+x；
where x represents the input feature map, H (x) represents the residual connection learned by the residual module, and y represents the output feature map.
S3, extracting characteristic information in the image by adopting a progressive residual block as a core module;
progressive residual blocks (Progressive Residual Block, PRB) are used to extract advanced features of the image, the PRB structure contains multiple residual blocks, each residual block is composed of two convolutional layers and one jump connection, the PRB structure increases the depth and complexity of the features step by step through multiple stages, thus effectively extracting advanced features in the image, and furthermore, the jump connection in the PRB structure can help prevent gradient vanishing and accelerate model convergence. The process can be expressed by the following formula:
y＝F(x)+x；
where x represents the input feature map, F (x) represents the feature map learned by the progressive residual block, and y represents the output feature map.
S4, fusing the characteristic information with different resolutions by using a characteristic fusion network;
the FFN structure comprises a plurality of convolution layers and pooling layers, and feature graphs with different resolutions can be fused, so that the perceptibility of the model to information with different scales is improved. In addition, the channel attention mechanism in the FFN structure can help the model to adaptively learn the relationship between different characteristic channels, and the specific formula is as follows:
where y represents the final output result, f is the feature extracted by the input x through each feature extractor, w is the weight coefficient of the corresponding feature, and C is the number of features.
S5, finishing rain removal;
the Neoprene gradually improves the resolution of the network in a multi-stage unfolding mode, so that the detail information of the image is better recovered. Specifically, the PReNet is divided into T stages, each stage uses a different resolution input image, and a different residual learning module is employed.
It can be explained in particular in the following parts: tensor join and convolution layer processing: the output rain-removed image of the previous stage and the original rain image y are tensor-connected, and then are jointly processed by a convolution layer (Conv) and a modified linear unit (modified LinearUnits, reLU).
LSTM module processing: the feature map output by the Encoder (Encoder) is processed to capture long-term dependencies in the sequence data. The input is a sequence of feature maps output by the encoder, each feature map representing a feature representation of the input image at a different scale.
The LSTM feature module comprises the following parts:
an input door: and controlling the flow of information into the memory unit.
i t ＝σ(w ii x t +b ii +w hi h t-1 +b hi )；
Forgetting the door: and controlling the flow of information flowing out of the memory unit.
f t ＝σ(w if x t +b if +w hf h t-1 +b hf )；
A memory unit: information from a previous time step is stored.
c t ＝tanh(w ic x t +w hc h t-1 +b c )；
Output door: and controlling the flow of information output from the memory unit.
h t ＝o t *tanh(c t )；
Residual block processing: and the input and output characteristic diagram of the last stage is registered on the characteristic diagram after weight allocation so as to further improve the network performance and the anti-interference capability.
Output convolution layer processing: the characteristic extraction network of the rain line is subjected to a convolution processing of 3*3, namely 3-channel processing. In the final stage, the processed output value is assigned to the corresponding pixel position of the output image through the output convolution operation, and a modified linear unit (ReLU) is added to strengthen the contrast of the image.
S6, outputting a rain removing image of 1-N stages;
according to the invention, weighting judgment is carried out by adopting a loss function MSE and SSIM for measuring the actual rain removal effect, and the minimum cycle optimization stage times T of the ideal rain removal effect under the minimum loss function requirement is judged.
The MSE loss function expression is:
the SSIM loss function expression is as follows:
the lower the MSE, the higher the SSIM, indicating that the actual image restoration reconstruction effect is better.
S7, training a model by adopting a polynomial regression algorithm.
For the actual demand, the actual weighted Loss function is Loss total ＝λ 1 L MSE +λ 2 (1-L SSIM ). Theoretically, the better the rain removing effect is, the smaller the value is, so the specific formula for the minimum cycle optimization number is as follows, t=min (L total ≤L max )。
Namely, on the premise of meeting the minimum loss function, the minimum recursion times T meeting the ideal rain removing effect are obtained.
Further, training a model according to the optimal recursion times under different precipitation under the conditions by using a polynomial regression algorithm in supervised learning, and outputting a polynomial fitting curve. The specific formula is as follows:
y＝b 0 +b 1 x+b 2 x 2 +...+b n x n ；
where y represents the precipitation amount, x represents the number of recursions, b0, b1, b2, etc. represent regression coefficients, and n represents the number of polynomials.
In training the model, the degree of the polynomial and the regression coefficients need to be determined, and the regression coefficients can be solved using a least squares method, i.e. minimizing the sum of squares of the residuals.
Specifically, assuming that m samples exist in the KATTI training set, the number of recursions of the ith sample is xi, and the corresponding precipitation amount is yi, the sum of squares of residuals may be expressed as:
RSS＝∑ i ＝1～m(yi-(b0+b1xi+b2xi 2 +...+bn*xi n )) 2 ；
finally, a polynomial regression curve with good error and good generalization capability is obtained.
Training a model by adopting a polynomial regression algorithm: the method adopts a polynomial regression training model of a gradient descent algorithm, takes precipitation of a known data set as input quantity x, and takes minimum recursion times T as output quantity training model.
The method comprises the following specific steps: selecting a polynomial model, and adopting a quadratic polynomial model, wherein the formula is y=w 0 +w 1 x+w 2 x 2 The method comprises the steps of carrying out a first treatment on the surface of the Randomly initializing model parameters such as w0, w1, w2; iterating the training data, and updating model parameters by using a gradient descent algorithm; in each iteration, calculating an error between a predicted value and an actual value of the current model, and calculating a partial derivative of the model parameter; updating model parameters according to the direction and the magnitude of the partial derivative; repeating the steps until the loss function converges or reaches the designated iteration times.
Working principle: and carrying out real-time precipitation detection by adopting an Arduino piezoelectric rainfall sensor, transmitting data to an edge server through a serial port, and simultaneously storing the data into a relational database Mysql. To reduce the memory space occupied by the program and to take into account the change in physical location in the autopilot situation, the interface updates the data to the edge server every 60 seconds.
And the edge server content distribution network receives the current precipitation parameter P, and calculates the optimal recursion optimization times T according to the curve fitted by the polynomial regression.
Meanwhile, in theory, under the condition that the precipitation-free weather is clear, the device is supposed to not perform the operation of image preprocessing and rain removing, and the video flow images acquired by the vehicle-mounted camera are directly distributed to the related traffic scene detection and recognition system. Thus, logic modules are designed on the basis of the above, and the required running states of the device and the method are judged. The time complexity and the space complexity of the whole automatic driving algorithm system are optimized as much as possible.
Meanwhile, special situations (such as damage of the piezoelectric precipitation sensor) possibly occurring in the actual working state of the piezoelectric precipitation sensor are considered, and on the basis, the device is added with data transmission and acquisition of a map weather-related interface, and the information is fused with the data acquired by the precipitation sensor, so that the accuracy and reliability of precipitation are improved, and the risk resistance and the detection efficiency of the whole system are assisted.
Including algorithms, systems and storage media, and database systems. The main neural network is PReNet, a multi-stage residual network is adopted, network parameters are shared through recursive computation among stages, and meanwhile, the recursive computation is also used in each stage, so that the problems of parameter redundancy and overfitting generated by a stacked network are relieved. And finally, obtaining a final rain-removing image through convolution processing. The invention aims to improve the rain removing accuracy of the image and simultaneously combine the characteristics of a rain chart to improve the generalization of rain removing and the integrity of an image structure.
It can be understood to those of ordinary skill in the art that: the implementation of the whole system requires an algorithm on software, an existing map software weather interface and a piezoelectric precipitation sensor on hardware, and the device is also provided with related storage media, including a mobile storage device, a ROM or a RAM and various media capable of storing codes.
Therefore, the image rain removing method and device based on the PReNet progressive network with the structure not only improves the distinguishing performance of traffic signs in rainy days, but also improves the speed and accuracy of the whole traffic target detecting and identifying system.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention and not for limiting it, and although the present invention has been described in detail with reference to the preferred embodiments, it will be understood by those skilled in the art that: the technical scheme of the invention can be modified or replaced by the same, and the modified technical scheme cannot deviate from the spirit and scope of the technical scheme of the invention.
Claims (8)
1. An image rain removing method and device based on a PReNet progressive network comprises the following steps: s1, acquiring a truly rendered image to be rain removed and corresponding precipitation amount information; s2, gradually extracting characteristic information of the image in a mode of gradually increasing resolution; s3, extracting characteristic information in the image by adopting a progressive residual block as a core module; s4, fusing the characteristic information with different resolutions by using a characteristic fusion network; s5, finishing rain removal; s6, outputting a rain removing image of 1-N stages; s7, solving regression coefficients by using a least square method, adopting a polynomial regression training model of a gradient descent algorithm, and taking precipitation of a known data set as input quantity x and minimum recursion times T as an output quantity training model.
2. The image rain removing method and device based on the PReNet progressive network according to claim 1, wherein the method is characterized in that: in step S1, a preparation data set and a test set are firstly obtained, a special data set KITTI for urban traffic in a clear environment is adopted, the data set carries out rainfall rendering of a related algorithm, an image to be processed is obtained by overlapping simulated rain lines and rain drops on a clear image, for the same image, real image rendering is simulated under different precipitation conditions, the clear images with rain and without rain are in one-to-one correspondence, and all the images are in accordance with 8:2 to divide the data set and the test set.
3. The image rain removing method and device based on the PReNet progressive network according to claim 1, wherein the method is characterized in that: in step S2, a plurality of progressive residual blocks are used, each residual block includes an up-sampling module and a residual module, where the up-sampling module up-samples the low-resolution feature map to a high resolution by using a bilinear interpolation method, and the residual module enhances the feature expression capability by learning residual connection, and the output feature map of each residual block is input into the next residual block, so as to gradually increase the resolution and expression capability of the feature map.
4. The image rain removing method and device based on the PReNet progressive network according to claim 1, wherein the method is characterized in that: in step S3, progressive residual blocks (Progressive ResidualBlock, PRB) are used to extract advanced features of the image, and the PRB structure contains multiple residual blocks, each consisting of two convolutional layers and one jump connection, and the PRB structure gradually increases the depth and complexity of the features through multiple stages, so as to effectively extract advanced features in the image, and furthermore, the jump connection in the PRB structure can help prevent gradient disappearance and accelerate model convergence.
5. The image rain removing method and device based on the PReNet progressive network according to claim 1, wherein the method is characterized in that: in step S4, the FFN structure includes a plurality of convolution layers and pooling layers, so that feature maps with different resolutions can be fused, thereby improving the perceptibility of the model to information with different scales.
6. The image rain removing method and device based on the PReNet progressive network according to claim 1, wherein the method is characterized in that: in step S5, the Neoprene gradually increases the resolution of the network in a multi-stage expansion manner, so as to better recover the detail information of the image, specifically, the PReNet is divided into T stages, each stage uses an input image with different resolutions, and different residual error learning modules are adopted; specifically, the following parts can be explained: tensor join and convolution layer processing: tensor connection is carried out on the output rain removal image of the previous stage and the original rain image y, and then the output rain removal image of the previous stage and the original rain image y are subjected to joint processing through a convolution layer (Conv) and a correction linear unit (Rectified Linear Units, reLU); LSTM module processing: processing a feature map output by an Encoder (Encoder) to capture long-term dependencies in the sequence data; the input is a sequence of feature maps output by the encoder, each feature map representing a feature representation of the input image at different scales; residual block processing: the input and output feature diagram of the last stage is registered on the feature diagram after weight distribution so as to further improve the network performance and the anti-interference capability; output convolution layer processing: performing a 3*3 convolution treatment, namely 3-channel treatment, on the characteristic extraction network of the rain line; in the final stage, the processed output value is assigned to the corresponding pixel position of the output image through the output convolution operation, and a modified linear unit (ReLU) is added to strengthen the contrast of the image.
7. The image rain removing method and device based on the PReNet progressive network according to claim 1, wherein the method is characterized in that: in step S6, since the rainless image obtained by the convolution operation is equal to the original pixel, the following two Loss functions (Loss) are adopted:
one is mean square error (Mean Squared Error, MSE):
wherein m and n represent the width and channel of the output image, I [ I, j ] represents the image after rain removal, K [ I, j ] represents the original image parameter, and meanwhile, a Structural Similarity Index (SSIM) is adopted, which is an image quality evaluation index, brightness and contrast are comprehensively considered, if the value of SSIM is more prone to 1, the similarity degree of two pictures is proved to be higher, and the rain removal degree of the image is considered, wherein:
r(SSIM)＝1-L(SSIM)；
the formula of the loss function L (SSIM) is as follows:
compared with the traditional PSNR loss function, the SSIM more reflects the structural information before and after the image distortion, so that the L=lambda is finally adopted 1 L MSE +λ 2 L r(SSIM) As a function of the loss of the model.
8. The image rain removing method and device based on the PReNet progressive network according to claim 1, wherein the method is characterized in that: in step S7, the data set is truly rendered and processed for the KITTI, an actual model is trained through a polynomial regression algorithm in supervised learning, and the actual weighted Loss function is Loss for the requirements total ＝λ 1 L MSE +λ 2 (1-L SSIM ) For such a loss function, the lower the data is in theory, the better the rain removing effect of the image processing is, so the specific formula for the minimum cycle optimization times is as follows:
T＝min(L total ≤L max )；
namely, on the premise of meeting the lowest loss function, the optimal recursion times T meeting the rain removing effect are obtained.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310438990.6A CN116524199B (en) | 2023-04-23 | 2023-04-23 | Image rain removing method and device based on PReNet progressive network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310438990.6A CN116524199B (en) | 2023-04-23 | 2023-04-23 | Image rain removing method and device based on PReNet progressive network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116524199A CN116524199A (en) | 2023-08-01 |
| CN116524199B true CN116524199B (en) | 2024-03-08 |
Family
ID=87396892
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310438990.6A Active CN116524199B (en) | 2023-04-23 | 2023-04-23 | Image rain removing method and device based on PReNet progressive network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116524199B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110675321A (en) * | 2019-09-26 | 2020-01-10 | 兰州理工大学 | Super-resolution image reconstruction method based on progressive depth residual error network |
| CN110796623A (en) * | 2019-10-31 | 2020-02-14 | 上海融军科技有限公司 | Infrared image rain removing method and device based on progressive residual error network |
| CN113506300A (en) * | 2021-06-25 | 2021-10-15 | 江苏大学 | Image semantic segmentation method and system based on rainy complex road scene |
| CN114140672A (en) * | 2021-11-19 | 2022-03-04 | 江苏大学 | Target detection network system and method applied to multi-sensor data fusion in rainy and snowy weather scene |
| CN115271154A (en) * | 2022-06-07 | 2022-11-01 | 中国长江电力股份有限公司 | Nonlinear regression flood element prediction method based on polynomial and partial least square coupling |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2801354B1 (en) * | 2004-10-08 | 2017-02-08 | Forward Pharma A/S | Controlled release pharmaceutical compositions comprising a fumaric acid ester |
-
2023
- 2023-04-23 CN CN202310438990.6A patent/CN116524199B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110675321A (en) * | 2019-09-26 | 2020-01-10 | 兰州理工大学 | Super-resolution image reconstruction method based on progressive depth residual error network |
| CN110796623A (en) * | 2019-10-31 | 2020-02-14 | 上海融军科技有限公司 | Infrared image rain removing method and device based on progressive residual error network |
| CN113506300A (en) * | 2021-06-25 | 2021-10-15 | 江苏大学 | Image semantic segmentation method and system based on rainy complex road scene |
| CN114140672A (en) * | 2021-11-19 | 2022-03-04 | 江苏大学 | Target detection network system and method applied to multi-sensor data fusion in rainy and snowy weather scene |
| CN115271154A (en) * | 2022-06-07 | 2022-11-01 | 中国长江电力股份有限公司 | Nonlinear regression flood element prediction method based on polynomial and partial least square coupling |
Non-Patent Citations (5)
| Title |
|---|
| Continual image deraining with hypergraph convolutional Networks;xueyang Fu等;《IEEE Transactions on Pattern Analysis and Machine Intelligence》;9534-9551 * |
| Polynomial Regression as an Alternative to Neural Nets;Xi Cheng等;《https://arxiv.org/abs/1806.06850》;1-26 * |
| Progressive Image Deraining Networks: A Better and Simpler Baseline;Dongwei Ren等;《https://arxiv.org/abs/1901.09221》;1-11 * |
| 单幅图像去雨算法的研究;刘家成;《中国优秀硕士学位论文全文数据库信息科技辑》;I138-690 * |
| 基于遗传算法的投影寻踪回归模型在参考作物滕发量预测中的应用;迟道才;曲霞;崔磊;刘婷婷;王琦;沈亚西;;节水灌溉(第02期);9-11 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116524199A (en) | 2023-08-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110189334B (en) | Medical image segmentation method of residual error type full convolution neural network based on attention mechanism | |
| CN111126359B (en) | High-definition image small target detection method based on self-encoder and YOLO algorithm | |
| CN111539887B (en) | Channel attention mechanism and layered learning neural network image defogging method based on mixed convolution | |
| CN111915530B (en) | End-to-end-based haze concentration self-adaptive neural network image defogging method | |
| CN114120102A (en) | Boundary-optimized remote sensing image semantic segmentation method, device, equipment and medium | |
| CN113469094A (en) | Multi-mode remote sensing data depth fusion-based earth surface coverage classification method | |
| CN112232349A (en) | Model training method, image segmentation method and device | |
| CN111951207A (en) | Image quality enhancement method based on deep reinforcement learning and semantic loss | |
| CN111209858A (en) | Real-time license plate detection method based on deep convolutional neural network | |
| CN112488025A (en) | Double-temporal remote sensing image semantic change detection method based on multi-modal feature fusion | |
| CN112967327A (en) | Monocular depth method based on combined self-attention mechanism | |
| CN115035172A (en) | Depth estimation method and system based on confidence degree grading and inter-stage fusion enhancement | |
| CN114445442A (en) | Multispectral image semantic segmentation method based on asymmetric cross fusion | |
| CN111861935B (en) | Rain removing method based on image restoration technology | |
| CN113689383A (en) | Image processing method, device, equipment and storage medium | |
| CN116524199B (en) | Image rain removing method and device based on PReNet progressive network | |
| CN117078553A (en) | Image defogging method based on multi-scale deep learning | |
| CN111275751A (en) | Unsupervised absolute scale calculation method and system | |
| CN116258854A (en) | Pavement crack segmentation method based on dense connection strategy and image gradient | |
| CN116012687A (en) | Image interaction fusion method for identifying tread defects of wheel set | |
| CN115170807A (en) | Image segmentation and model training method, device, equipment and medium | |
| CN115187775A (en) | Semantic segmentation method and device for remote sensing image | |
| CN114581789A (en) | Hyperspectral image classification method and system | |
| CN113240589A (en) | Image defogging method and system based on multi-scale feature fusion | |
| CN117422689B (en) | Rainy day insulator defect detection method based on improved MS-PReNet and GAM-YOLOv7 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |