CN112581377A

CN112581377A - Image processing method and device and electronic equipment

Info

Publication number: CN112581377A
Application number: CN201910939909.6A
Authority: CN
Inventors: 董振; 黄明杨; 刘春晓; 石建萍
Original assignee: Shanghai Sensetime Lingang Intelligent Technology Co Ltd
Current assignee: Shanghai Sensetime Lingang Intelligent Technology Co Ltd
Priority date: 2019-09-30
Filing date: 2019-09-30
Publication date: 2021-03-30
Anticipated expiration: 2039-09-30
Also published as: CN112581377B

Abstract

The embodiment of the disclosure provides an image processing method, an image processing device and electronic equipment, wherein the method comprises the following steps: extracting a first rain layer distribution characteristic in the first rain-carrying image; and respectively fusing the first rain layer distribution characteristics with at least one rain-free image to obtain at least one first synthesized rain-carrying image. The embodiment of the disclosure can improve the rain removing effect of the rain image.

Description

Image processing method and device and electronic equipment

Technical Field

The present disclosure relates to machine learning technologies, and in particular, to an image processing method and apparatus, and an electronic device.

Background

Rain is a very common weather in real life. However, it affects visual visibility. Especially in heavy rainy weather, rain strips from various directions are accumulated together to obscure background scenes, which seriously affects the accuracy of many computer vision systems, such as the analysis results of the computer vision systems in automatic driving, security monitoring, vision detection and other scenes.

Disclosure of Invention

In view of the above, the present disclosure provides at least an image processing method, an image processing apparatus, and an electronic device.

In a first aspect, an image processing method is provided, the method comprising:

extracting a first rain layer distribution characteristic in the first rain-carrying image;

and respectively fusing the first rain layer distribution characteristics with at least one rain-free image to obtain at least one first synthesized rain-carrying image.

In some alternative embodiments, said fusing said first rainfly distribution features with at least one respective rainless image to obtain at least one first composite rained image, includes: respectively fusing the first rain layer distribution characteristic and the plurality of different rain-free images to obtain a plurality of different first synthetic rain-carrying images; after the obtaining a plurality of different first composite rain images, the method further comprises: and combining the plurality of different rain-free images with the corresponding first synthesized rain-carrying images respectively to obtain a plurality of pairs of different image pairs.

In some optional embodiments, the method further comprises: extracting a second raining layer distribution characteristic in a second rain-carrying image, wherein the difference between the second raining layer distribution characteristic and the first raining layer distribution characteristic meets a preset requirement; and fusing the second rain layer distribution characteristics with at least one rain-free image respectively to obtain at least one second synthesized rain-carrying image.

In some alternative embodiments, the fusing the second raining layer distribution characteristics with the at least one rainless image to obtain at least one second composite rained image includes: fusing the second rain layer distribution characteristics with a plurality of different rain-free images respectively to obtain a plurality of different second synthetic rain-carrying images; after the obtaining a plurality of different second composite rain images, the method further comprises: and combining the plurality of different rain-free images with the corresponding second composite rain-carrying images respectively to obtain a plurality of pairs of different image pairs.

In some alternative embodiments, the first rain-bearing image or the second rain-bearing image is a truly captured rain-bearing image, and the background portion of the first rain-bearing image or the second rain-bearing image that is not rain is a monochrome patch.

In some optional embodiments, the first rain-bearing image or the second rain-bearing image is a sub image block extracted from a real captured rain-bearing image, and a background portion of the sub image block, which is not rain water, is a monochrome color block.

In some optional embodiments, the first rainlayer distribution feature is extracted through a feature extraction network; the training process of the feature extraction network comprises the following steps: inputting the first rain-carrying image into the feature extraction network to obtain the rain layer distribution feature predicted and output by the feature extraction network; and taking the predicted and output rain layer distribution characteristics and the corresponding labels as the input of a discriminator of the generative confrontation network, and adjusting the network parameters of the characteristic extraction network according to the output result of the discriminator.

In some alternative embodiments, the fusing the first rainfly distribution features with the at least one rain-free image to obtain at least one first composite rain-bearing image includes: and linearly adding the first rain layer distribution characteristic and one rain-free image to obtain a corresponding first synthesized rain-carrying image.

In some alternative embodiments, the fusing the first rainfly distribution features with the at least one rain-free image to obtain at least one first composite rain-bearing image includes: linearly adding the first rainbed distribution characteristic to one of the rainless images; and inputting the linearly added images into an image synthesis network to obtain one first synthesized rain-carrying image output by the image synthesis network.

In some optional embodiments, the training process of the image synthesis network comprises: linearly adding the first rain layer distribution characteristic to the rain-free image; inputting the images subjected to linear addition into an image synthesis network to obtain a synthesized rain-carrying image predicted and output by the image synthesis network; and taking the synthesized rain-bearing image and the corresponding label as the input of a discriminator of the generating type confrontation network, and adjusting the network parameters of the image synthesis network according to the output result of the discriminator.

In some optional embodiments, before the inputting the composite rain image and the corresponding tag as the discriminator of the generative confrontation network, the method further comprises: respectively carrying out guiding filtering processing on the synthetic rain-carrying image and the corresponding label; and removing the images after the guiding filtering processing respectively corresponding to the synthesized rain-carrying images and the corresponding labels to obtain respective corresponding detail maps, wherein the detail maps are used for inputting into the discriminator.

In some optional embodiments, the method further comprises: training an image rain-removing neural network using the plurality of pairs of different image pairs.

In some alternative embodiments, the training an image degraining neural network using the plurality of pairs of different images comprises: inputting the synthesized rain-carrying image in the image pair into the rain-removing neural network of the image to obtain a predicted rain-removing image; the composite rain image is the first composite rain image or the second composite rain image; adjusting network parameters of the image rain-removing neural network based on a difference between the predicted rain-removing image and the corresponding rain-free image of the image pair.

In some optional embodiments, the adjusting the network parameters of the image rain-removing neural network based on the difference between the predicted rain-removing image and the corresponding rain-free image in the image pair comprises: obtaining a difference between the predicted rain-removed image and the rain-free image through a structural similarity loss function, and adjusting network parameters of a neural network for removing rain of the image based on the difference; and/or inputting the predicted rain removing image and the rain-free image into a discriminator, and adjusting the network parameters of the image rain removing neural network according to the output result of the discriminator.

In a second aspect, there is provided an image processing method, the method comprising:

acquiring an input image to be processed, wherein the input image comprises a rain layer and a background;

carrying out rain removing treatment on the input image to obtain a rain-free image comprising the background of the input image;

processing a preset task on the rain-free image;

the predetermined task includes at least one of: object detection, object tracking, scene segmentation, object classification, object identification.

In some optional embodiments, before the raining the input image, the method further comprises: and detecting and determining that the rain layer is contained in the input image.

In some optional embodiments, the performing the rain removal processing on the input image includes: according to a pre-trained neural network for image rain removal, performing rain removal processing on the input image to obtain a rain-free image comprising the background of the input image; the image rain-removing neural network is obtained by training through the method of any embodiment of the disclosure.

In a third aspect, an image processing apparatus is provided, the apparatus comprising:

the characteristic extraction module is used for extracting first rain layer distribution characteristics in the first rain-carrying image;

and the image fusion module is used for fusing the first rain layer distribution characteristics with at least one rain-free image respectively to obtain at least one first synthesized rain-carrying image.

In some optional embodiments, the image fusion module is specifically configured to fuse the first rain layer distribution characteristic and a plurality of different rain-free images respectively to obtain a plurality of different first synthesized rain-carrying images; the device further comprises: an image pair processing module configured to, after the obtaining the plurality of different first composite rain images, further include: and combining the plurality of different rain-free images with the corresponding first synthesized rain-carrying images respectively to obtain a plurality of pairs of different image pairs.

In some optional embodiments, the feature extraction module is further configured to extract a second raining layer distribution feature in the second rain-carrying image, where a difference between the second raining layer distribution feature and the first raining layer distribution feature satisfies a predetermined requirement; the image fusion module is further configured to fuse the second raining layer distribution characteristics with the at least one rain-free image, respectively, to obtain at least one second synthesized rain-carrying image.

In some optional embodiments, the image fusion module is further configured to fuse the second raining layer distribution characteristics with a plurality of different rain-free images, respectively, to obtain a plurality of different second composite rain-carrying images; the image pair processing module is further configured to combine the plurality of different rain-free images with the respective corresponding second composite rain-carrying images, so as to obtain a plurality of pairs of different image pairs.

In some optional embodiments, the apparatus further comprises: a first network training module, configured to train the feature extraction network by: inputting the first rain-carrying image into the feature extraction network to obtain the rain layer distribution feature predicted and output by the feature extraction network; taking the predicted and output rain layer distribution characteristics and corresponding labels as the input of a discriminator of a generative confrontation network, and adjusting the network parameters of the characteristic extraction network according to the output result of the discriminator; the first rain layer distribution characteristic is extracted through the characteristic extraction network.

In some optional embodiments, the image fusion module is specifically configured to linearly add the first raining layer distribution characteristic to one of the rainless images to obtain a corresponding one of the first synthesized rained images.

In some optional embodiments, the image fusion module is specifically configured to linearly add the first rainfly distribution characteristic to one of the rainless images; and inputting the linearly added images into an image synthesis network to obtain one first synthesized rain-carrying image output by the image synthesis network.

In some optional embodiments, the apparatus further comprises: a second network training module for training the image synthesis network by: linearly adding the first rain layer distribution characteristic to the rain-free image; inputting the images subjected to linear addition into an image synthesis network to obtain a synthesized rain-carrying image predicted and output by the image synthesis network; and taking the synthesized rain-bearing image and the corresponding label as the input of a discriminator of the generating type confrontation network, and adjusting the network parameters of the image synthesis network according to the output result of the discriminator.

In some optional embodiments, the second network training module is further configured to perform a guided filtering process on the synthetic rain image and the corresponding label, respectively, before the second network training module is configured to use the synthetic rain image and the corresponding label as inputs of the discriminator of the generative confrontation network; and removing the images after the guiding filtering processing respectively corresponding to the synthesized rain-carrying images and the corresponding labels to obtain respective corresponding detail maps, wherein the detail maps are used for inputting into the discriminator.

In some optional embodiments, the apparatus further comprises: a third network training module to train an image rain removing neural network using the plurality of pairs of different image pairs.

In some optional embodiments, the third network training module is specifically configured to: inputting the synthesized rain-carrying image in the image pair into the rain-removing neural network of the image to obtain a predicted rain-removing image; the composite rain image is the first composite rain image or the second composite rain image; adjusting network parameters of the image rain-removing neural network based on a difference between the predicted rain-removing image and the corresponding rain-free image of the image pair.

In some optional embodiments, the third network training module, when used to adjust the network parameters of the image degraining neural network, comprises: obtaining a difference between the predicted rain-removed image and the rain-free image through a structural similarity loss function, and adjusting network parameters of a neural network for removing rain of the image based on the difference; and/or inputting the predicted rain removing image and the rain-free image into a discriminator, and adjusting the network parameters of the image rain removing neural network according to the output result of the discriminator.

In a fourth aspect, there is provided an image processing apparatus, the apparatus comprising: the image acquisition module is used for acquiring an input image to be processed, wherein the input image comprises a rain layer and a background; the rain removing processing module is used for carrying out rain removing processing on the input image to obtain a rain-free image comprising the background of the input image; the task processing module is used for processing a preset task on the rain-free image; the predetermined task includes at least one of: object detection, object tracking, scene segmentation, object classification, object identification.

In some optional embodiments, the rain-removing processing module is further configured to detect and determine that the input image includes the rain layer before performing rain-removing processing on the input image.

In some optional embodiments, the rain removal processing module is specifically configured to: according to a pre-trained neural network for image rain removal, performing rain removal processing on the input image to obtain a rain-free image comprising the background of the input image; the image rain-removing neural network is obtained by training through the method of any embodiment of the disclosure.

In a fifth aspect, an electronic device is provided, the device comprising a memory for storing computer instructions executable on a processor, the processor being configured to implement the image processing method of any of the embodiments of the present disclosure when executing the computer instructions.

In a sixth aspect, a computer-readable storage medium is provided, on which a computer program is stored, which when executed by a processor implements the image processing method of any of the embodiments of the present disclosure.

According to the image processing method, the image processing device and the electronic equipment, the first synthetic rain-carrying image is obtained through the first rain layer distribution characteristic extracted from the first rain-carrying image which is really collected, and the obtained rain layer distribution characteristic is more vivid and the synthetic rain-carrying image is also more real because the rain layer distribution characteristic is the rain layer distribution extracted from the rain-carrying image; furthermore, when the image rain removing neural network is trained through the more real synthetic rain-carrying image, the neural network has better effect when used for rain removing processing of the real rain-carrying image.

Drawings

In order to more clearly illustrate one or more embodiments of the present disclosure or technical solutions in related arts, the drawings used in the description of the embodiments or related arts will be briefly described below, it is obvious that the drawings in the description below are only some embodiments described in one or more embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without inventive exercise.

Fig. 1 illustrates an image processing method provided by at least one embodiment of the present disclosure;

FIG. 2 illustrates a method for training an image rain removing neural network, provided by at least one embodiment of the present disclosure;

fig. 3 is a system architecture diagram illustrating a training method of an image rain removing neural network according to at least one embodiment of the present disclosure;

fig. 4 illustrates a structure of a feature extraction network provided by at least one embodiment of the present disclosure;

FIG. 5 illustrates a network structure of an image rain-removing neural network provided by at least one embodiment of the present disclosure;

FIG. 6 illustrates a local network structure of an image rain-removing neural network provided by at least one embodiment of the present disclosure;

fig. 7 illustrates an image processing method provided by at least one embodiment of the present disclosure;

fig. 8 illustrates an image processing apparatus provided in at least one embodiment of the present disclosure;

fig. 9 illustrates an image processing apparatus provided in at least one embodiment of the present disclosure;

fig. 10 illustrates an image processing apparatus according to at least one embodiment of the present disclosure.

Detailed Description

In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of the present disclosure, the technical solutions in one or more embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in one or more embodiments of the present disclosure, and it is apparent that the described embodiments are only a part of the embodiments of the present disclosure, and not all embodiments. All other embodiments that can be derived by one of ordinary skill in the art based on one or more embodiments of the disclosure without inventive faculty are intended to be within the scope of the disclosure.

The rain removing can be carried out on the rain-carrying image through the neural network, so that the background of the image after the rain removing is obtained, the detection or analysis processing of the predetermined task is further carried out on the basis of the background part of the image after the rain removing, and the method can be applied to but not limited to application scenes such as automatic driving, security monitoring and the like.

Fig. 1 provides an image processing method that may be used to generate a composite rain-bearing image, which may be used, for example, to train an image to a neural network for rain, but may also be used for other purposes, and the embodiment is not limited thereto. As shown in fig. 1, the method may include:

in step 100, a first rain layer distribution feature in a first rain-bearing image is extracted.

In this step, the first rain-bearing image may be, for example, a really captured rain-bearing image, such as an image actually captured by a camera on a certain road in rainy days, and the image content includes a rain part (which may be referred to as "foreground" in the image) and other road captured contents (which may be referred to as "background" in the image) except for rain. The first rain layer distribution characteristic may be used to represent a distribution characteristic of a rain layer in the first rain-bearing image. The first rain layer distribution characteristic can be extracted from the first rain-carrying image through a neural network, or can be extracted from the first rain-carrying image through other modes.

In step 102, the first rain layer distribution characteristics are respectively fused with at least one rain-free image to obtain at least one first composite rain-carrying image.

In this step, the rain-free image may be an acquired image without rain, and the rain layer corresponding to the first rain layer distribution feature may be added to the rain-free image by fusing the first rain layer distribution feature extracted in step 100 with the rain-free image, so as to obtain the first composite rain-containing image.

In the image processing method of the embodiment, the first composite rain-carrying image is obtained by the first rain layer distribution characteristic extracted from the actually acquired first rain-carrying image, and the obtained rain layer distribution characteristic is more vivid and the composite rain-carrying image is more real because the rain layer distribution characteristic is the rain layer distribution extracted from the rain-carrying image; furthermore, when the image rain removing neural network is trained through the more real synthetic rain-carrying image, the neural network has better effect when used for rain removing processing of the real rain-carrying image.

In addition, on the basis of the composite rain image obtained in fig. 1, the composite rain image may be paired with a rain-free image, and the paired images may be used for training a neural network for rain removal of the image, or may be used for other purposes. If an image rain-removing neural network is to be trained, a plurality of image pairs are usually required as a training sample set, each image pair is used as a training sample, and each image pair comprises an image with rain and an image without rain, and the backgrounds of the images are the same.

When the image pair is specifically obtained, the distribution characteristics of the first rain layer and a plurality of different rain-free images can be respectively fused to obtain a plurality of different first synthesized rain-carrying images; and combining the plurality of different rain-free images with the corresponding first synthesized rain-carrying images respectively to obtain a plurality of pairs of different images.

For example, assuming that the first rain layer distribution characteristic is T, there are three rain-free images C1, C2, and C3, whose backgrounds may be different. The T and the C1 are fused to obtain a first synthetic rain image H1, the T and the C2 are fused to obtain a first synthetic rain image H2, and the T and the C3 are fused to obtain a first synthetic rain image H3. Then, the rain-free image C1 and H1 may be combined into one image pair, the rain-free image C2 and H2 may be combined into one image pair, and the rain-free image C3 and H3 may be combined into one image pair.

In other embodiments, different rainbed distribution characteristics may also be selected for obtaining a composite rained image, so as to generate an image rich in rainbed distribution characteristics. For example, not only is the first rainlaid distribution characteristic described above used to fuse with multiple rainless images to obtain a first composite rained image, but also: and extracting second rain layer distribution characteristics in the second rain-carrying image, and fusing the second rain layer distribution characteristics with the at least one rain-free image respectively to obtain at least one second synthesized rain-carrying image. The process of obtaining the second composite rain image is the same as the first composite rain image, and is not described again.

The difference between the second rain layer distribution characteristic and the first rain layer distribution characteristic satisfies a predetermined requirement, for example, a characteristic difference value between the second rain layer distribution characteristic and the first rain layer distribution characteristic may be calculated, and the characteristic difference value may be greater than a certain difference threshold, that is, the difference between the two rain layer distribution characteristics is relatively large. In this way, a composite rain-bearing image containing rich distribution characteristics of the rain layer can be obtained.

Similarly, the second composite rain-carrying image may also be paired with a corresponding rain-free image, for example, the second rain layer distribution characteristics are respectively fused with a plurality of different rain-free images to obtain a plurality of different second composite rain-carrying images. And combining the plurality of different rain-free images with the corresponding second composite rain-carrying images respectively to obtain a plurality of pairs of different image pairs.

If the image pairs obtained by corresponding the first synthetic rain-carrying image and the second synthetic rain-carrying image are used for training the neural network for removing rain from the images, the distribution characteristics of the rain layer in the image pairs are rich, so that the generalization capability of the network is good, and the rain layer in the images containing different rain layer characteristics can be well removed.

In one embodiment, the first rain-bearing image or the second rain-bearing image described above is a true captured rain-bearing image, and the background portion of the non-rain water in the first rain-bearing image or the second rain-bearing image may be made a monochrome patch at the time of capturing the true rain-bearing image. Therefore, the background noise interference during the extraction of the distribution characteristics of the rain layer can be reduced, and the accurate distribution characteristics of the rain layer can be extracted.

In another embodiment, the first rain-bearing image or the second rain-bearing image that is actually captured may not be a monochrome patch, but a sub image block in which a background portion other than rain is a monochrome patch may be extracted from these rain-bearing images. And then, the sub-image blocks are used for extracting the distribution characteristics of the rain layer, so that the noise interference in the process of extracting the distribution characteristics of the rain layer can be reduced, and more accurate distribution characteristics of the rain layer can be extracted.

Fig. 2 illustrates a training method of a neural network for image rain removal, which can obtain a training sample with better quality (better network performance when used for training a network) so as to train a neural network with better performance through the training sample. Wherein, the method can comprise the following steps:

in step 200, the distribution characteristics of the rain layer in the real rain-bearing image are extracted.

The real rain-bearing image may be a real photographed image of rain. The real rain-carrying image may be the first rain-carrying image or the second rain-carrying image, the rain layer distribution feature may be a first rain layer distribution feature extracted from the first rain-carrying image or a second rain layer distribution feature extracted from the second rain-carrying image, and the first rain layer distribution feature and the second rain layer distribution feature are different.

In this embodiment, the rainfly distribution feature extracted from the real rainfly image may be, for example, a feature map used for representing the rainfly distribution (including, but not limited to, the density of rain lines, the inclination of rain lines, etc.) in the rainfly image. For example, the real rain-bearing image may be an image block with background information being black and with an apparent rain layer. For example, the size of the real rain-bearing image is 512 x 512, and the size of the rain layer distribution feature may be 64 x 64. In addition, the background information in the distribution characteristics of the raining layer is as small as possible, the smaller the background information, the better the background information is, and the background information is preferably black. The above-mentioned "background information" refers to other information in the image than the rain layer.

The extraction method of the distribution characteristics of the raincoats is not limited in the step, and the distribution characteristics of the raincoats can be extracted from the real images with rain in various ways, including but not limited to:

for example, in an alternative embodiment, the image block with the smallest pixel mean value may be selected as the distribution feature of the raining layer by traversing the whole real rain-carrying image in the form of a sliding window (window size is 64, and step size is 50) in one real rain-carrying image.

For another example, in another alternative embodiment, the real rain-carrying image may be input into a feature extraction network, and the rain layer distribution features of the real rain-carrying image may be output through the network.

In step 202, the rain layer distribution features are fused with the rain-free image to obtain a composite rain-bearing image.

In this step, the rain-free image is captured with certain background information in non-rainy weather, for example, the rain-free image may be a landscape photograph or a portrait photograph without rain. The rain layer distribution features are the same size as the rain-free image, e.g., each is 64 x 64.

The fusion of the distribution characteristics of the rain layer and the rain-free image can be a linear addition of the two. And adding the rain layer distribution characteristics on the rain-free image to obtain the synthetic rain-carrying image.

In step 204, a neural network for image rain removal is trained using the pair of the synthetic rain-bearing image and the rain-free image as training samples.

In the step, the neural network for removing rain from the image can be trained through the training sample set. The present embodiment does not limit the specific network structure of the image rain-removing neural network.

The training sample set may include a plurality of image pairs of a neural network for training images to remove rain, each image pair serving as a training sample, and each image pair may include the composite rain image and the rain-free image, the composite rain image being obtained by fusing the rain-free image and the rain layer distribution features. For example, an image pair of a first composite rain image and a corresponding rain-free image, and an image pair of a second composite rain image and a corresponding rain-free image may be used to train the image rain-removing neural network.

According to the training method of the neural network for image rain removal, the synthetic rain-carrying image is obtained through the rain layer distribution characteristics extracted from the real rain-carrying image, on one hand, the real rain-carrying image is easy to obtain, so that the rain layer distribution characteristics are easy to obtain, and the synthetic rain-carrying image is more convenient to obtain; on the other hand, the distribution characteristics of the rain layer are the distribution of the rain layer extracted from the real rain-carrying image, so that the obtained distribution characteristics of the rain layer are more vivid, and the synthetic rain-carrying image is more real; furthermore, the rain removing neural network of the image is trained through the more real synthetic rain-carrying image, so that the neural network has better effect when being used for rain removing processing of the real rain-carrying image.

A system architecture diagram of a training method of an image rain removing neural network is exemplarily described as follows by fig. 3, as shown in fig. 3, wherein:

a Template Module (Template Module)21 for generating a composite rain-bearing image (SynRainout);

and a rain removing network module 22, for performing network training on the image rain removing neural network according to the synthesized rain image and rain-free image (Clean).

The template module 21 and the rain removal network module 22 described below describe in detail how the neural network is trained. In the following description, the real rain-bearing image may be, for example, a first rain-bearing image or a second rain-bearing image, the composite rain-bearing image may be a first composite rain-bearing image or a second composite rain-bearing image, or the like.

First, taking the acquisition and raining process of a training sample (i.e., an image pair) as an example: referring to fig. 3, a real Rain Image (Rain Image)211 may be input into a feature extraction network 212 to obtain a Rain layer distribution feature predicted and output by the feature extraction network 212, and the predicted Rain layer distribution feature is referred to as a "Template" (TP) 213 in this embodiment. Of course, in other embodiments, the template TP may be extracted in a non-network manner, and the description of the embodiment is given by taking a network extraction manner as an example.

On the same real rain-carrying image, the distribution of rain is the same (although the position of rain is different) at different parts of the image, and the background information of the different parts is different. When extracting the template from the real rain-bearing image, the background information in the extracted template is as small as possible, for example, the size of the real rain-bearing image is 512 × 512, the background information in the template is black, and the rain layer is very obvious. In this embodiment, the real rain-carrying image 211 input to the network may be downsampled by 8 times through the feature extraction network 212, and the obtained network output may be the template 213.

In the training stage of the image rain-removing network, the network parameters of the feature extraction network 212 are adjusted. Referring to fig. 3, the present embodiment adopts a generative adaptive networks (gan) mode for training. A Label (Template Label)214 of distribution characteristics of the rain layer can be extracted from the real rain-bearing image 211, and the Label 214 can be an image block with the lowest pixel mean value found from the real rain-bearing image 211, for example, in a sliding window manner, an image block with the lowest pixel mean value is found and cut out to serve as Label information 214 of the Template 213 for supervising training. The template 213 and the label 214 may be the same size, e.g., 64 x 64 each.

The template 213 and the tag 214 can be input into the discriminator C1, and the discriminator C1 can perform discriminant training on the two, which is a process of counterlearning. And extracts the network parameters of the network 212 according to the discriminant training adjustment features, and the process of the counterlearning and adjusting the network parameters can be performed in the conventional manner of GAN, which is not described in detail. After training, the feature extraction network 212 can extract a template from the input real rain-bearing image 211, wherein the template is similar to or even consistent with the rain layer distribution features of the label 214 as much as possible.

Fig. 4 illustrates a structure of a feature extraction network 212, as shown in fig. 4, the feature extraction network 212 may include several convolutional layers and several residual blocks Res Block, and the real rain image 211 input to the feature extraction network 212 may be processed by the convolutional layers Conv and then processed by at least one residual Block (Res Block 1 to Res Block n) to obtain prediction information of the distribution features of the rain layer, that is, a template TP. Fig. 4 only shows the structure of Res Block 1, and the other residual Block Res Block may be the same as Res Block 1. After obtaining the template TP, the template TP and the label may be input to a discriminator, where the feature extraction network corresponds to a generator, and the feature extraction network is trained by using a training mechanism of a Generative adaptive network GAN (GAN including the generator and the discriminator), so that the template output by the feature extraction network is as consistent as possible with the corresponding label.

After the template TP is obtained, the TP may be applied to obtain a composite rain-bearing image. In this embodiment, a rain-free Image 215 having the same size as the template TP is obtained, for example, the rain-free Image 215 may be a 64 × 64 Image block randomly selected from a random rain-free Image (Clean Image), and the size of the rain-free Image 215 is the same as that of the template 213.

The rain-free image 215 may serve as background information and the template 213 may serve as a rain layer distribution feature. In an alternative embodiment, a composite rain image may be obtained by linearly adding the template 213 to the rain-free image 215.

Fig. 3 provides another way to obtain the composite rain image, please continue to refer to fig. 3, wherein the "+" sign indicates that TP (template) and Clean (no rain image) are linearly added, and after the addition, the linearly added image can be input to the image synthesis network 216, resulting in the composite rain image (SynRainOut)217 predicted and output by the image synthesis network 216, and the composite rain image 217 is the image output by the image synthesis network 216 after the template TP and the no rain image are linearly added.

The image synthesis network 216 described above may also be trained in a generative confrontation manner. As shown in fig. 3, a corresponding label 218 of the composite rain image may be obtained, and the label 218 may be extracted from the real rain image 211, for example, assuming that the size of the composite rain image 217 is 64 × 64, a 64 × 64 image block may be randomly selected from the real rain image 211 as the label 218 of the composite rain image. The synthesized rain image 217 and the label 218 of the synthesized rain image are input into a discriminator C2, the image synthesis network corresponds to a generator, the generator and the discriminator C2 form a generation type countermeasure network GAN, the image synthesis network 216 is trained through a training mechanism of GAN, and network parameters of the image synthesis network are adjusted, so that the synthesized rain image 217 and the label 218 output by the image synthesis network prediction have the same rain layer distribution characteristics as possible.

Still taking fig. 3 as an example, when the image synthesis network 216 and the feature extraction network 212 are trained, then, given a real rain-bearing image 211 and a rain-free image 215, a template 213 representing distribution features of a rain layer can be automatically extracted by the feature extraction network 212, and a synthesized rain-bearing image 217 can be automatically obtained by the image synthesis network 216 according to the template 213 and the rain-free image 215. The composite rain image 217 and the rain-free image 215 may be combined into an image pair as a training sample for subsequent use in training the neural network for rain removal from the image. Also, as described above, the image synthesis network 216 and the feature extraction network 212 are trained by generating antagonistic networks such that the raining layer distribution features of the template 213 and the synthesized rain-bearing image 217 are as consistent as possible with the raining layer distribution features of the real rain-bearing image 211.

In the example of fig. 3, the composite rain image 217 and the label 218 of the composite rain image may be directly input to the discriminator C2; alternatively, in another embodiment, the composite rain image and the label information may be subjected to the guide filtering process, and the composite rain image and the label information after the guide filtering process may be input to the discriminator C2.

For example, after obtaining the predicted composite rain image SynRainOut, the composite rain image and the corresponding label may be subjected to the guiding filtering process, that is, the guiding filtering process is performed on the composite rain image and the guiding filtering process is also performed on the label. After the processing, the images after the guiding filtering processing corresponding to the synthesized rain-bearing images and the labels are respectively removed, so that the detail maps (detailimages) corresponding to the synthesized rain-bearing images and the labels can be respectively obtained.

The detail map DetailImage is acquired by firstly performing guided filtering processing on the original image and then acquiring a detail map based on the original image and the guided filtering map as follows:

assuming that the original image (which may be a composite rained image SynRainOut or corresponding label information) is identified as InputImage, the guiding filter map is identified as GuildFilterImage, the detail map is identified as DetailImage, and the guiding filter operation is f (x), then:

GuildFilterImage ═ f (inputimage); # denotes: after the guiding filtering, the image becomes fuzzy and loses the information of the edge and the rain layer;

DetailImage-guildfilterimage; only the edge information and the rain layer information are contained in the # detail diagram.

The advantage of using the detail map input discriminator is that when the backgrounds of the synthesized rain-carrying images and the corresponding label information are inconsistent, the detail map processing can omit the background inconsistency in the training process, and pay more attention to consistency comparison in the aspect of rain layer distribution characteristics, so that the network training effect is better.

The composite rain image and the label information can respectively obtain the corresponding detail maps according to the method, and the detail maps are input into C2 for discrimination supervision training.

By the above method, a large number of image pairs can be obtained. One way is exemplified: suppose there are 100 true rain images and 1 million no rain images. Each template 213 extracted from the real rain-bearing image can be obtained by the trained feature extraction network 212, and a part of the better-quality templates 213 can be selected from the templates, for example, 80 templates 213 with black background information and obvious rain layers are selected. Then, assuming that the size of the template 213 is 64 × 64, one image block of 64 × 64 size in each rain-free image can be selected from the 1 ten thousand rain-free images described above, respectively, as the rain-free image to be synthesized with the template 213. For each rain-free image, one template 213 may be randomly selected from the 80 templates 213 as a template for synthesizing with the rain-free image. After the rain-free image and the template are linearly added, a composite rain-bearing image is obtained through the image synthesis network 216 which is trained in advance. The composite rain-bearing image and the corresponding rain-free image can be used as an image pair for training the image to remove rain.

In order to improve the generalization capability of the subsequently trained image rain-removing neural network, when the template is selected, templates with different rain layer distribution characteristics can be selected, and the characteristic difference value between the different rain layer distribution characteristics is greater than the difference threshold value. The measurement manner of the feature difference value is not limited in this embodiment, for example, the similarity between two images may be compared, or the difference between two templates in the aspects of rain line inclination, rain line thickness, and the like may be compared. The image pair of the neural network used for training the image to remove rain is made to include various rain layer distribution characteristics as much as possible.

With continued reference to fig. 3, the following describes a training process of the image rain-removing neural network, wherein the training process is to train the specific network structure of the image rain-removing neural network 221 according to the embodiment without limitation by synthesizing the image pairs composed of the rain-bearing image and the rain-free image.

FIG. 5 provides an exemplary network architecture of an image rain-removing neural network, as shown in FIG. 5, the composite rain-carrying image may be subjected to a cyclic rain-removing process by a cyclic convolution neural network LSTM to increase the context information extraction capability, and the network may be performed cyclically (e.g., cyclically)Other number of rings 3, 6). Where y represents the composite rain image of the neural network for rain removal of the initial input image, x^t-1Representing the image, x, output from the previous cycle^tAn image representing the output of the present cycle, C represents x^t-1And carrying out image feature splicing with y. When the loop is repeated a predetermined number of times, e.g. 6 times, x is finally output^tThe image showing the completion of rain removal corresponds to the predicted rain removal image (RainOut)222 in fig. 3.

With continued reference to FIG. 5, after LSTM, the LSTM output image may be processed through ResBlock and the Squeeze-Excitation network SE (Squeeze-Excitation). Fig. 6 illustrates a network structure of the ResBlocks and SE processing, in which modules other than SEBlock belong to ResBlocks.

With continued reference to fig. 3, after the predicted rain-removed image 222 is obtained, the network parameters of the image rain-removed neural network 221 may be adjusted based on the difference between the predicted rain-removed image 222 and the rain-free image 215. Fig. 3 illustrates that it can be supervised that the predicted rain-removed image 222 is as similar or even identical as possible to the rain-free image 215, both by means of the structural similarity loss function SSIMLoss and by means of the discriminator C3. It is understood that in actual training, SSIMLoss and arbiter C3 may choose to use at least one of the two.

For example, this embodiment may use SSIMLoss and a discriminator C3 together, and may obtain the difference between the predicted rain-removed image and the rain-free image through a structural similarity loss function, and adjust the network parameters of the image rain-removed neural network based on the difference. And inputting the predicted rain-removing image and the rain-free image into a discriminator, and adjusting the network parameters of the image rain-removing neural network according to the output result of the discriminator.

Referring to fig. 3, if a real rain image and a rain-free image are given, a synthetic rain image can be automatically obtained through the network structure of fig. 3, and the pair of the synthetic rain image and the rain-free image is automatically used as the image pair of the training sample, and the neural network for inputting the images to remove rain is used for network training. Of course, the embodiment is not limited to this, and each part of the system shown in fig. 3 may be divided and executed separately.

In addition, the above describes the processing flow of one image pair under the architecture of fig. 3, when the image degraining neural network is actually implemented, there may be a large number of image pairs, training may be performed in groups, for example, the training sample set is divided into a plurality of batchs, and the network parameters of the image degraining neural network are adjusted by combining the loss functions of the respective image pairs included in each batch.

It should be noted that, although the image rain-removing neural network of the embodiment of the present disclosure is called "image rain-removing" in name, the practical application scenario is not limited to "rain-removing", for example, snow-removing may also be performed. In other application scenarios, the network may also be referred to by the name "image rain-removing neural network," i.e., the network of the present disclosure is not limited to rain removal.

Fig. 7 provides an image processing method, which may also be used for rain, snow, etc. scenes. As shown in fig. 7, the method may include the following processes:

in step 700, an input image to be processed is acquired, the input image including a rain layer and a background.

In step 702, the input image is subjected to a rain removal process to obtain a rain-free image including a background of the input image.

In the step, the rain layer in the input image is removed, so that the input image only has a background and becomes a rain-free image. For example, in practical implementation, before performing the rain removing process on the input image, it may be detected whether the input image includes a rain layer. In this embodiment, the method for detecting a raining layer is not limited, and if the raining layer is included, the raining removal process is performed; otherwise, if the rain layer is not included, the task processing in step 704 may be directly performed. In addition, the rain removing processing of the embodiment may use an image rain removing neural network to remove rain, and the image rain removing neural network may be obtained by a training method according to any embodiment of the disclosure.

In step 704, processing a predetermined task on the rain-free image; the predetermined task includes at least one of: object detection, object tracking, scene segmentation, object classification, object identification.

According to the image processing method, the reserved task processing is performed after the rain layer in the image is removed, so that the task processing process is not interfered by the rain layer of the original image, and the quality and the effect of the task processing are improved.

Fig. 8 provides an image processing apparatus that can execute the image processing method of any of the embodiments of the present disclosure. As shown in fig. 8, the apparatus may include: a feature extraction module 81 and an image fusion module 82.

A feature extraction module 81, configured to extract a first raining layer distribution feature in the first rain-carrying image;

and an image fusion module 82, configured to fuse the first raining layer distribution characteristics with at least one rain-free image respectively to obtain at least one first synthesized rain-carrying image.

Referring to fig. 9, fig. 9 provides another image processing apparatus, which may further include, on the basis of the structure shown in fig. 8: image pair processing module 83.

The image fusion module 82 is specifically configured to fuse the first raining layer distribution characteristic and the plurality of different rain-free images respectively to obtain a plurality of different first synthetic rain-carrying images;

the device further comprises: an image pair processing module 83, configured to, after the obtaining the plurality of different first composite rain images, further include: and combining the plurality of different rain-free images with the corresponding first synthesized rain-carrying images respectively to obtain a plurality of pairs of different image pairs.

In one example, the feature extraction module 81 is further configured to extract a second raining layer distribution feature in the second rain-carrying image, where a difference between the second raining layer distribution feature and the first raining layer distribution feature satisfies a predetermined requirement;

the image fusion module 82 is further configured to fuse the second raining layer distribution characteristics with at least one rain-free image, respectively, to obtain at least one second composite rain-carrying image.

In an example, the image fusion module 82 is further configured to fuse the second raining layer distribution characteristics with a plurality of different rain-free images respectively to obtain a plurality of different second composite rain-carrying images;

the image pair processing module 83 is further configured to combine the plurality of different rain-free images with the respective corresponding second composite rain images to obtain a plurality of pairs of different image pairs.

In one example, the first rain-bearing image or the second rain-bearing image is a really captured rain-bearing image, and a background portion of non-rain in the first rain-bearing image or the second rain-bearing image is a monochrome patch.

In one example, the first rain-bearing image or the second rain-bearing image is a sub image block extracted from a real captured rain-bearing image, and a background portion of the sub image block, which is not rain, is a monochrome patch.

In one example, the apparatus further comprises: a first network training module 84 for training the feature extraction network by: inputting the first rain-carrying image into the feature extraction network to obtain the rain layer distribution feature predicted and output by the feature extraction network; taking the predicted and output rain layer distribution characteristics and corresponding labels as the input of a discriminator of a generative confrontation network, and adjusting the network parameters of the characteristic extraction network according to the output result of the discriminator; the first rain layer distribution characteristic is extracted through the characteristic extraction network.

In an example, the image fusion module 82 is specifically configured to linearly add the first raining layer distribution characteristic to one of the rainless images to obtain a corresponding one of the first composite rained images.

In one example, the image fusion module 82 is specifically configured to linearly add the first rainfly distribution characteristic to one of the rainless images; and inputting the linearly added images into an image synthesis network to obtain one first synthesized rain-carrying image output by the image synthesis network.

In one example, the apparatus further comprises: a second network training module 85, configured to train the image synthesis network by: linearly adding the first rain layer distribution characteristic to the rain-free image; inputting the images subjected to linear addition into an image synthesis network to obtain a synthesized rain-carrying image predicted and output by the image synthesis network; and taking the synthesized rain-bearing image and the corresponding label as the input of a discriminator of the generating type confrontation network, and adjusting the network parameters of the image synthesis network according to the output result of the discriminator.

In one example, the second network training module 85 is further configured to perform a guiding filtering process on the composite rain image and the corresponding label, respectively, before the input of the discriminator for using the composite rain image and the corresponding label as the generating countermeasure network; and removing the images after the guiding filtering processing respectively corresponding to the synthesized rain-carrying images and the corresponding labels to obtain respective corresponding detail maps, wherein the detail maps are used for inputting into the discriminator.

In one example, the apparatus further comprises: a third network training module 86 trains an image rain removing neural network using the plurality of pairs of different image pairs.

In an example, the third network training module 86 is specifically configured to: inputting the synthesized rain-carrying image in the image pair into the rain-removing neural network of the image to obtain a predicted rain-removing image; the composite rain image is the first composite rain image or the second composite rain image; adjusting network parameters of the image rain-removing neural network based on a difference between the predicted rain-removing image and the corresponding rain-free image of the image pair.

In one example, the third network training module 86, when used to adjust the network parameters of the image rain removing neural network, includes: obtaining a difference between the predicted rain-removed image and the rain-free image through a structural similarity loss function, and adjusting network parameters of a neural network for removing rain of the image based on the difference; and/or inputting the predicted rain removing image and the rain-free image into a discriminator, and adjusting the network parameters of the image rain removing neural network according to the output result of the discriminator.

Fig. 10 provides an image processing apparatus, which may include, as shown in fig. 10: an image acquisition module 1001, a rain removal processing module 1002 and a task processing module 1003.

An image obtaining module 1001 configured to obtain an input image to be processed, where the input image includes a rain layer and a background;

a rain removing processing module 1002, configured to perform rain removing processing on the input image to obtain a rain-free image including a background of the input image;

a task processing module 1003, configured to perform processing of a predetermined task on the rainless image; the predetermined task includes at least one of: object detection, object tracking, scene segmentation, object classification, object identification.

In an example, the rain removing module 1002 is further configured to detect and determine that the input image includes the rain layer before performing rain removing processing on the input image.

In one example, the rain removal processing module 1002 is specifically configured to: according to a pre-trained neural network for image rain removal, performing rain removal processing on the input image to obtain a rain-free image comprising the background of the input image; the image rain-removing neural network is obtained by training through the training method of any embodiment of the disclosure.

The present disclosure also provides an electronic device comprising a memory for storing computer instructions executable on a processor, and a processor for implementing the image processing method of any of the embodiments of the present disclosure when executing the computer instructions.

The present disclosure also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the image processing method according to any of the embodiments of the present disclosure.

One skilled in the art will appreciate that one or more embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, one or more embodiments of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

Embodiments of the present disclosure also provide a computer-readable storage medium, on which a computer program may be stored, where the computer program, when executed by a processor, implements the steps of the method for training a neural network for word recognition described in any of the embodiments of the present disclosure, and/or implements the steps of the method for word recognition described in any of the embodiments of the present disclosure. Wherein "and/or" means having at least one of the two, e.g., "multi and/or B" includes three schemes: poly, B, and "poly and B".

The embodiments in the disclosure are described in a progressive manner, and the same and similar parts among the embodiments can be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the data processing apparatus embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and for the relevant points, reference may be made to part of the description of the method embodiment.

The foregoing description of specific embodiments of the present disclosure has been described. Other embodiments are within the scope of the following claims. In some cases, the acts or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.

Embodiments of the subject matter and functional operations described in this disclosure may be implemented in: digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this disclosure and their structural equivalents, or a combination of one or more of them. Embodiments of the subject matter described in this disclosure can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on a tangible, non-transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode and transmit information to suitable receiver apparatus for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

The processes and logic flows described in this disclosure can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPG multi (field programmable gate array) or a SIC multi (application-specific integrated circuit).

Computers suitable for executing computer programs include, for example, general and/or special purpose microprocessors, or any other type of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory and/or a random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer does not necessarily have such a device. Further, the computer may be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PD multi), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device such as a Universal Serial Bus (USB) flash drive, to name a few.

Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., an internal hard disk or a removable disk), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

Although this disclosure contains many specific implementation details, these should not be construed as limiting the scope of any disclosure or of what may be claimed, but rather as merely describing features of particular embodiments of the disclosure. Certain features that are described in this disclosure in the context of separate embodiments can also be implemented in combination in a single embodiment. In other instances, features described in connection with one embodiment may be implemented as discrete components or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. Further, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous.

The above description is only for the purpose of illustrating the preferred embodiments of the present disclosure, and is not intended to limit the scope of the present disclosure, which is to be construed as being limited by the appended claims.

Claims

1. An image processing method, characterized in that the method comprises:

2. The method of claim 1,

the fusing the first rain layer distribution characteristics with at least one rain-free image respectively to obtain at least one first synthesized rain-carrying image, comprising: respectively fusing the first rain layer distribution characteristic and the plurality of different rain-free images to obtain a plurality of different first synthetic rain-carrying images;

after the obtaining a plurality of different first composite rain images, the method further comprises: and combining the plurality of different rain-free images with the corresponding first synthesized rain-carrying images respectively to obtain a plurality of pairs of different image pairs.

3. The method according to claim 1 or 2, wherein the first rain layer distribution feature is extracted through a feature extraction network; the training process of the feature extraction network comprises the following steps:

inputting the first rain-carrying image into the feature extraction network to obtain the rain layer distribution feature predicted and output by the feature extraction network;

and taking the predicted and output rain layer distribution characteristics and the corresponding labels as the input of a discriminator of the generative confrontation network, and adjusting the network parameters of the characteristic extraction network according to the output result of the discriminator.

4. The method according to any one of claims 1 to 3, further comprising: training an image rain-removing neural network using the plurality of pairs of different image pairs.

5. An image processing method, characterized in that the method comprises:

processing a preset task on the rain-free image;

6. An image processing apparatus, characterized in that the apparatus comprises:

7. The apparatus of claim 6,

the image fusion module is specifically configured to fuse the first raining layer distribution characteristic and a plurality of different rain-free images respectively to obtain a plurality of different first synthetic rain-carrying images;

the device further comprises: an image pair processing module configured to, after the obtaining the plurality of different first composite rain images, further include: and combining the plurality of different rain-free images with the corresponding first synthesized rain-carrying images respectively to obtain a plurality of pairs of different image pairs.

8. An image processing apparatus, characterized in that the apparatus comprises:

the image acquisition module is used for acquiring an input image to be processed, wherein the input image comprises a rain layer and a background;

the rain removing processing module is used for carrying out rain removing processing on the input image to obtain a rain-free image comprising the background of the input image;

the task processing module is used for processing a preset task on the rain-free image; the predetermined task includes at least one of: object detection, object tracking, scene segmentation, object classification, object identification.

9. An electronic device, comprising a memory for storing computer instructions executable on a processor, the processor being configured to implement the method of any one of claims 1 to 4 when executing the computer instructions or to implement the method of claim 5.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method of any one of claims 1 to 4, or carries out the method of claim 5.