WO2021164234A1 - 图像处理方法以及图像处理装置 - Google Patents

图像处理方法以及图像处理装置 Download PDF

Info

Publication number
WO2021164234A1
WO2021164234A1 PCT/CN2020/112619 CN2020112619W WO2021164234A1 WO 2021164234 A1 WO2021164234 A1 WO 2021164234A1 CN 2020112619 W CN2020112619 W CN 2020112619W WO 2021164234 A1 WO2021164234 A1 WO 2021164234A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
channel
processing
raw domain
processed
Prior art date
Application number
PCT/CN2020/112619
Other languages
English (en)
French (fr)
Inventor
刘林
贾旭
刘健庄
陈帅军
田奇
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to EP20920655.6A priority Critical patent/EP4109392A4/en
Publication of WO2021164234A1 publication Critical patent/WO2021164234A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4015—Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/77—Retouching; Inpainting; Scratch removal
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20004—Adaptive image processing
    • G06T2207/20012—Locally adaptive
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • This application relates to the field of artificial intelligence, and more specifically, to image processing methods and image processing devices in the field of computer vision.
  • Computer vision is an inseparable part of various intelligent/autonomous systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military. It is about how to use cameras/video cameras and computers to obtain What we need is the knowledge of the data and information of the subject. To put it vividly, it is to install eyes (camera/camcorder) and brain (algorithm) on the computer to replace the human eye to identify, track and measure the target, so that the computer can perceive the environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multi-dimensional data.
  • computer vision is to use various imaging systems to replace the visual organs to obtain input information, and then the computer replaces the brain to complete the processing and interpretation of the input information.
  • the ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously.
  • demosaicing refers to the restoration/reconstruction of a full-color image from the incompletely sampled color signal output by the color filter array photosensitive device Process: Because the image containing noise will affect the display effect of the image, as well as the analysis and recognition of the image, the noise existing in the color filter array data can be removed by image denoising.
  • the joint demosaicing and denoising method can be obtained by sampling a single-channel Bayer image to obtain a four-channel image and a three-channel mask image, and then fusing the four-channel image and the three-channel mask image.
  • the present application provides an image processing method and an image processing device, which can avoid artifacts to a certain extent, and improve the image quality of raw domain images after combined demosaicing and denoising.
  • an image processing method including: acquiring a channel image corresponding to an original raw domain image to be processed, wherein the channel image includes at least two first channel images; performing image restoration on the first channel image Processing to obtain a restored first channel image; performing image restoration processing on the raw domain image to be processed according to the restored first channel image to obtain a raw domain image after combined demosaicing and denoising processing.
  • the raw image to be processed may refer to the raw image collected by the sensor, and the raw image to be processed refers to the raw image that has not been processed by an image signal processor (ISP).
  • ISP image signal processor
  • the aforementioned raw domain image to be processed may refer to an image in Bayer format with demosaicing and denoising requirements, that is, there may be some additional information in the image, which may be noise or artifacts.
  • the image restoration processing refers to the technology of restoring the lost parts of the image and reconstructing them based on the image information.
  • Image restoration processing can try to estimate the original image information, repair and improve the damaged area, thereby improving the visual quality of the image.
  • the image restoration processing may include convolution processing, reconstruction processing, and so on.
  • performing image restoration processing on the first channel image described above to obtain the restored first channel image may refer to performing preliminary image restoration processing on the first channel image to obtain the initially restored first channel image.
  • the channel image of the raw domain image to be processed may be subjected to pre-image restoration to obtain the preliminary restored channel image; based on the preliminary restored channel image, the raw domain image to be processed is guided to perform image restoration processing,
  • the image processing method of the embodiment of the present application can avoid the appearance of color texture to a certain extent, so that the processed raw domain image can retain the image details and improve the image quality.
  • it further includes: acquiring a dense image of the raw domain image to be processed, wherein the dense image is used to indicate textures of different frequencies in the raw domain image to be processed area;
  • the performing image restoration processing on the raw domain image to be processed according to the restored first channel image to obtain the raw domain image after joint demosaicing and denoising processing includes: The raw domain image to be processed is subjected to convolution processing to obtain the first image feature; through the restored first channel image, the first image feature is guided to perform convolution processing to obtain the combined demosaicing and denoising raw Domain image.
  • the above dense map can be used to indicate the texture area of different frequencies in the raw domain image to be processed; the dense map has a higher response to the high-frequency texture area and lower response to the low-frequency texture area, so that it can be based on the dense map Distinguish the different texture areas in the raw image to be processed.
  • the above-mentioned first image feature may refer to the image feature obtained by preliminary restoration of the raw domain image to be processed, that is, there may be some noise and artifacts in the first image feature, and no real joint demosaicing and denoising has been obtained. After the raw domain image.
  • the neural network by inputting the dense image into the neural network, the neural network can distinguish different texture regions in the raw image to be processed when performing convolution processing on the raw image to be processed. Texture dense areas and sparse texture areas can improve the denoising effect and maintain edge details.
  • preliminary image restoration may be performed on the raw image to be processed, that is, the convolution operation is performed on the raw image to be processed to obtain the output first image feature, and the first image feature A clean image that is not truly clean may still have some noise or artifacts; further, it is possible to perform adaptive convolution processing on the first image feature according to the initially restored channel image, so as to completely remove the noise and mosaic in the image feature to obtain the final Raw domain image after combined demosaicing and denoising.
  • the first image feature is guided to perform convolution processing through the restored first channel image to obtain the combined demosaicing and denoising raw Domain images, including:
  • the pre-repaired image feature can be guided by the pixel spatial distribution of the restored channel image, that is, the pre-repaired channel image, so that the pre-repaired image feature is subjected to further convolution processing, that is, the pre-repaired image
  • the feature can also satisfy to a certain extent the same or similar pixel spatial distribution as the pre-repaired channel image, where the pixel spatial distribution can be used to indicate the correlation of different pixel point distributions.
  • it further includes: obtaining a random noise map
  • the obtaining the first image feature by performing convolution processing on the dense image and the raw domain image to be processed includes: convoluting the dense image, the random noise image, and the raw domain image to be processed Product processing to obtain the first image feature.
  • a random noise map can also be input to the neural network, so as to improve the denoising effect of the raw domain image to be processed by the neural network.
  • the first channel image is a green G channel image.
  • the channel image corresponding to the raw domain image to be processed may refer to the RGGB channel image, or the channel image corresponding to the raw domain image to be processed may also refer to the GBBR channel image; then the first channel image may be Refers to the G channel image.
  • the first channel image is a yellow Y channel image.
  • the channel image corresponding to the raw domain image to be processed may refer to the RYYB channel image
  • the first channel image may refer to the Y channel image
  • the first channel image may refer to the channel image with the most image information among the multiple channel images corresponding to the raw domain image to be processed; wherein, the image information may refer to the high frequency information in the image; for example, Refers to the detailed information, texture information and edge information in the image.
  • an image processing method using an electronic device with a display screen and a camera characterized in that it includes: detecting an operation instructing the camera by a user; in response to the operation, displaying an output image in the display screen , Wherein the output image is obtained based on the original raw domain image after joint demosaicing and denoising, and the raw domain image after joint demosaicing and denoising is for the raw domain image to be processed collected by the camera
  • the raw domain image obtained after image restoration processing, the restored first channel image is used in the image restoration process of the raw domain image to be processed, and the raw domain image to be processed includes at least two first channel images,
  • the restored first channel image refers to a channel image obtained by performing image restoration processing on the first channel image.
  • the detecting the operation of the camera instructed by the user includes:
  • An operation indicating a first processing mode by the user is detected, and the first processing mode is used to perform image restoration processing on the raw domain image to be processed; or,
  • An operation for instructing shooting by the user is detected.
  • the process of performing image restoration processing on the raw domain image to be processed further includes: acquiring a dense image of the raw domain image to be processed, wherein the dense image is used for Indicate the texture regions of different frequencies in the raw domain image to be processed; the first image feature is obtained by convolution processing the dense image and the raw domain image to be processed; the first channel image after the restoration is obtained Guide the first image feature to perform convolution processing to obtain the raw domain image after the joint demosaicing and denoising.
  • the first image feature is guided to perform convolution processing through the restored first channel image to obtain the combined demosaicing and denoising raw
  • the domain image includes: performing convolution processing on the first image feature according to the pixel spatial distribution of the restored first channel image to obtain the raw domain image after the joint demosaicing and denoising, wherein the The spatial distribution of pixels is used to indicate the relevance of the distribution of different pixel points in the restored first channel image.
  • it further includes: obtaining a random noise map
  • the obtaining the first image feature by performing convolution processing on the dense image and the raw domain image to be processed includes: convoluting the dense image, the random noise image, and the raw domain image to be processed Product processing to obtain the first image feature.
  • the first channel image is a green G channel image.
  • the first channel image is a yellow Y channel image.
  • an image processing device which includes functional modules/units for executing the first aspect and the image processing method in any one of the possible implementation manners of the first aspect.
  • an image processing device which includes functional modules/units for executing the second aspect and the image processing method in any one of the possible implementation manners of the second aspect.
  • a computer-readable medium stores program code for device execution, and the program code includes image processing for executing the first aspect or any one of the implementation manners of the first aspect method.
  • a computer-readable medium stores program code for execution by a device, and the program code includes an image used to execute the second aspect or any one of the implementation manners of the second aspect Approach.
  • a computer program product includes: computer program code, which when the computer program code runs on a computer, causes the computer to execute the methods in the above aspects.
  • the above-mentioned computer program code may be stored in whole or in part on a first storage medium, where the first storage medium may be packaged with the processor, or may be packaged separately with the processor. Specific restrictions.
  • a chip in an eighth aspect, includes a processor and a data interface.
  • the processor reads instructions stored in a memory through the data interface, and executes any one of the first aspect or the first aspect.
  • the image processing method in the implementation mode is provided.
  • the chip may further include a memory in which instructions are stored, and the processor is configured to execute instructions stored on the memory.
  • the processor is configured to execute the image processing method in the first aspect or any one of the implementation manners in the first aspect.
  • a chip in a ninth aspect, includes a processor and a data interface, the processor reads instructions stored in a memory through the data interface, and executes any one of the second aspect or the second aspect described above The image processing method in the implementation mode.
  • the chip may further include a memory in which instructions are stored, and the processor is configured to execute instructions stored on the memory.
  • the processor is configured to execute the image processing method in the second aspect or any one of the implementation manners in the second aspect.
  • FIG. 1 is a schematic diagram of an application scenario of an image processing method provided by an embodiment of the present application
  • Figure 2 is a schematic diagram of an application scenario provided by an embodiment of the present application.
  • FIG. 3 is a schematic diagram of an application scenario provided by an embodiment of the present application.
  • FIG. 4 is a schematic diagram of an application scenario provided by an embodiment of the present application.
  • Figure 5 is a schematic structural diagram of a system architecture provided by an embodiment of the present application.
  • Fig. 6 is a schematic diagram of a convolutional neural network structure provided by an embodiment of the present application.
  • FIG. 7 is a schematic diagram of a chip hardware structure provided by an embodiment of the present application.
  • FIG. 8 is a schematic diagram of a system architecture provided by an embodiment of the present application.
  • Fig. 9 is a schematic diagram of a joint demosaicing and denoising system provided by an embodiment of the present application.
  • FIG. 10 is a schematic flowchart of an image processing method provided by an embodiment of the present application.
  • FIG. 11 is a schematic flowchart of an image joint demosaicing and denoising method provided by an embodiment of the present application.
  • FIG. 12 is a schematic diagram of preliminary image restoration provided by an embodiment of the present application.
  • FIG. 13 is a schematic diagram of a repaired channel image provided by the implementation of this application.
  • FIG. 14 is a schematic diagram of obtaining a dense image of raw domain images to be processed provided by the implementation of this application.
  • 15 is a schematic diagram of adaptive convolution through repaired channel images provided by an embodiment of the present application.
  • FIG. 16 is a schematic flowchart of an image processing method provided by an embodiment of the present application.
  • FIG. 17 is a schematic diagram of a set of display interfaces provided by an embodiment of the present application.
  • FIG. 18 is a schematic diagram of another set of display interfaces provided by an embodiment of the present application.
  • FIG. 19 is a schematic diagram of another display interface provided by an embodiment of the present application.
  • FIG. 20 is a schematic diagram of another display interface provided by an embodiment of the present application.
  • FIG. 21 is a schematic block diagram of an image processing device provided by an embodiment of the present application.
  • FIG. 22 is a schematic block diagram of an image processing device provided by an embodiment of the present application.
  • FIG. 23 is a schematic block diagram of an image processing device provided by an embodiment of the present application.
  • Fig. 1 is a schematic diagram of an application scenario of an image processing method provided by an embodiment of the present application.
  • the image processing method of the embodiment of the present application can be applied to a smart terminal.
  • the raw raw domain image to be processed collected by a camera in the smart device can be image repaired, and the raw raw domain image to be processed can be obtained.
  • the raw domain image after the joint demosaicing and denoising process is processed, and the subsequent image processing can be performed on the raw domain image after the joint demosaicing and denoising process to obtain the output image; among them, the subsequent image processing can include the joint
  • the raw domain image after demosaicing and denoising undergoes image processing such as white balance, color correction, and tone mapping; the raw domain image can refer to an image in Bayer format.
  • the image restoration processing refers to the technology of restoring the lost parts of the image and reconstructing them based on the image information. Image restoration processing can try to estimate the original image information, repair and improve the damaged area, thereby improving the visual quality of the image.
  • the above-mentioned smart terminal may be mobile or fixed.
  • the smart terminal may be a mobile phone with image processing functions, a tablet personal computer (TPC), a media player, a smart TV, or a laptop computer.
  • computer LC), personal digital assistant (PDA), personal computer (PC), camera, video camera, smart watch, augmented reality (AR)/virtual reality (VR), Wearable devices (WD) or self-driving vehicles, etc., are not limited in the embodiment of the present application.
  • Application scenario 1 Smart terminal camera field
  • the image processing method of the embodiment of the present application can be applied to the shooting of a smart terminal device (for example, a mobile phone).
  • a smart terminal device for example, a mobile phone.
  • images with dense texture may be encountered.
  • the image information processor in the smart terminal may cause problems such as color artifacts or zipper-like noise in the acquired images.
  • the image method of the embodiment of the present application that is, the method of joint demosaicing and denoising, can perform image restoration processing on the acquired original raw image with poor quality to obtain an output image (or output video) with improved visual quality.
  • the color image portion is represented by oblique line filling.
  • the image processing method of the embodiment of the present application may be used to perform image restoration processing on the acquired original raw domain image when the smart terminal is taking real-time photographs, and the raw domain image obtained after joint demosaicing and denoising processing may be obtained. And perform subsequent image processing on the raw domain image after joint demosaicing and denoising to obtain the output image; among them, the subsequent image processing may include performing white balance, color correction, and color correction on the raw domain image after joint demosaicing and denoising.
  • Image processing such as tone mapping; display the output image on the screen of the smart terminal.
  • the original raw domain image (for example, color filter array signal) that can be obtained can be subjected to image restoration processing by the image processing method of the embodiment of the present application to obtain the raw domain after joint demosaicing and denoising, and combine
  • the raw domain image after demosaicing and denoising is subjected to subsequent image processing to obtain an output image.
  • the subsequent image processing may include white balance, color correction, tone mapping, etc. of the raw domain image after joint demosaicing and denoising Image processing; save the output image to the album of the smart terminal.
  • this application proposes an image processing method applied to an electronic device with a display screen and a camera, including: detecting an operation instructed by a user to the camera; in response to the operation, displaying an output image in the display screen , Wherein the output image is obtained based on the original raw domain image after joint demosaicing and denoising, and the raw domain image after joint demosaicing and denoising is for the raw domain image to be processed collected by the camera
  • the raw domain image obtained after image restoration processing, the restored first channel image is used in the image restoration process of the raw domain image to be processed, and the raw domain image to be processed includes at least two first channel images,
  • the restored first channel image refers to a channel image obtained by performing image restoration processing on the first channel image.
  • image processing method provided by the embodiment of the present application is also applicable to the extension, limitation, explanation and description of the related content of the image processing method in the related embodiments in FIG. 5 to FIG. 15, which will not be repeated here.
  • the image processing method of the embodiment of the present application can be applied to the field of automatic driving.
  • it can be applied to the navigation system of an autonomous vehicle.
  • the image processing method in this application can enable the autonomous vehicle to obtain a low-quality raw road image (or original Raw domain road video) perform image restoration processing, that is, combine the raw domain image after demosaicing and denoising, and perform subsequent image processing on the raw domain image after combined demosaicing and denoising to obtain the processed road image (or road video). ), so as to realize the safety of the self-driving vehicle, where the subsequent image processing may include image processing such as white balance, color correction, and tone mapping on the raw domain image after joint demosaicing and denoising.
  • the present application provides an image processing method, including: acquiring a channel image corresponding to an original raw domain road image, wherein the channel image includes at least two first channel images; Image restoration processing to obtain a restored first channel image; image restoration processing is performed on the raw domain road image according to the restored first channel image to obtain a raw domain road image after joint demosaicing and denoising.
  • image processing method provided by the embodiment of the present application is also applicable to the extension, limitation, explanation and description of the related content of the image processing method in the related embodiments in FIG. 5 to FIG. 15, which will not be repeated here.
  • the image processing method of the embodiment of the present application can be applied to the field of safe cities, for example, the field of security.
  • the image processing method of the embodiment of the present application can be applied to the surveillance image processing of a safe city.
  • raw domain images or raw domain videos collected by surveillance equipment in public places are often affected by factors such as weather and distance. There are problems such as blurred images and low image quality.
  • the image processing method of the present application it is possible to perform image restoration processing on the collected raw domain image, that is, joint demosaicing and denoising processing, to obtain the raw domain image after joint demosaicing and denoising; further, the joint demosaicing and denoising
  • the raw domain image is subjected to subsequent image processing to obtain a processed street scene image.
  • the subsequent image processing may include image processing such as white balance, color correction, tone mapping, etc., on the raw domain image after joint demosaicing and denoising ;
  • the public security personnel can recover important information such as license plate numbers and clear faces, and provide important clues for the detection of cases.
  • this application provides an image processing method, including: acquiring a channel image corresponding to an original raw domain street view image, wherein the channel image includes at least two first channel images; Image restoration processing to obtain a restored first channel image; image restoration processing is performed on the raw domain street view image according to the restored first channel image to obtain a raw domain street view image after combined demosaicing and denoising processing.
  • image processing method provided by the embodiment of the present application is also applicable to the extension, limitation, explanation and description of the related content of the image processing method in the related embodiments in FIG. 5 to FIG. 15, which will not be repeated here.
  • a neural network can be composed of neural units.
  • a neural unit can refer to an arithmetic unit that takes x s and intercept 1 as inputs.
  • the output of the arithmetic unit can be:
  • s 1, 2,...n, n is a natural number greater than 1
  • W s is the weight of x s
  • b is the bias of the neural unit.
  • f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal.
  • the output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function.
  • a neural network is a network formed by connecting multiple above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit.
  • the input of each neural unit can be connected with the local receptive field of the previous layer to extract the characteristics of the local receptive field.
  • the local receptive field can be a region composed of several neural units.
  • Deep neural network also known as multi-layer neural network
  • the DNN is divided according to the positions of different layers.
  • the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer.
  • the first layer is the input layer
  • the last layer is the output layer
  • the number of layers in the middle are all hidden layers.
  • the layers are fully connected, that is to say, any neuron in the i-th layer must be connected to any neuron in the i+1th layer.
  • DNN looks complicated, it is not complicated as far as the work of each layer is concerned. Simply put, it is the following linear relationship expression: in, Is the input vector, Is the output vector, Is the offset vector, W is the weight matrix (also called coefficient), and ⁇ () is the activation function.
  • Each layer is just the input vector After such a simple operation, the output vector is obtained Due to the large number of DNN layers, the coefficient W and the offset vector The number is also relatively large.
  • DNN The definition of these parameters in DNN is as follows: Take coefficient W as an example: Suppose in a three-layer DNN, the linear coefficients from the fourth neuron in the second layer to the second neuron in the third layer are defined as The superscript 3 represents the number of layers where the coefficient W is located, and the subscript corresponds to the output third-level index 2 and the input second-level index 4.
  • the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as
  • Convolutional neural network (convolutional neuron network, CNN) is a deep neural network with a convolutional structure.
  • the convolutional neural network contains a feature extractor composed of a convolutional layer and a sub-sampling layer.
  • the feature extractor can be regarded as a filter.
  • the convolutional layer refers to the neuron layer that performs convolution processing on the input signal in the convolutional neural network.
  • a neuron can be connected to only part of the neighboring neurons.
  • a convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. Neural units in the same feature plane share weights, and the shared weights here are the convolution kernels.
  • Sharing weight can be understood as the way of extracting image information has nothing to do with location.
  • the convolution kernel can be initialized in the form of a matrix of random size. In the training process of the convolutional neural network, the convolution kernel can obtain reasonable weights through learning. In addition, the direct benefit of sharing weights is to reduce the connections between the layers of the convolutional neural network, and at the same time reduce the risk of overfitting.
  • the neural network can use the back propagation (BP) algorithm to modify the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, forwarding the input signal to the output will cause error loss, and the parameters in the initial neural network model are updated by backpropagating the error loss information, so that the error loss is converged.
  • the back-propagation algorithm is a back-propagation motion dominated by error loss, and aims to obtain the optimal parameters of the neural network model, such as the weight matrix.
  • JDD Joint demosaicing and denoising
  • Image noise is a random change of brightness or color information in an image (the object itself is not photographed), and it is usually a manifestation of electronic noise. It is generally produced by the sensors and circuits of scanners or digital cameras, and may also be produced by the inevitable shot noise of film grains or ideal photodetectors. Image noise is an undesirable by-product in the image capture process, which brings errors and additional information to the image.
  • the number of pixels may be the sum of r pixels and g pixels.
  • Fig. 5 shows a system architecture 100 provided by an embodiment of the present application.
  • Fig. 5 shows a system architecture 100 provided by an embodiment of the present application.
  • the data collection device 160 is used to collect training data.
  • the image processing model also referred to as a joint demosaicing and denoising network
  • the training data collected by the data collection device 160 can be training images.
  • the training data for training the image processing model in the embodiment of the present application may include an original image and a sample image corresponding to the original image.
  • the original image may refer to the raw domain image that has not passed through the image signal processing pipeline; for example, the Bayer format image
  • the sample image may refer to the output image after image restoration processing, for example, it may refer to the original raw domain image
  • the image after the joint demosaicing and denoising processing has been improved.
  • the data collection device 160 stores the training data in the database 130, and the training device 120 trains based on the training data maintained in the database 130 to obtain the target model/rule 101 (that is, the image processing model in the embodiment of the present application) .
  • the training device 120 inputs the training data into the image processing model until the difference between the predicted image output by the training image processing model and the sample image satisfies a preset condition (for example, the difference between the predicted image and the sample image is less than a certain threshold, or the predicted image and The difference of the sample image remains unchanged or no longer decreases), thereby completing the training of the target model/rule 101.
  • a preset condition for example, the difference between the predicted image and the sample image is less than a certain threshold, or the predicted image and The difference of the sample image remains unchanged or no longer decreases
  • the image processing model used to execute the image processing method in the embodiment of the present application can realize end-to-end training.
  • the image processing model can combine the input image and the input image corresponding to the joint demosaicing and denoising image ( For example, true value images) realize end-to-end training.
  • the target model/rule 101 is obtained by training an image processing model, that is, the image processing model may refer to a model that combines demosaicing and denoising. It should be noted that in actual applications, the training data maintained in the database 130 may not all come from the collection of the data collection device 160, and may also be received from other devices.
  • the training device 120 does not necessarily perform the training of the target model/rule 101 completely based on the training data maintained by the database 130. It may also obtain training data from the cloud or other places for model training. The above description should not be used as a reference to this application. Limitations of the embodiment. It should also be noted that at least part of the training data maintained in the database 130 may also be used to execute the process of the processing to be processed by the execution device 110.
  • the target model/rule 101 trained according to the training device 120 can be applied to different systems or devices, such as the execution device 110 shown in FIG. 5, which can be a terminal, such as a mobile phone terminal, a tablet computer, Laptops, AR/VR, vehicle-mounted terminals, etc., can also be servers or clouds.
  • the execution device 110 shown in FIG. 5 can be a terminal, such as a mobile phone terminal, a tablet computer, Laptops, AR/VR, vehicle-mounted terminals, etc., can also be servers or clouds.
  • the execution device 110 is configured with an input/output (input/output, I/O) interface 112 for data interaction with external devices.
  • the user can input data to the I/O interface 212 through the client device 140.
  • the input data in this embodiment of the present application may include: a to-be-processed image input by the client device.
  • the preprocessing module 113 and the preprocessing module 114 are used to perform preprocessing according to the input data (such as the image to be processed) received by the I/O interface 112.
  • the preprocessing module 113 and the preprocessing module may not be provided.
  • 114 there may only be one preprocessing module, and the calculation module 111 is directly used to process the input data.
  • the execution device 110 may call data, codes, etc. in the data storage system 150 for corresponding processing .
  • the data, instructions, etc. obtained by corresponding processing may also be stored in the data storage system 150.
  • the I/O interface 112 returns the processing result to the raw domain image after the joint demosaicing and denoising of the raw domain image to be processed as described above, and returns the resulting output image to the client device 140 to provide it to the user.
  • the training device 120 can generate corresponding target models/rules 101 based on different training data for different goals or tasks, and the corresponding target models/rules 101 can be used to achieve the above goals or complete The above tasks provide users with the desired results.
  • the user can manually set input data, and the manual setting can be operated through the interface provided by the I/O interface 112.
  • the client device 140 can automatically send input data to the I/O interface 112. If the client device 140 is required to automatically send the input data and the user's authorization is required, the user can set the corresponding authority in the client device 140. The user can view the result output by the execution device 110 on the client device 140, and the specific presentation form may be a specific manner such as display, sound, and action.
  • the client device 140 can also be used as a data collection terminal to collect the input data of the input I/O interface 112 and the output result of the output I/O interface 112 as new sample data, and store it in the database 130 as shown in the figure.
  • the I/O interface 112 directly uses the input data input to the I/O interface 112 and the output result of the output I/O interface 112 as a new sample as shown in the figure.
  • the data is stored in the database 130.
  • FIG. 5 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship among the devices, devices, modules, etc. shown in the figure does not constitute any limitation.
  • the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.
  • the target model/rule 101 is obtained through training according to the training device 120.
  • the target model/rule 101 may be an image processing model in the embodiment of the present application.
  • the image processing model provided in the embodiment of the present application may be Deep neural network, convolutional neural network, or, it can be deep convolutional neural network, etc.
  • a convolutional neural network is a deep neural network with a convolutional structure. It is a deep learning architecture.
  • the deep learning architecture refers to the algorithm of machine learning. Multi-level learning is carried out on the abstract level of.
  • the convolutional neural network is a feed-forward artificial neural network, and each neuron in the feed-forward artificial neural network can respond to the input image.
  • the convolutional neural network 200 may include an input layer 210, a convolutional layer/pooling layer 220 (wherein the pooling layer is optional), a fully connected layer 230, and an output layer 240.
  • the input layer 210 can obtain the image to be processed, and pass the obtained image to be processed to the convolutional layer/pooling layer 220 and the fully connected layer 230 for processing, and the processing result of the image can be obtained.
  • the convolutional layer/pooling layer 220 may include layers 221-226, for example: in an implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, and layer 223 is a convolutional layer. Layers, 224 is the pooling layer, 225 is the convolutional layer, and 226 is the pooling layer; in another implementation, 221 and 222 are the convolutional layers, 223 is the pooling layer, and 224 and 225 are the convolutional layers. Layer, 226 is the pooling layer, that is, the output of the convolutional layer can be used as the input of the subsequent pooling layer, or can be used as the input of another convolutional layer to continue the convolution operation.
  • the convolution layer 221 can include many convolution operators.
  • the convolution operator is also called a kernel. Its function in image processing is equivalent to a filter that extracts specific information from the input image matrix.
  • the convolution operator is essentially It can be a weight matrix. This weight matrix is usually pre-defined. In the process of convolution on the image, the weight matrix is usually one pixel after one pixel (or two pixels after two pixels) along the horizontal direction on the input image. And so on, it depends on the value of stride) to complete the work of extracting specific features from the image.
  • the size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix and the depth dimension of the input image are the same.
  • the weight matrix will extend to Enter the entire depth of the image. Therefore, convolution with a single weight matrix will produce a single depth dimension convolution output, but in most cases, a single weight matrix is not used, but multiple weight matrices with the same size (row ⁇ column) are applied. That is, multiple homogeneous matrices.
  • the output of each weight matrix is stacked to form the depth dimension of the convolutional image, where the dimension can be understood as determined by the "multiple" mentioned above.
  • Different weight matrices can be used to extract different features in the image. For example, one weight matrix is used to extract the edge information of the image, another weight matrix is used to extract the specific color of the image, and the other weight matrix is used to correct the unwanted images in the image. The noise is blurred and so on.
  • the multiple weight matrices have the same size (row ⁇ column), the size of the convolution feature maps extracted by the multiple weight matrices of the same size are also the same, and then the multiple extracted convolution feature maps of the same size are merged to form The output of the convolution operation.
  • weight values in these weight matrices need to be obtained through a lot of training in practical applications.
  • Each weight matrix formed by the weight values obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions. .
  • the initial convolutional layer for example, 221
  • the general features can also be called low-level features
  • the features extracted by the subsequent convolutional layers become more and more complex, for example, features such as high-level semantics, and features with higher semantics are more suitable for the problem to be solved .
  • the pooling layer can be a convolutional layer followed by a layer.
  • the pooling layer can also be a multi-layer convolutional layer followed by one or more pooling layers.
  • the purpose of the pooling layer is to reduce the size of the image space.
  • the pooling layer may include an average pooling operator and/or a maximum pooling operator for sampling the input image to obtain an image with a smaller size.
  • the average pooling operator can calculate the pixel values in the image within a specific range to generate an average value as the result of the average pooling.
  • the maximum pooling operator can take the pixel with the largest value within a specific range as the result of the maximum pooling.
  • the operators in the pooling layer should also be related to the image size.
  • the size of the image output after processing by the pooling layer can be smaller than the size of the image of the input pooling layer, and each pixel in the image output by the pooling layer represents the average value or the maximum value of the corresponding sub-region of the image input to the pooling layer.
  • the convolutional neural network 200 After processing by the convolutional layer/pooling layer 220, the convolutional neural network 200 is not enough to output the required output information. Because as mentioned above, the convolutional layer/pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolutional neural network 200 needs to use the fully connected layer 230 to generate one or a group of required classes of output. Therefore, the fully connected layer 230 can include multiple hidden layers (231, 232 to 23n as shown in FIG. 6) and an output layer 240. The parameters contained in the multiple hidden layers can be based on specific task types. The relevant training data of the, for example, the task type can include image enhancement, image recognition, image classification, image detection, and image super-resolution reconstruction, etc.
  • the output layer 240 After the multiple hidden layers in the fully connected layer 230, that is, the final layer of the entire convolutional neural network 200 is the output layer 240.
  • the output layer 240 has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error.
  • the convolutional neural network shown in FIG. 6 is only used as an example of the structure of the image processing model of the embodiment of the present application.
  • the convolutional neural network used in the image processing method of the embodiment of the present application It can also exist in the form of other network models.
  • the image processing device may include the convolutional neural network 200 shown in FIG. Raw domain image.
  • FIG. 7 is a hardware structure of a chip provided by an embodiment of the present application.
  • the chip includes a neural network processor 300 (neural-network processing unit, NPU).
  • the chip can be set in the execution device 110 as shown in FIG. 5 to complete the calculation work of the calculation module 111.
  • the chip can also be set in the training device 120 as shown in FIG. 5 to complete the training work of the training device 120 and output the target model/rule 101.
  • the algorithms of each layer in the convolutional neural network as shown in FIG. 6 can be implemented in the chip as shown in FIG. 7.
  • the NPU 300 is mounted on a main central processing unit (CPU) as a coprocessor, and the main CPU distributes tasks.
  • the core part of the NPU 300 is the arithmetic circuit 303.
  • the controller 304 controls the arithmetic circuit 303 to extract data from the memory (weight memory or input memory) and perform calculations.
  • the arithmetic circuit 303 includes multiple processing units (process engines, PE).
  • the arithmetic circuit 303 is a two-dimensional systolic array; the arithmetic circuit 303 may also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition.
  • the arithmetic circuit 303 is a general-purpose matrix processor.
  • the arithmetic circuit 303 fetches the corresponding data of the matrix B from the weight memory 302 and caches it on each PE in the arithmetic circuit 303; the arithmetic circuit 303 receives the input
  • the memory 301 takes the data of matrix A and matrix B to perform matrix operations, and the partial results or final results of the obtained matrix are stored in an accumulator 308 (accumulator).
  • the vector calculation unit 307 can perform further processing on the output of the arithmetic circuit 303, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, and so on.
  • the vector calculation unit 307 can be used for network calculations in the non-convolutional/non-FC layer of the neural network, such as pooling, batch normalization, local response normalization, etc. .
  • the vector calculation unit 307 can store the processed output vector to the unified memory 306.
  • the vector calculation unit 307 may apply a nonlinear function to the output of the arithmetic circuit 303, such as a vector of accumulated values, to generate the activation value.
  • the vector calculation unit 307 generates a normalized value, a combined value, or both.
  • the processed output vector can be used as an activation input to the arithmetic circuit 303, for example for use in a subsequent layer in a neural network.
  • the unified memory 306 is used to store input data and output data.
  • the weight data directly passes through the storage unit access controller 305 (direct memory access controller, DMAC) to store the input data in the external memory into the input memory 401 and/or unified memory 406, and the weight data in the external memory into the weight memory 302 , And store the data in the unified memory 306 into the external memory.
  • DMAC direct memory access controller
  • the bus interface unit 310 (bus interface unit, BIU) is used to implement interaction between the main CPU, the DMAC, and the instruction fetch memory 309 through the bus.
  • the instruction fetch buffer 309 connected to the controller 304 is used to store instructions used by the controller 304; the controller 304 is used to call the instructions buffered in the instruction fetch memory 309 to control the working process of the computing accelerator.
  • unified memory 306, input memory 301, weight memory 302, and fetch memory 309 are all on-chip (On-Chip) memories.
  • Memory double data rate synchronous dynamic random access memory, DDR SDRAM), high bandwidth memory (HBM) or other readable and writable memory.
  • each layer in the convolutional neural network shown in FIG. 6 can be executed by the arithmetic circuit 303 or the vector calculation unit 307.
  • the execution device 110 in FIG. 5 introduced above can execute each step of the image processing method of the embodiment of the present application.
  • the CNN model shown in FIG. 6 and the chip shown in FIG. 7 can also be used to execute the image of the embodiment of the present application.
  • the various steps of the processing method can be performed by the execution device 110 in FIG. 5 introduced above.
  • FIG. 8 shows a system architecture 400 provided by an embodiment of the present application.
  • the system architecture includes a local device 420, a local device 430, an execution device 410, and a data storage system 450.
  • the local device 420 and the local device 430 are connected to the execution device 410 through a communication network.
  • the execution device 410 may be implemented by one or more servers.
  • the execution device 410 can be used in conjunction with other computing devices.
  • data storage for example: data storage, routers, load balancers and other equipment.
  • the execution device 410 may be arranged on one physical site or distributed on multiple physical sites.
  • the execution device 410 may use the data in the data storage system 450 or call the program code in the data storage system 450 to implement the image processing method of the embodiment of the present application.
  • execution device 410 may also be referred to as a cloud device, and in this case, the execution device 410 may be deployed in the cloud.
  • the execution device 410 may perform the following process: obtain a channel image corresponding to the original raw domain image to be processed, where the channel image includes at least two first channel images; perform image restoration processing on the first channel image, Obtain a restored first channel image; perform image restoration processing on the raw domain image to be processed according to the restored first channel image, to obtain a raw domain image after combined demosaicing and denoising processing.
  • the image processing method of the embodiment of the present application may be an offline method executed in the cloud.
  • the image processing method of the embodiment of the present application may be executed by the execution device 410 described above.
  • the image processing method in the embodiment of the present application may be executed by the local device 420 or the local device 430.
  • the user may operate respective user devices (for example, the local device 420 and the local device 430) to interact with the execution device 410.
  • Each local device can represent any computing device, for example, a personal computer, a computer workstation, a smart phone, a tablet computer, a smart camera, a smart car or other types of cellular phones, a media consumption device, a wearable device, a set-top box, a game console, etc.
  • the local device of each user can interact with the execution device 410 through a communication network of any communication mechanism/communication standard.
  • the communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.
  • the local device 420 and the local device 430 may obtain the relevant parameters of the aforementioned neural network model from the execution device 410, deploy the neural network model on the local device 420 and the local device 430, and use the neural network model to perform Image processing, etc.
  • the neural network model can be directly deployed on the execution device 410.
  • the execution device 410 obtains the channel image corresponding to the raw domain image to be processed from the local device 420 and the local device 430, and uses the raw domain image to be processed in the neural network model.
  • the image undergoes image restoration processing to obtain the raw domain image after joint demosaicing and denoising.
  • the method of joint demosaicing and denoising based on deep learning can be to obtain the Bayer image of the original image, and sample the single-channel Bayer image into a four-channel RGGB image. In addition, it can also obtain a three-channel image from the single-channel Bayer image.
  • the RGB mask images are respectively the red mask image, the blue mask image and the green mask image.
  • the feature map output by the RGGB image through the convolution layer and the RGB mask image are channel-parallel fused to obtain a clean RGB
  • the image is the image after combined demosaicing and denoising; however, because only the RGB mask image is used to constrain the original image, when the dense texture area in the original image is processed by the RGB mask, the image remains in the image There may be aliasing and color artifacts, resulting in lower image quality after processing.
  • this application proposes an image processing method and an image processing device.
  • the channel image corresponding to the original raw domain image can be obtained.
  • the channel image can include at least two first channel images; Image restoration processing to obtain the restored first channel image; image restoration processing is performed on the raw domain image according to the restored first channel image, and finally the raw domain image after combined demosaicing and denoising processing is obtained; implemented in this application
  • the channel image of the raw domain image can be repaired in advance, and the raw domain image is guided through the repaired channel image, so as to avoid the appearance of color texture, so that the processed image can retain the image details and improve the image quality.
  • the image processing method provided by the embodiment of the present application can be deployed on the computing node of the related device, and the combined demosaicing and denoising method in the embodiment of the present application can be effectively implemented through software algorithms.
  • the system architecture of the embodiment of the present application will be described in detail below in conjunction with FIG. 9.
  • the image processing method provided by the embodiment of the present application can be applied to a joint demosaicing and denoising system 500, where the joint demosaicing and denoising system 500 may include a main repair module 510, a channel repair module 520, Guide module 530; further, it may also include a dense map construction module 540 and a pre-training feature warehouse 550; each module will be described in detail below.
  • the joint demosaicing and denoising system 500 may include a main repair module 510, a channel repair module 520, Guide module 530; further, it may also include a dense map construction module 540 and a pre-training feature warehouse 550; each module will be described in detail below.
  • Main restoration module 510 used to perform convolution operations on the acquired original raw domain image to achieve preliminary joint demosaicing and denoising processing to obtain the image characteristics of the raw domain image; among them, the raw domain image can refer to a demosaic or Bayer format image for denoising needs.
  • the main repair module 510 may reorganize the color filter array signal data (for example, raw domain image) obtained by the signal processor of the smart terminal to obtain a four-channel image; input the four-channel image and the random noise image to the convolution
  • the neural network performs convolution processing to obtain the image features that are initially restored.
  • the above-mentioned four-channel image may refer to an RGGB four-channel image; alternatively, it may also refer to a BGGR four-channel image; or, it may also refer to an RYYB four-channel image.
  • the dense map can also be obtained from the dense construction module 540, so as to input the dense map into the convolutional neural network, where the dense map construction module 540 is used to obtain the dense map.
  • the dense map can be used to indicate the frequency of different texture regions in the raw domain image to be processed; the dense map has a higher response to high-frequency texture regions and a lower response to low-frequency texture regions, so that the raw domain can be distinguished from the dense map Texture areas of different frequency sizes in the image.
  • the dense image can make the convolutional neural network better recognize the high-frequency area and the low-frequency area in the image.
  • the high-frequency area in the image refers to the edge and detail area in the image, which is compared with the low-frequency area. It is said that the high-frequency region is more difficult to repair; if the dense map is not introduced into the convolutional neural network, for the convolutional neural network, each area of the image is uniformly repaired; after the dense map is input, the convolutional neural network can focus on the raw domain image to be processed The medium and high frequency areas are repaired with emphasis, so as to distinguish the texture dense area and the texture sparse area in the raw domain image to be processed, and realize the repair of different dense areas.
  • Channel repair module 520 used to pre-repair the main channel in the four-channel image of the raw domain image, so as to obtain the repaired channel image, that is, the channel image after the preliminary repair.
  • the channel repair module 520 may be used to pre-repair the two G channels.
  • the channel repair module 520 can be used to pre-repair the two Y channels.
  • the restoration module 520 will select the channel images with a larger amount of information among the four channels for pre-repair, so as to provide the subsequent guidance module 530 with more information about the raw domain image to be processed, which is beneficial to the subsequent guidance module 530 performs further adaptive convolution processing on the image characteristics of the raw domain image to be processed.
  • Guide module 530 used to perform further convolution processing on the image features output by the main repair module 510 through the repaired first channel image input by the channel repair module 520, so as to obtain the final output image, that is, joint demosaicing and denoising processing After the image.
  • the joint demosaicing and denoising system 500 may further include a training feature warehouse 550, and the training feature warehouse 550 may be used to store sample images and image features corresponding to the sample images for demosaicing and denoising repair.
  • training images for training the joint demosaicing and denoising model can be obtained from the training feature warehouse 550.
  • FIG. 10 shows a schematic flowchart of an image processing method 600 provided by an embodiment of the present application.
  • the method may be applied to the joint demosaicing and denoising system shown in FIG. 9; the method may be executed by a device capable of image processing
  • the method may be executed by the execution device 410 in FIG. 8 or may also be executed by the local device 420.
  • the method 600 includes steps 610 to 630, and these steps are respectively described in detail below.
  • Step 610 Obtain a channel image corresponding to the original raw domain image to be processed, where the channel image includes at least two first channel images.
  • the raw image to be processed may be a raw image collected by a sensor, where the raw image to be processed refers to a raw image that has not been processed by an image signal processor (ISP).
  • ISP image signal processor
  • the aforementioned raw domain image to be processed may refer to an image in Bayer format with demosaicing and denoising requirements, that is, there may be some additional information in the image, which may be noise or artifacts.
  • the channel image corresponding to the raw domain image to be processed may refer to the four-channel image corresponding to the raw domain image to be processed.
  • it can refer to the RGGB four-channel image; or, the BGGR four-channel image; or, the RYYB four-channel image.
  • the acquired channel image of the raw domain image to be processed may be an RGGB four-channel image or a BGGR four-channel image
  • the above-mentioned first channel image may represent a green G channel image
  • the acquired channel image of the raw domain image to be processed is an RYYB four-channel image
  • the above-mentioned first channel image may represent a yellow Y channel image
  • Step 620 Perform image restoration processing on the first channel image to obtain a restored first channel image.
  • the above-mentioned performing image restoration processing on the first channel image to obtain the restored first channel image may refer to performing preliminary image restoration processing on the first channel image to obtain the initially restored first channel image.
  • the image restoration processing refers to the technique of restoring the lost parts of the image and reconstructing them based on the image information.
  • Image restoration processing can try to estimate the original image information, repair and improve the damaged area, thereby improving the visual quality of the image.
  • performing image restoration processing on the first channel image may refer to performing convolution processing on the first channel image, where the first channel image may refer to having an image in multiple channel images corresponding to the raw domain image to be processed.
  • the image information can refer to the high-frequency information in the image; for example, it can refer to the detail information, texture information, and edge information in the image.
  • Exemplarily, for the specific process of pre-repairing the first channel image refer to the following figure 13.
  • the first channel image and the random noise image may be channel-paralleled and then input into the convolutional neural network for convolution processing.
  • the aforementioned convolutional neural network may refer to Residual-in-Residual Dense Block (RRDB) or other network models.
  • RRDB Residual-in-Residual Dense Block
  • Step 630 Perform image restoration processing on the raw domain image to be processed according to the restored first channel image to obtain a raw domain image after combined demosaicing and denoising processing.
  • the neural network can have more image information when performing convolution processing on the raw domain image to be processed, so that the neural network can be used in the raw domain to be processed.
  • the appearance of color texture can be avoided to the greatest extent.
  • a dense image can also be input to the neural network.
  • the above-mentioned image processing method further includes: obtaining a dense image of the raw domain image to be processed, where the dense image can be used to indicate texture regions of different frequencies in the raw domain image to be processed; Process the raw domain image to perform image restoration processing to obtain the raw domain image after joint demosaicing and denoising, including: convolution processing on the dense image and the raw domain image to be processed to obtain the first image feature; through the restored The first channel image guides the first image feature to perform convolution processing to obtain the raw domain image after joint demosaicing and denoising.
  • the above-mentioned guiding the first image feature to perform convolution processing through the restored first channel image to obtain the image after joint demosaicing and denoising includes: comparing the first channel image according to the pixel spatial distribution of the restored first channel image.
  • An image feature is subjected to convolution processing to obtain a raw domain image that is combined with demosaicing and denoising, where the spatial distribution of pixels is used to indicate the correlation of the distribution of different pixel points in the restored first channel image.
  • the first image feature is guided to perform convolution processing through the restored first channel image to obtain the raw domain image after joint demosaicing and denoising.
  • the multi-channel image corresponding to the raw domain image (for example, the RGGB channel image, the GBBR channel image or the RYYB channel image) is spliced, so that the raw domain image to be processed is guided to obtain the raw domain image after joint demosaicing and denoising.
  • first image feature may refer to the image feature obtained by preliminary restoration of the raw domain image to be processed, that is, there may be some noise and artifacts in the first image feature, and no real joint demosaicing and de-mosaic has been obtained.
  • the image after noise may refer to the image feature obtained by preliminary restoration of the raw domain image to be processed, that is, there may be some noise and artifacts in the first image feature, and no real joint demosaicing and de-mosaic has been obtained. The image after noise.
  • the above dense map can be used to indicate the texture area of different frequencies in the raw domain image to be processed; the dense map has a higher response to the high-frequency texture area and lower response to the low-frequency texture area, so that it can be based on the dense map Distinguish the different texture areas in the raw image to be processed.
  • the specific process of obtaining the dense image of the raw domain image to be processed can refer to the following figure 14; it can include the following steps:
  • Step 1 Obtain multiple channel images corresponding to the raw domain image to be processed, and perform averaging processing on the pixels of the multiple channel images to obtain a grayscale image Img.
  • the four-channel image of RGGB is averaged to obtain a grayscale image.
  • Step 2 The gray image is subjected to Gaussian filtering to obtain a blurred gray image Img_blur.
  • Step 3 Subtract the gray-scale image Img and the blurred gray-scale image Img_blur to obtain the residual image Img_minus, where the high pixel value area in the residual image can indicate that the texture frequency is high.
  • Step 4 Perform Gaussian blur on the residual image Img_minus first, and then perform normalization processing to obtain the dense image MD.
  • random noise images can also be input into the neural network, that is, the dense image, raw domain image to be processed, and random noise images can be input in parallel after channels To the convolutional neural network, which further improves the image restoration effect of the raw domain image to be processed by the neural network.
  • the first image feature obtained by performing convolution processing on the dense image and the raw-domain image to be processed includes: obtaining a random noise image; convolving the dense image, the random noise image, and the raw-domain image to be processed Processing to obtain the first image feature.
  • the RRDB network structure or other network models may be used.
  • the image feature is not a true clean image and may still exist Partial noise or artifacts, etc.; further, adaptive convolution processing can be performed on the output image features according to the repaired channel image, so that the noise and mosaic in the image features can be completely removed to obtain the final joint demosaicing and denoising Raw domain image; raw domain image.
  • the specific processing flow of the adaptive convolution processing can be referred to as shown in the subsequent FIG. 15.
  • adaptive convolution can take into account the relevance of the spatial distribution of different pixels on the first channel image; in addition, due to the raw domain image to be processed The difference of image information in different regions in the image, therefore, different convolution windows can be used for different regions in the raw domain image to be processed, that is, the parameters in the convolution window for different regions in the image can be different; Mosaic and denoised raw domain images play a guiding role in pixel spatial distribution and image texture information.
  • the adaptive convolution processing can be performed by the following formula:
  • o i represents the image feature at position i in the image feature corresponding to the raw domain image after joint demosaicing and denoising
  • ⁇ represents the convolution window
  • W represents the weight parameter of the convolution
  • Pi ,j represents the weight
  • W B represents the offset of the convolution
  • f i represents a vector at position i in the image feature corresponding to the repaired channel image
  • f j represents a vector at position j in the image feature corresponding to the repaired channel image
  • the position j has a correlation with the position i, that is, the position j can refer to any position in the convolution window corresponding to the position i
  • G represents the Gaussian function, which can be used to find the Gaussian distance between two vectors .
  • the image processing method provided by the embodiment of the application can obtain the channel image corresponding to the raw domain image to be processed, and the channel image can include at least two first channel images; the first channel image is subjected to pre-image restoration processing to obtain the restoration The first channel image after the restoration; according to the restored first channel image, the raw domain image to be processed can be guided to perform image restoration processing to obtain the raw domain image after the combined demosaicing and denoising processing; the image processing in the embodiment of the application
  • the channel image of the raw image to be processed can be repaired in advance, and the raw image to be processed is guided to perform image repair through the repaired channel image, thereby avoiding the appearance of color texture, so that the processed image can retain the image details and improve the image quality.
  • the dense map of the raw domain image to be processed can also be input so that the neural network can be guided to distinguish dense texture regions and sparse texture regions during convolution processing. The area is restored.
  • FIG. 11 is a schematic flowchart of an image processing method provided by an embodiment of the present application, that is, an image joint demosaicing and denoising method.
  • the joint demosaicing and denoising method 700 includes steps 701 to 705, and these steps are respectively performed below Detailed description.
  • Step 701 Obtain a raw domain image to be processed.
  • the camera is a module that may include a sensor and a lens
  • the raw image to be processed may be a raw image collected by a sensor, where the raw image to be processed refers to an image signal processor (image signal processor, ISP) processed original domain image.
  • ISP image signal processor
  • the aforementioned raw domain image to be processed may be a Bayer format image with demosaicing and denoising requirements, that is, there may be phenomena such as noise or artifacts in the raw domain image.
  • the raw image to be processed may also refer to the color filter array data collected by the camera sensor in the smart terminal.
  • Step 702 The repaired channel image, that is, the initially repaired channel image.
  • the raw domain image to be processed may be reorganized to obtain a four-channel image or other number of channel images.
  • a four-channel image is used as an example.
  • the foregoing four-channel image may refer to an RGGB four-channel image; alternatively, it may also refer to a BGGR four-channel image; or, it may also refer to an RYYB four-channel image.
  • the main channel image in the above four-channel image is pre-repaired, where the main channel may refer to the channel image with the most image information in the channel image.
  • the channel repair module 520 may be used to pre-repair the two G channels.
  • the channel repair module 520 can be used to pre-repair the two Y channels.
  • the above step 702 can be performed in the channel repair module 520 shown in FIG. 9.
  • FIG. 13 is a schematic diagram of a repaired channel image provided by an embodiment of the present application.
  • the channel image for pre-repair can be selected from multiple channel images. For example, for the RGGB channel image or the BGGR channel image, you can select two G channel images and perform pre-repair;
  • the G channel image undergoes image restoration processing, that is, convolution processing, to obtain a restored G channel image.
  • two selected G-channel images and random noise images can be processed in parallel, that is, two G-channel images of the same scale and random noise images can be combined with feature maps; the image features after the parallel processing of channels are input
  • the convolutional neural network performs convolution processing, and then performs an up-sampling operation on the pre-repaired channel image features to obtain the repaired channel image; as shown in Figure 13, the output size is 2w*2h*1 for the pre-repaired channel
  • the true value image of the channel image can be used for supervision, so that the channel image can be subjected to preliminary image restoration.
  • the convolutional neural network shown in FIG. 13 may refer to a residual-in-residual dense block (RRDB) or other network models.
  • RRDB residual-in-residual dense block
  • Step 703 The first image feature is the image feature after preliminary restoration of the raw domain image to be processed.
  • the aforementioned preliminary restoration of the image may refer to the preliminary image restoration processing performed on each channel image of the raw domain image to be processed to obtain the output first image feature, but there may be some noise and artifacts in the first image feature. Phenomenon, a truly clean image has not been obtained.
  • steps 702 and 703 may be performed first and then step 703; alternatively, step 703 may be performed first and then step 702.
  • This application does not limit the execution order of step 702 and step 703. .
  • FIG. 12 is a schematic diagram of a preliminary repaired image provided by an embodiment of the present application.
  • the input data can refer to a raw domain image (for example, Bayer image) with a size of 2w*2h to be processed collected by the camera sensor, and the channel image is obtained by resampling the raw domain image; for example, for Bayer
  • the images are grouped according to 2*2, and then pixels are sampled from a fixed position (upper left, upper right, lower left, and lower right) from each 2*2 small block to generate four channel images (for example, RGGB, BGGR, RYYB) Etc.); the four-channel RGGB image is input into the convolutional neural network for convolution processing, so as to obtain the image characteristics of the raw domain image after the initial image restoration.
  • a random noise image can also be input into the convolutional neural network during the aforementioned initial image restoration process.
  • the dense image can also be input to the convolutional neural network, where,
  • the dense map can be used to indicate the texture area of different frequencies in the raw domain image to be processed; the dense image has a higher response to the high-frequency texture area and lower response to the low-frequency texture area, so that the raw domain to be processed can be distinguished from the dense image Different texture areas in the image.
  • the dense image can make the convolutional neural network better recognize the high-frequency area and the low-frequency area in the image.
  • the high-frequency area in the image refers to the edge and detail area in the image, which is compared with the low-frequency area. It is said that the high-frequency region is more difficult to repair; if the dense map is not introduced into the convolutional neural network, for the convolutional neural network, each area of the image is uniformly repaired; after the dense map is input, the convolutional neural network can focus on the raw domain image to be processed The middle and high frequency areas are repaired with emphasis, so as to distinguish the dense and sparse texture areas in the raw domain image to be processed, and realize the image restoration of different texture areas.
  • FIG. 14 is a schematic diagram of obtaining a dense map provided by an embodiment of the present application.
  • the process of obtaining dense graphs can include the following steps:
  • Step 1 Obtain multiple channel images corresponding to the raw domain image to be processed, and perform averaging processing on the pixels of the multiple channel images to obtain a grayscale image Img.
  • the four-channel image of RGGB is averaged to obtain a grayscale image.
  • Step 2 The gray image is subjected to Gaussian filtering to obtain a blurred gray image Img_blur.
  • Step 3 Subtract the gray-scale image Img and the blurred gray-scale image Img_blur to obtain the residual image Img_minus, where the high pixel value area in the residual image can indicate that the texture frequency is high.
  • Step 4 Perform Gaussian blur on the residual image Img_minus first, and then perform normalization processing to obtain the dense image M D.
  • Step 704 Adaptive convolution processing.
  • adaptive convolution processing is performed.
  • adaptive convolution can take into account the correlation of the spatial distribution of different pixels on the first channel image; in addition, because the raw domain images to be processed are different The difference in the image information of the region, therefore, different convolution windows can be used for different regions in the raw domain image to be processed, that is, the parameters in the convolution window can be different for different regions in the image; thus, the joint demosaicing and The denoised raw domain image plays a guiding role in pixel spatial distribution and image texture information.
  • the pre-repaired image features can be guided by the spatial distribution of the pixels of the pre-repaired channel image, so that the pre-repaired image features are subjected to further convolution processing, so that the pre-repaired image features can also be to a certain extent It satisfies the same or similar pixel spatial distribution as in the pre-repaired channel image, where the pixel spatial distribution can be used to indicate the correlation of different pixel point distributions.
  • the adaptive convolution processing can be performed by the following formula:
  • o i represents the image feature at position i in the image feature corresponding to the raw domain image after joint demosaicing and denoising
  • ⁇ represents the convolution window
  • W represents the weight parameter of the convolution
  • Pi ,j represents the weight
  • W B represents the offset of the convolution
  • f i represents a vector at position i in the image feature corresponding to the repaired channel image
  • f j represents a vector at position j in the image feature corresponding to the repaired channel image
  • the position j has a correlation with the position i, that is, the position j can refer to any position in the convolution window corresponding to the position i
  • G represents the Gaussian function, which can be used to find the Gaussian distance between two vectors .
  • FIG. 15 is a schematic diagram of adaptive convolution processing provided by an embodiment of the present application.
  • the first channel image for example, the pre-repaired G channel image or the pre-repaired Y channel image
  • the first channel image can be upgraded through two layers of convolution to obtain an 8-channel feature map; then according to the above
  • the adaptive convolution processing formula performs further convolution processing on the initially restored image features, and finally obtains the raw domain image after joint demosaicing and denoising.
  • the size of the convolution window in FIG. 15 can refer to 3*3; f i can represent a vector at any position in the image feature; for example, f i represents the position of f 2,2 in the convolution window.
  • f j can represent any vector in the convolution window except f 2,2 , that is, there may be multiple related f j at the same time for one f i .
  • Step 705 Output the raw domain image after joint demosaicing and denoising.
  • subsequent image processing can be performed on the raw domain image after joint demosaicing and denoising, so as to obtain a standard full-color image, that is, an SRGB image.
  • the subsequent image processing may refer to joint demosaicing and denoising.
  • the raw domain image after noise is processed for white balance, color correction, tone mapping and other image processing.
  • the image processing method of the embodiment of the present application will be described in detail below with reference to FIG. 16.
  • the image processing method can be applied to electronic equipment with a display screen and a camera.
  • the electronic device may specifically be a mobile terminal (for example, a smart phone), a computer, a personal digital assistant, a wearable device, a vehicle-mounted device, an Internet of Things device or other devices capable of displaying images.
  • the image processing method 800 shown in FIG. 16 includes steps 810 and 820, which are respectively described in detail below.
  • Step 810 It is detected that the user has instructed the operation of the camera.
  • the first processing mode may be a professional shooting mode, or the first processing mode may also refer to an AI ISP processing mode, that is, the method shown in FIG. 10 is used.
  • the image processing method performs image restoration processing on the raw domain image to be processed collected by the camera, or the first processing mode may also be a combined demosaicing and denoising shooting mode, so that images with higher image quality can be obtained.
  • the shooting interface includes a shooting option 960.
  • the electronic device detects that the user clicks on the shooting option 960, see (b) in FIG. 18, and the electronic device displays a shooting mode interface.
  • the electronic device detects that the user clicks on the shooting mode interface to indicate the professional shooting mode 961, the mobile phone enters the professional shooting mode.
  • this operation may be the detection of an operation used by the user to instruct shooting, and this operation may be used as an operation instructing to perform image restoration on the acquired raw domain image to be processed.
  • an operation 970 for the user to instruct shooting can be detected.
  • the operation used by the user to instruct the shooting behavior may include pressing the shooting button in the camera of the electronic device, or may include the user device instructing the electronic device to perform the shooting behavior through voice, or may also mean that the user instructs the electronic device to perform the shooting behavior through a shortcut key.
  • the shooting behavior of the device may also include other user instructing the electronic device to perform shooting behavior.
  • the aforementioned detection of the operation of the user instructing the camera may refer to the operation of directly performing shooting by the user through a shortcut key.
  • the method further includes: detecting a first operation for opening the camera by the user is detected; and in response to the first operation, displaying a shooting interface on the display screen.
  • the user's shooting behavior may include a first operation of the user to turn on the camera; in response to the first operation, displaying a shooting interface on the display screen.
  • FIG. 17 shows a graphical user interface (GUI) of the mobile phone, and the GUI is the desktop 910 of the mobile phone.
  • GUI graphical user interface
  • the electronic device detects that the user has clicked the icon 920 of the camera application (application, APP) on the desktop 910, it can start the camera application and display another GUI as shown in (b) in FIG. 17, which can be called It is the shooting interface 930.
  • the shooting interface 930 may include a viewfinder frame 940; in the preview state, the viewfinder frame 940 may display a preview image in real time.
  • a preview image may be displayed in the viewfinder frame 940, and the preview image is a color image; the shooting interface may also include a control for indicating the photographing mode 950, and other shooting controls.
  • the color image portion is represented by oblique line filling.
  • the user's shooting behavior may include a first operation of the user to turn on the camera; in response to the first operation, displaying a shooting interface on the display screen.
  • the shooting interface may include a viewfinder frame.
  • the size of the viewfinder frame may be different in the photo mode and the video mode.
  • the viewfinder frame may be the viewfinder frame in the photographing mode.
  • the viewfinder frame can be the entire display screen.
  • the preview state that is, before the user turns on the camera and does not press the photo/video button, the preview image can be displayed in the viewfinder frame in real time.
  • Step 820 In response to the operation, display an output image on the display screen, where the output image is obtained based on the original raw domain image after joint demosaicing and denoising, and the joint demosaicing and
  • the denoised raw domain image is the raw domain image obtained after image restoration processing is performed on the raw domain image to be processed collected by the camera, and the restored first channel image is used for the image restoration of the raw domain image to be processed
  • the raw domain image to be processed includes at least two first channel images
  • the restored first channel image refers to a channel image obtained by performing image restoration processing on the first channel image.
  • the output image may refer to a standard full-color image obtained by subsequent image processing of the raw domain image after joint demosaicing and denoising.
  • the subsequent image processing may include, but is not limited to: joint demosaicing and denoising
  • image processing such as white balance, color correction, and tone mapping. For example, see Figure 20.
  • the above-mentioned convolution processing on the first channel image may refer to pre-repairing the first channel image, where the first channel image may refer to the multiple channel images corresponding to the raw domain image to be processed.
  • the specific process of pre-repairing the first channel image can be referred to as shown in FIG. 13, which will not be repeated here.
  • the channel image corresponding to the raw domain image to be processed may refer to the RGGB four-channel image; or, the BGGR four-channel image; or, the RYYB four-channel image.
  • the above-mentioned first channel image may represent a green G channel image.
  • the above-mentioned first channel image may represent a yellow Y channel image.
  • the neural network can have more image information when performing convolution processing, so that the neural network can maximize the restoration of the raw domain image to be processed.
  • the dense image can also be input to the neural network.
  • the image restoration processing process further includes: obtaining a dense map of the raw-domain image to be processed, wherein the dense map is used to indicate texture regions of different frequencies in the raw-domain image to be processed; Perform convolution processing according to the dense map and the raw domain image to be processed to obtain a first image feature; guide the first image feature to perform convolution processing through the restored first channel image to obtain the joint Raw domain image after demosaicing and denoising.
  • first image feature may refer to the image feature obtained by preliminary restoration of the raw domain image to be processed, that is, there may be some noise and artifacts in the first image feature, and no real joint demosaicing has been obtained. And the raw domain image after denoising.
  • the image feature is not a true clean image, and there may be some noise or false
  • the adaptive convolution takes into account the correlation of the spatial distribution of different pixels on the first channel image shown; in addition, due to the image of different regions in the raw domain image
  • different convolution windows can be used for different regions in the image, that is, the parameters in the convolution window can be different for different regions in the image; thus, the raw domain image after joint demosaicing and denoising Play a guiding role in pixel spatial distribution and image texture information.
  • guiding the first image feature to perform convolution processing through the restored first channel image to obtain the raw domain image after joint demosaicing and denoising includes: according to the pixel space of the restored first channel image The distribution performs convolution processing on the first image feature to obtain a raw domain image after joint demosaicing and denoising, where the pixel spatial distribution is used to indicate the correlation of the distribution of different pixel points in the restored first channel image.
  • the adaptive convolution can take into account the correlation of the spatial distribution of different pixels on the first channel image; in addition, because the raw domain images to be processed are different The difference in the image information of the region, therefore, different convolution windows can be used for different regions in the raw domain image to be processed, that is, the parameters in the convolution window can be different for different regions in the image; thus, the joint demosaicing and The denoised raw domain image plays a guiding role in pixel spatial distribution and image texture information.
  • the pre-repaired image features can be guided by the spatial distribution of the pixels of the pre-repaired channel image, so that the pre-repaired image features are subjected to further convolution processing, so that the pre-repaired image features can also be to a certain extent It satisfies the same or similar pixel spatial distribution as in the pre-repaired channel image, where the pixel spatial distribution can be used to indicate the correlation of different pixel point distributions.
  • the adaptive convolution processing can be performed by the following formula:
  • o i represents the image feature at position i in the image feature corresponding to the raw domain image after joint demosaicing and denoising
  • ⁇ represents the convolution window
  • W represents the weight parameter of the convolution
  • Pi ,j represents the weight
  • W B represents the offset of the convolution
  • f i represents a vector at position i in the image feature corresponding to the repaired channel image
  • f j represents a vector at position j in the image feature corresponding to the repaired channel image
  • the position j has a correlation with the position i, that is, the position j can refer to any position in the convolution window corresponding to the position i
  • G represents the Gaussian function, which can be used to find the Gaussian distance between two vectors .
  • the first image feature is obtained by performing convolution processing on the dense image and the raw-domain image to be processed, including: obtaining a random noise image; performing convolution processing on the dense image, random noise image, and raw-domain image to be processed, Obtain the first image feature.
  • the image processing method provided by the embodiment of the application can obtain the channel image corresponding to the raw domain image to be processed, and the channel image can include at least two first channel images; the first channel image is subjected to pre-image restoration processing to obtain the restoration The first channel image after the restoration; according to the restored first channel image, the raw domain image to be processed can be guided to perform image restoration processing to obtain the raw domain image after the combined demosaicing and denoising processing; the image processing in the embodiment of the application
  • the channel image of the raw domain image to be processed can be repaired in advance, and the restored channel image is the initially restored channel image; based on the preliminary restored channel image, the raw domain image to be processed is guided to perform image restoration processing, thereby avoiding the appearance of color texture , So that the processed image can retain the image details and improve the image quality.
  • the dense map of the raw domain image to be processed can also be input so that the neural network can be guided to distinguish dense texture regions and sparse texture regions during convolution processing. The area is restored.
  • Table 1 is the test result of joint demosaicing and denoising performed on different models in different data sets provided in the embodiments of the present application.
  • the noise during the test is Gaussian noise with a standard deviation of 10
  • the test result is based on the peak signal-to-noise ratio (PSNR), learning perceptual image patch similarity (LPIPS) and structure Similarity (Structural similarity index, SSIM);
  • the tested algorithms include the image processing method provided in the embodiments of this application, the flexible image signal processing algorithm (Flexible image signal processing, FlexISP), and the sequential energy minimization algorithm (Sequential energy minimization, SEM), Alternating Direction Method of Multipliers (ADMM), Deep Joint Demosaicking and Denoising (Deep Joint) and Kokkinos Algorithm. It can be seen from the test performance indicators shown in Table 1 that the image processing method proposed in the embodiment of the present application, that is, the combined demosaicing and denoising method, achieves better results in the test
  • FIG. 21 is a schematic block diagram of an image processing device provided by an embodiment of the present application. It should be understood that the image processing apparatus 1000 can execute the image processing method shown in FIG. 10.
  • the image processing device 1000 includes: an acquisition unit 1010 and a processing unit 1020.
  • the acquiring unit 1010 is configured to acquire a channel image corresponding to the original raw domain image to be processed, wherein the channel image includes at least two first channel images; the processing unit 1020 is configured to compare the first channel image Perform image restoration processing to obtain a restored first channel image; perform image restoration processing on the raw domain image according to the restored first channel image to obtain a raw domain image after combined demosaicing and denoising processing.
  • the acquiring unit 1010 is further configured to:
  • the processing unit 1020 is specifically configured to:
  • the first image feature is guided to perform convolution processing through the restored first channel image to obtain the raw domain image after the joint demosaicing and denoising.
  • processing unit 1020 is specifically configured to:
  • the acquiring unit 1010 is further configured to:
  • the processing unit 1020 is specifically configured to:
  • the first channel image is a green G channel image.
  • the first channel image is a yellow Y channel image.
  • FIG. 22 is a schematic block diagram of an image processing device provided by an embodiment of the present application. It should be understood that the image processing apparatus 1100 may execute the image processing method shown in FIG. 16; the image processing apparatus 1100 includes: a detection unit 1110 and a processing unit 1120.
  • the detection unit 1110 is used to detect an operation instructed by the user to the camera; the processing unit 1120 is used to display an output image in the display screen in response to the operation, wherein the output image is based on a joint operation.
  • the original raw domain image after mosaic and denoising, the raw domain image after joint demosaicing and denoising is the raw domain image obtained after image restoration processing is performed on the raw domain image to be processed collected by the camera,
  • the restored first channel image is used in the image restoration process of the raw domain image to be processed, the raw domain image to be processed includes at least two first channel images, and the restored first channel image refers to A channel image obtained by performing image repair processing on the first channel image.
  • the aforementioned image processing apparatus 1100 may be an electronic device having a display screen and a camera.
  • the detection unit 1110 is specifically configured to:
  • An operation indicating a first processing mode by the user is detected, and the first processing mode is used to perform image restoration processing on the raw domain image to be processed; or,
  • An operation for instructing shooting by the user is detected.
  • the image processing apparatus further includes an acquiring unit, and the acquiring unit is configured to:
  • the processing unit 1120 is specifically configured to:
  • the first image feature is guided to perform convolution processing through the restored first channel image to obtain the raw domain image after the joint demosaicing and denoising.
  • processing unit 1120 is specifically configured to:
  • the acquiring unit is further configured to:
  • the processing unit 1120 is specifically configured to:
  • the first channel image is a green G channel image.
  • the first channel image is a yellow Y channel image.
  • image processing device 1000 and image processing device 1100 are embodied in the form of functional units.
  • unit herein can be implemented in the form of software and/or hardware, which is not specifically limited.
  • a "unit” may be a software program, a hardware circuit, or a combination of the two that realizes the above-mentioned functions.
  • the hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, and a processor for executing one or more software or firmware programs (such as a shared processor, a dedicated processor, or a group processor). Etc.) and memory, merged logic circuits and/or other suitable components that support the described functions.
  • the units of the examples described in the embodiments of the present application can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.
  • FIG. 23 is a schematic diagram of the hardware structure of an image processing apparatus provided by an embodiment of the present application.
  • the image processing apparatus 1200 shown in FIG. 23 (the apparatus 1200 may specifically be a computer device) includes a memory 1201, a processor 1202, a communication interface 1203, and a bus 1204.
  • the memory 1201, the processor 1202, and the communication interface 1203 implement communication connections between each other through the bus 1204.
  • the memory 1201 may be a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM).
  • the memory 1201 may store a program.
  • the processor 1202 is configured to execute each step of the image processing method of the embodiment of the present application, for example, execute each of the steps shown in FIG. 10 to FIG. 20 step.
  • the image processing apparatus shown in the embodiment of the present application may be a server, for example, it may be a server in the cloud, or may also be a chip configured in a server in the cloud; or, the image processing apparatus shown in the embodiment of the present application
  • the device can be a smart terminal or a chip configured in the smart terminal.
  • the image processing method disclosed in the foregoing embodiments of the present application may be applied to the processor 1202 or implemented by the processor 1202.
  • the processor 1202 may be an integrated circuit chip with signal processing capabilities.
  • the steps of the above-mentioned image processing method can be completed by hardware integrated logic circuits in the processor 1202 or instructions in the form of software.
  • the processor 1202 may be a chip including the NPU shown in FIG. 7.
  • the aforementioned processor 1202 may be a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose processor, a digital signal processor (DSP), and an application specific integrated circuit (application integrated circuit).
  • CPU central processing unit
  • GPU graphics processing unit
  • DSP digital signal processor
  • ASIC application specific integrated circuit
  • ASIC application specific integrated circuit
  • FPGA field programmable gate array
  • the general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.
  • the steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor.
  • the software module can be located in random access memory (RAM), flash memory, read-only memory (read-only memory, ROM), programmable read-only memory or electrically erasable programmable memory, registers, etc. mature in the field Storage medium.
  • the storage medium is located in the memory 1201, and the processor 1202 reads the instructions in the memory 1201, and combines its hardware to complete the functions required by the units included in the image processing apparatus shown in FIG. 21 or FIG. 22 in the implementation of this application, or execute Each step of the image processing method shown in FIG. 10 to FIG. 20 of the method embodiment of the present application.
  • the communication interface 1203 uses a transceiver device such as but not limited to a transceiver to implement communication between the device 1200 and other devices or a communication network.
  • a transceiver device such as but not limited to a transceiver to implement communication between the device 1200 and other devices or a communication network.
  • the bus 1204 may include a path for transferring information between various components of the image processing apparatus 1200 (for example, the memory 1201, the processor 1202, and the communication interface 1203).
  • image processing apparatus 1200 only shows a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the image processing apparatus 1200 may also include other necessary for normal operation. Device. At the same time, according to specific needs, those skilled in the art should understand that the above-mentioned image processing apparatus 1200 may further include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the above-mentioned image processing apparatus 1200 may also include only the components necessary for implementing the embodiments of the present application, and not necessarily all the components shown in FIG. 23.
  • the embodiment of the present application also provides a chip, which includes a transceiver unit and a processing unit.
  • the transceiver unit may be an input/output circuit or a communication interface;
  • the processing unit is a processor, microprocessor, or integrated circuit integrated on the chip.
  • the chip can execute the image processing method in the above method embodiment.
  • the embodiment of the present application also provides a computer-readable storage medium on which an instruction is stored, and the image processing method in the foregoing method embodiment is executed when the instruction is executed.
  • the embodiments of the present application also provide a computer program product containing instructions that, when executed, execute the image processing method in the foregoing method embodiments.
  • the memory may include a read-only memory and a random access memory, and provide instructions and data to the processor.
  • a part of the processor may also include a non-volatile random access memory.
  • the processor may also store device type information.
  • the memory may include a read-only memory and a random access memory, and provide instructions and data to the processor.
  • a part of the processor may also include a non-volatile random access memory.
  • the processor may also store device type information.
  • the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not correspond to the embodiments of the present application.
  • the implementation process constitutes any limitation.
  • the disclosed system, device, and method can be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or It can be integrated into another system, or some features can be ignored or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
  • the functional modules in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the technical solution of the present application essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disks or optical disks and other media that can store program codes. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Biophysics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Processing (AREA)

Abstract

本申请公开了人工智能领域中计算机视觉领域的一种图像处理方法以及图像处理装置，该图像处理方法包括：获取待处理原始raw域图像对应的通道图像，其中，该通道图像包括至少两个第一通道图像；对该第一通道图像进行图像修复处理，得到修复后的第一通道图像；根据该修复后的第一通道图像对该待处理raw域图像进行图像修复处理，得到联合去马赛克和去噪处理后的raw域图像。本申请的技术方案能够提升联合去马赛克和去噪后的raw域图像的图像质量。

Description

图像处理方法以及图像处理装置
本申请要求于2020年2月21日提交中国专利局、申请号为202010107896.9、申请名称为“图像处理方法以及图像处理装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及人工智能领域，更具体地，涉及计算机视觉领域中的图像处理方法以及图像处理装置。
背景技术
计算机视觉是各个应用领域，如制造业、检验、文档分析、医疗诊断，和军事等领域中各种智能/自主***中不可分割的一部分，它是一门关于如何运用照相机/摄像机和计算机来获取我们所需的，被拍摄对象的数据与信息的学问。形象地说，就是给计算机安装上眼睛(照相机/摄像机)和大脑(算法)用来代替人眼对目标进行识别、跟踪和测量等，从而使计算机能够感知环境。因为感知可以看作是从感官信号中提取信息，所以计算机视觉也可以看作是研究如何使人工***从图像或多维数据中“感知”的科学。总的来说，计算机视觉就是用各种成像***代替视觉器官获取输入信息，再由计算机来代替大脑对这些输入信息完成处理和解释。计算机视觉的最终研究目标就是使计算机能像人那样通过视觉观察和理解世界，具有自主适应环境的能力。
在计算机视觉领域内，常常需要利用成像设备获取数字图像，并对数字图像进行识别或分析。例如，终端设备产生的标准彩色图像通常是通过对色彩滤波阵列数据经过一系列的图像信号处理管道算法处理得到，因此，生成图像的质量好坏往往和图像信号处理管道算法密切相关，而去马赛克(demosacing)和去噪(denoising)是图像信号处理管道算法的重要组成部分；其中，去马赛克是指从彩色滤波阵列感光器件所输出的不完全取样的色彩信号中恢复/重建出全彩图像的过程；由于含有噪声的图像会影响图像的显示效果、以及图像的分析和识别，通过图像去噪可以去除彩色滤波阵列数据中存在的噪声。
目前，联合去马赛克与去噪的方法可以是通过对单通道的拜耳图像进行采样从而获取四通道的图像与三通道的掩膜图像，通过四通道的图像与三通道的掩膜图像进行融合从而得到联合去马赛克与去噪处理后的标准图像；但是，由于在联合去马赛克与去噪过程中并没有充分利用获取的色彩滤波阵列数据，从而导致联合去马赛克与去噪后的图像中可能存在伪影现象，甚至造成图像模糊；因此，如何提高联合去马赛克与去噪后的图像质量成为一个亟需解决的问题。
发明内容
本申请提供一种图像处理方法以及图像处理装置，能够在一定程度上避免出现伪影现 象，提高联合去马赛克和去噪后的raw域图像的图像质量。
第一方面，提供一种图像处理方法，包括：获取待处理原始raw域图像对应的通道图像，其中，所述通道图像包括至少两个第一通道图像；对所述第一通道图像进行图像修复处理，得到修复后的第一通道图像；根据所述修复后的第一通道图像对所述待处理raw域图像进行图像修复处理，得到联合去马赛克和去噪处理后的raw域图像。
其中，待处理原始raw图像可以是指传感器采集的raw域图像，其中，待处理raw域图像是指未经过图像信号处理器(image signal processor，ISP)处理的原始域图像。
在一种可能的实现方式中，上述待处理raw域图像可以是指图像具有去马赛克和去噪需求的拜耳格式的图像，即图像中可能存在部分额外的信息，可以是噪声或者伪影等。
应理解，图像修复处理是指恢复图像损失的部分并基于图像信息将它们重建的技术。通过图像修复处理可以试图估计原始图像信息，对破损区域进行修复和改善，从而提高图像的视觉质量。其中，图像修复处理可以包括卷积处理、重建处理等。
还应理解，上述对第一通道图像进行图像修复处理，得到修复后的第一通道图像可以是指对第一通道图像进行初步的图像修复处理，得到初步修复后的第一通道图像。
本申请实施例提供的图像处理方法中，可以对待处理raw域图像的通道图像进行预先的图像修复，得到初步修复的通道图像；基于初步修复的通道图像引导待处理raw域图像进行图像修复处理，从而得到联合去马赛克和去噪后的raw域；通过本申请实施例的图像处理方法能够在一定程度上避免彩色纹理的出现，使得处理后raw域图像能够保留图像细节，提升图像质量。
结合第一方面，在一种可能的实现方式中，还包括：获取所述待处理raw域图像的密集图，其中，所述密集图用于指示所述待处理raw域图像中不同频率的纹理区域；
所述根据所述修复后的第一通道图像对所述待处理raw域图像进行图像修复处理，得到联合去马赛克和去噪处理后的raw域图像，包括：通过对所述密集图和所述待处理raw域图像进行卷积处理，得到第一图像特征；通过所述修复后的第一通道图像引导所述第一图像特征进行卷积处理，得到所述联合去马赛克和去噪后的raw域图像。
需要说明的是，上述密集图可以用于指示待处理raw域图像中不同频率的纹理区域；在密集图中对于高频纹理区域响应较高，对于低频纹理区域响应较低，从而能够根据密集图区分出待处理raw域图像中的不同纹理区域。
其中，上述第一图像特征可以是指对待处理raw域图像进行初步修复得到的图像特征，即第一图像特征中可能还存在部分噪声与伪影现象，并未得到真正的联合去马赛克和去噪后的raw域图像。
在本申请实施例提供的图像处理方法中，通过向神经网络中输入密集图，可以使得神经网络在对待处理raw域图像进行卷积处理时，能够区分待处理raw域图像中的不同纹理区域即纹理密集区域与纹理稀疏区域，从而能够提升去噪效果以及保持边缘细节。
在本申请的实施例提供的图像处理方法中，可以先对待处理raw域图像进行初步的图像修复，即对待处理raw域图像进行卷积操作，得到输出的第一图像特征，该第一图像特征并非真正的干净图像可能还存在部分噪声或者伪影等；进一步，可以根据初步修复的通道图像对第一图像特征进行自适应卷积处理从而能够实现完全去除图像特征中的噪声和马赛克得到最终的联合去马赛克和去噪后的raw域图像。
结合第一方面，在一种可能的实现方式中，所述通过所述修复后的第一通道图像引导所述第一图像特征进行卷积处理，得到所述联合去马赛克和去噪后的raw域图像，包括：
根据所述修复后的第一通道图像的像素空间分布对所述第一图像特征进行卷积处理，得到所述联合去马赛克和去噪后的raw域图像，其中，所述像素空间分布用于指示所述修复后的第一通道图像中不同像素点分布的关联性。
需要说明的是，通过可以上述修复后的通道图像即预先修复的通道图像的像素空间分布对初步修复的图像特征引导，使得初步修复的图像特征进行进一步的卷积处理，即使得初步修复的图像特征也能够在一定程度上满足与预先修复的通道图像中相同或者近似的像素空间分布，其中，像素空间分布可以用于指示不同像素点分布的关联性。
结合第一方面，在一种可能的实现方式中，还包括：获取随机噪声图；
所述通过对所述密集图和所述待处理raw域图像进行卷积处理，得到第一图像特征，包括：对所述密集图、所述随机噪声图以及所述待处理raw域图像进行卷积处理，得到所述第一图像特征。
在本申请实施例提供的图像处理方法中，还可以向神经网络中输入随机噪声图，从而提升神经网络对待处理raw域图像的去噪效果。
结合第一方面，在一种可能的实现方式中，所述第一通道图像为绿色G通道图像。
在一种可能的实现方式中，待处理raw域图像对应的通道图像可以指RGGB通道图像，或者，待处理raw域图像对应的通道图像也可以是指GBBR通道图像；则第一通道图像可以是指G通道图像。
结合第一方面，在一种可能的实现方式中，所述第一通道图像为黄色Y通道图像。
在一种可能的实现方式中，待处理raw域图像对应的通道图像可以指RYYB通道图像，则第一通道图像可以是指Y通道图像。
需要说明的是，第一通道图像可以是指在待处理raw域图像对应的多个通道图像中具有图像信息最多的通道图像；其中，图像信息可以是指图像中的高频信息；比如，可以是指图像中的细节信息、纹理信息以及边缘信息。
第二方面，提供一种图像处理方法，应用具有显示屏和摄像头的电子设备，其特征在于，包括：检测到用户指示相机的操作；响应于所述操作，在所述显示屏内显示输出图像，其中，所述输出图像是根据联合去马赛克和去噪后的原始raw域图像得到的，所述联合去马赛克和去噪后的raw域图像是针对所述摄像头采集到的待处理raw域图像进行图像修复处理后得到的raw域图像，修复后的第一通道图像用于所述待处理raw域图像的图像修复处理过程中，所述待处理raw域图像包括至少两个第一通道图像，所述修复后的第一通道图像是指对所述第一通道图像进行图像修复处理后的通道图像。
应理解，在上述第一方面中对相关内容的扩展、限定、解释和说明也适用于第二方面中相同的内容。
结合第二方面，在一种可能的实现方式中，所述检测到用户指示相机的操作，包括：
检测到所述用户指示第一处理模式的操作，所述第一处理模式用于对所述待处理raw域图像进行图像修复处理；或者，
检测到所述用户用于指示拍摄的操作。
结合第二方面，在一种可能的实现方式中，所述对待处理raw域图像进行图像修复处 理的过程还包括：获取所述待处理raw域图像的密集图，其中，所述密集图用于指示所述待处理raw域图像中不同频率的纹理区域；通过对所述密集图和所述待处理raw域图像进行卷积处理，得到第一图像特征；通过所述修复后的第一通道图像引导所述第一图像特征进行卷积处理，得到所述联合去马赛克和去噪后的raw域图像。
结合第二方面，在一种可能的实现方式中，所述通过所述修复后的第一通道图像引导所述第一图像特征进行卷积处理，得到所述联合去马赛克和去噪后的raw域图像，包括：根据所述修复后的第一通道图像的像素空间分布对所述第一图像特征进行卷积处理，得到所述联合去马赛克和去噪后的raw域图像，其中，所述像素空间分布用于指示所述修复后的第一通道图像中不同像素点分布的关联性。
结合第二方面，在一种可能的实现方式中，还包括：获取随机噪声图；
所述通过对所述密集图和所述待处理raw域图像进行卷积处理，得到第一图像特征，包括：对所述密集图、所述随机噪声图以及所述待处理raw域图像进行卷积处理，得到所述第一图像特征。
结合第二方面，在一种可能的实现方式中，所述第一通道图像为绿色G通道图像。
结合第二方面，在一种可能的实现方式中，所述第一通道图像为黄色Y通道图像。
第三方面，提供了一种图像处理装置，包括用于执行第一方面以及第一方面任意一种可能实现方式的图像处理方法的功能模块/单元。
第四方面，提供了一种图像处理装置，包括用于执行第二方面以及第二方面任意一种可能实现方式的图像处理方法的功能模块/单元。
第五方面，提供一种计算机可读介质，该计算机可读介质存储用于设备执行的程序代码，该程序代码包括用于执行第一方面或者第一方面的任意一种实现方式中的图像处理方法。
第六方面，提供一种计算机可读介质，该计算机可读介质存储用于设备执行的程序代码，该程序代码包括用于执行第二方面或者第二方面中的任意一种实现方式中的图像处理方法。
第七方面，提供了一种计算机程序产品，所述计算机程序产品包括：计算机程序代码，当所述计算机程序代码在计算机上运行时，使得计算机执行上述各方面中的方法。
需要说明的是，上述计算机程序代码可以全部或者部分存储在第一存储介质上，其中第一存储介质可以与处理器封装在一起的，也可以与处理器单独封装，本申请实施例对此不作具体限定。
第八方面，提供一种芯片，所述芯片包括处理器与数据接口，所述处理器通过所述数据接口读取存储器上存储的指令，执行上述第一方面或第一方面中的任意一种实现方式中的图像处理方法。
可选地，作为一种实现方式，所述芯片还可以包括存储器，所述存储器中存储有指令，所述处理器用于执行所述存储器上存储的指令，当所述指令被执行时，所述处理器用于执行第一方面或者第一方面中的任意一种实现方式中的图像处理方法。
第九方面，提供一种芯片，所述芯片包括处理器与数据接口，所述处理器通过所述数据接口读取存储器上存储的指令，执行上述第二方面或第二方面中的任意一种实现方式中的图像处理方法。
可选地，作为一种实现方式，所述芯片还可以包括存储器，所述存储器中存储有指令，所述处理器用于执行所述存储器上存储的指令，当所述指令被执行时，所述处理器用于执行第二方面或者第二方面中的任意一种实现方式中的图像处理方法。
附图说明
图1是本申请实施例提供的图像处理方法的应用场景的示意图；
图2是本申请实施例提供的一种应用场景的示意图；
图3是本申请实施例提供的一种应用场景的示意图；
图4是本申请实施例提供的一种应用场景的示意图；
图5是本申请实施例提供的***架构的结构示意图；
图6是本申请实施例提供的一种卷积神经网络结构示意图；
图7是本申请实施例提供的一种芯片硬件结构示意图；
图8是本申请实施例提供了一种***架构的示意图；
图9是本申请实施例提供的联合去马赛克和去噪***的示意图；
图10是本申请实施例提供的图像处理方法的示意性流程图；
图11是本申请实施例提供的图像联合去马赛克和去噪方法的示意性流程图；
图12是本申请实施例提供的初步图像修复的示意图；
图13是本申请实施提供的修复通道图像的示意图；
图14是本申请实施提供的获取待处理raw域图像的密集图的示意图；
图15是本申请实施例提供的通过修复后的通道图像进行自适应卷积的示意图；
图16是本申请实施例提供的图像处理方法的示意性流程图；
图17是本申请实施例提供的一组显示界面示意图；
图18是本申请实施例提供的另一组显示界面示意图；
图19是本申请实施例提供的另一个显示界面示意图；
图20是本申请实施例提供的另一个显示界面示意图；
图21是本申请实施例提供的图像处理装置的示意性框图；
图22是本申请实施例提供的图像处理装置的示意性框图；
图23是本申请实施例提供的图像处理装置的示意性框图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行描述，显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
应理解，在本申请的各实施例中，“第一”、“第二”等仅是为了指代不同的对象，并不表示对指代的对象有其它限定。
图1是本申请实施例提供的图像处理方法的应用场景的示意图。
如图1所示，本申请实施例的图像处理方法可以应用于智能终端，比如，可以对智能设备中的摄像头采集的待处理原始raw域图像进行图像修复处理，得到该待处理raw域图 像经过图像修复处理后的联合去马赛克和去噪处理后的raw域图像，对联合去马赛克和去噪后的raw域图像进行后续的图像处理可以得到输出图像；其中，后续的图像处理可以包括对联合去马赛克和去噪后的raw域图像进行白平衡、颜色校正、色调映射等图像处理；raw域图像可以是指拜耳格式的图像。
应理解，图像修复处理是指恢复图像损失的部分并基于图像信息将它们重建的技术。通过图像修复处理可以试图估计原始图像信息，对破损区域进行修复和改善，从而提高图像的视觉质量。其中，上述智能终端可以为移动的或固定的，例如，该智能终端可以是具有图像处理功能的移动电话、平板个人电脑(tablet personal computer，TPC)、媒体播放器、智能电视、笔记本电脑(laptop computer，LC)、个人数字助理(personal digital assistant，PDA)、个人计算机(personal computer，PC)、照相机、摄像机、智能手表、增强现实(augmented reality，AR)/虚拟现实(virtual reality，VR)，可穿戴式设备(wearable device，WD)或者自动驾驶的车辆等，本申请实施例对此不作限定。
下面对本申请实施例的具体应用场景进行举例说明。
应用场景一：智能终端拍照领域
在一个实施例中，如图2所示，本申请实施例的图像处理方法可以应用于智能终端设备(例如，手机)的拍摄。在通过智能终端的摄像头对物体进行拍照的时候，可能会遇到密集纹理的图像，此时智能终端中的图像信息处理器可能会导致获取的图像存在彩色伪影或者拉链状的噪声等问题。通过本申请实施例的图像方法，即联合去马赛克和去噪的方法可以对获取的质量较差的原始raw域图像进行图像修复处理得到视觉质量提升的输出图像(或者输出视频)。
需要说明的是，在图2中为了区别于灰度图像部分，彩色图像部分通过斜线填充来表示。
示例性地，可以通过本申请实施例的图像处理方法在智能终端进行实时拍照时，可以对获取的原始raw域图像进行图像修复处理，得到的联合去马赛克和去噪处理后的raw域图像，并对联合去马赛克和去噪后的raw域图像进行后续的图像处理，得到输出图像；其中，后续的图像处理可以包括对联合去马赛克和去噪后的raw域图像进行白平衡、颜色校正、色调映射等图像处理；将输出图像显示在智能终端的屏幕上。
示例性地，可以通过本申请实施例的图像处理方法对可以获取的原始raw域图像(例如，色彩滤波阵列信号)进行图像修复处理，得到联合去马赛克和去噪后的raw域，并对联合去马赛克和去噪后的raw域图像进行后续的图像处理，得到输出图像，其中，后续的图像处理可以包括对联合去马赛克和去噪后的raw域图像进行白平衡、颜色校正、色调映射等图像处理；将输出图像保存至智能终端的相册中。
示例性地，本申请提出了一种图像处理方法，应用于具有显示屏和摄像头的电子设备，包括：检测到用户指示相机的操作；响应于所述操作，在所述显示屏内显示输出图像，其中，所述输出图像是根据联合去马赛克和去噪后的原始raw域图像得到的，所述联合去马赛克和去噪后的raw域图像是针对所述摄像头采集到的待处理raw域图像进行图像修复处理后得到的raw域图像，修复后的第一通道图像用于所述待处理raw域图像的图像修复处理过程中，所述待处理raw域图像包括至少两个第一通道图像，所述修复后的第一通道图像是指对所述第一通道图像进行图像修复处理后的通道图像。
需要说明的是，本申请实施例提供的图像处理方法同样适用于后面图5至图15中相关实施例中对图像处理方法相关内容的扩展、限定、解释和说明，此处不再赘述。
应用场景二：自动驾驶领域
在一个实施例中，如图3所示，本申请实施例的图像处理方法可以应用于自动驾驶领域。例如，可以应用于自动驾驶车辆的导航***中，通过本申请中的图像处理方法可以使得自动驾驶车辆在道路行驶的导航过程中，通过获取的画质较低的原始raw域道路图像(或原始raw域道路视频)进行图像修复处理即联合去马赛克和去噪后的raw域图像，对联合去马赛克和去噪后的raw域图像进行后续的图像处理，得到处理后的道路图像(或者道路视频)，从而实现自动驾驶车辆的安全性，其中，后续的图像处理可以包括对联合去马赛克和去噪后的raw域图像进行白平衡、颜色校正、色调映射等图像处理。
示例性地，本申请提供了一种图像处理方法，包括：获取原始raw域道路图像对应的通道图像，其中，所述通道图像包括至少两个第一通道图像；对所述第一通道图像进行图像修复处理，得到修复后的第一通道图像；根据所述修复后的第一通道图像对所述raw域道路图像进行图像修复处理，得到联合去马赛克与去噪后的raw域道路图像。
需要说明的是，本申请实施例提供的图像处理方法同样适用于后面图5至图15中相关实施例中对图像处理方法相关内容的扩展、限定、解释和说明，此处不再赘述。
应用场景三：平安城市领域
在一个实施例中，如图4所示，本申请实施例的图像处理方法可以应用于平安城市领域，比如，安防领域。例如，本申请实施例的图像处理方法可以应用于平安城市的监控图像处理，比如，公共场合的监控设备采集到的raw域图像(或者，raw域视频)往往受到天气、距离等因素的影响，存在图像模糊，图像画质较低等问题。通过本申请的图像处理方法可以对采集到的raw域图像进行图像修复处理即联合去马赛克和去噪处理，得到联合去马赛克和去噪后的raw域图像；进一步，对联合去马赛克和去噪后的raw域图像进行后续的图像处理，得到处理后的街景图像，其中，后续的图像处理可以包括对联合去马赛克和去噪后的raw域图像进行白平衡、颜色校正、色调映射等图像处理；基于处理后的街景图像可以为公安人员恢复出车牌号码、清晰人脸等重要信息，为案件侦破提供重要的线索信息。
示例性地，本申请提供了一种图像处理方法，包括：获取原始raw域街景图像对应的通道图像，其中，所述通道图像包括至少两个第一通道图像；对所述第一通道图像进行图像修复处理，得到修复后的第一通道图像；根据所述修复后的第一通道图像对所述raw域街景图像进行图像修复处理，得到联合去马赛克与去噪处理后的raw域街景图像。
需要说明的是，本申请实施例提供的图像处理方法同样适用于后面图5至图15中相关实施例中对图像处理方法相关内容的扩展、限定、解释和说明，此处不再赘述。
应理解，上述为对应用场景的举例说明，并不对本申请的应用场景作任何限定。
由于本申请实施例涉及大量神经网络的应用，为了便于理解，下面先对本申请实施例可能涉及的神经网络的相关术语和概念进行介绍。
(1)神经网络
神经网络可以是由神经单元组成的，神经单元可以是指以x s和截距1为输入的运算单元，该运算单元的输出可以为：
Figure PCTCN2020112619-appb-000001
其中，s＝1、2、……n，n为大于1的自然数，W s为x s的权重，b为神经单元的偏置。f为神经单元的激活函数(activation functions)，用于将非线性特性引入神经网络中，来将神经单元中的输入信号转换为输出信号。该激活函数的输出信号可以作为下一层卷积层的输入，激活函数可以是sigmoid函数。神经网络是将多个上述单一的神经单元联结在一起形成的网络，即一个神经单元的输出可以是另一个神经单元的输入。每个神经单元的输入可以与前一层的局部接受域相连，来提取局部接受域的特征，局部接受域可以是由若干个神经单元组成的区域。
(2)深度神经网络
深度神经网络(deep neural network，DNN)，也称多层神经网络，可以理解为具有多层隐含层的神经网络。按照不同层的位置对DNN进行划分，DNN内部的神经网络可以分为三类：输入层，隐含层，输出层。一般来说第一层是输入层，最后一层是输出层，中间的层数都是隐含层。层与层之间是全连接的，也就是说，第i层的任意一个神经元一定与第i+1层的任意一个神经元相连。
虽然DNN看起来很复杂，但是就每一层的工作来说，其实并不复杂，简单来说就是如下线性关系表达式：
Figure PCTCN2020112619-appb-000002
其中，
Figure PCTCN2020112619-appb-000003
是输入向量，
Figure PCTCN2020112619-appb-000004
是输出向量，
Figure PCTCN2020112619-appb-000005
是偏移向量，W是权重矩阵(也称系数)，α()是激活函数。每一层仅仅是对输入向量
Figure PCTCN2020112619-appb-000006
经过如此简单的操作得到输出向量
Figure PCTCN2020112619-appb-000007
由于DNN层数多，系数W和偏移向量
Figure PCTCN2020112619-appb-000008
的数量也比较多。这些参数在DNN中的定义如下所述：以系数W为例：假设在一个三层的DNN中，第二层的第4个神经元到第三层的第2个神经元的线性系数定义为
Figure PCTCN2020112619-appb-000009
上标3代表系数W所在的层数，而下标对应的是输出的第三层索引2和输入的第二层索引4。
综上，第L-1层的第k个神经元到第L层的第j个神经元的系数定义为
Figure PCTCN2020112619-appb-000010
需要注意的是，输入层是没有W参数的。在深度神经网络中，更多的隐含层让网络更能够刻画现实世界中的复杂情形。理论上而言，参数越多的模型复杂度越高，“容量”也就越大，也就意味着它能完成更复杂的学习任务。训练深度神经网络的也就是学习权重矩阵的过程，其最终目的是得到训练好的深度神经网络的所有层的权重矩阵(由很多层的向量W形成的权重矩阵)。
(3)卷积神经网络
卷积神经网络(convolutional neuron network，CNN)是一种带有卷积结构的深度神经网络。卷积神经网络包含了一个由卷积层和子采样层构成的特征抽取器，该特征抽取器可以看作是滤波器。卷积层是指卷积神经网络中对输入信号进行卷积处理的神经元层。在卷积神经网络的卷积层中，一个神经元可以只与部分邻层神经元连接。一个卷积层中，通常包含若干个特征平面，每个特征平面可以由一些矩形排列的神经单元组成。同一特征平面的神经单元共享权重，这里共享的权重就是卷积核。共享权重可以理解为提取图像信息的方式与位置无关。卷积核可以以随机大小的矩阵的形式初始化，在卷积神经网络的训练过程中卷积核可以通过学习得到合理的权重。另外，共享权重带来的直接好处是减少卷积神经网络各层之间的连接，同时又降低了过拟合的风险。
(4)损失函数
在训练深度神经网络的过程中，因为希望深度神经网络的输出尽可能的接近真正想要预测的值，所以可以通过比较当前网络的预测值和真正想要的目标值，再根据两者之间的差异情况来更新每一层神经网络的权重向量(当然，在第一次更新之前通常会有初始化的过程，即为深度神经网络中的各层预先配置参数)，比如，如果网络的预测值高了，就调整权重向量让它预测低一些，不断地调整，直到深度神经网络能够预测出真正想要的目标值或与真正想要的目标值非常接近的值。因此，就需要预先定义“如何比较预测值和目标值之间的差异”，这便是损失函数(loss function)或目标函数(objective function)，它们是用于衡量预测值和目标值的差异的重要方程。其中，以损失函数举例，损失函数的输出值(loss)越高表示差异越大，那么深度神经网络的训练就变成了尽可能缩小这个loss的过程。
(5)反向传播算法
神经网络可以采用误差反向传播(back propagation，BP)算法在训练过程中修正初始的神经网络模型中参数的大小，使得神经网络模型的重建误差损失越来越小。具体地，前向传递输入信号直至输出会产生误差损失，通过反向传播误差损失信息来更新初始的神经网络模型中参数，从而使误差损失收敛。反向传播算法是以误差损失为主导的反向传播运动，旨在得到最优的神经网络模型的参数，例如权重矩阵。
(6)联合去马赛克与和去噪
联合去马赛克和去噪(Joint demosaicing and denoising，JDD)是指将终端设备(例如，手机)，或者相机等设备图像信号处理管道中的去马赛克和去噪作为联合问题一起处理的算法，其中，去马赛克是指从彩色滤波阵列感光器件所输出的不完全取样的色彩信号中恢复/重建出全彩图像的过程；去噪可以是指去除图像中错误和额外的信息。
(7)图像噪声
图像噪声(image noise)是图像中一种亮度或颜色信息的随机变化(被拍摄物体本身并没有)，通常是电子噪声的表现。它一般是由扫描仪或数码相机的传感器和电路产生的，也可能是受胶片颗粒或者理想光电探测器中不可避免的散粒噪声影响产生的。图像噪声是图像拍摄过程中不希望存在的副产品，给图像带来了错误和额外的信息。
(8)拜耳(bayer)图像
利用拜耳(bayer)格式的图像时，即可以在一块滤镜上设置的不同的颜色，通过分析人眼对颜色的感知发现，人眼对绿色比较敏感，所以在bayer格式的图片中绿色格式的像素的数目可以是r像素和g像素的和。
图5示出了本申请实施例提供的一种***架构100。
图5示出了本申请实施例提供的一种***架构100。
在图5中，数据采集设备160用于采集训练数据。针对本申请实施例的图像处理方法来说，可以通过训练数据对图像处理模型(又称为联合去马赛克和去噪网络)进行进一步训练，即数据采集设备160采集的训练数据可以是训练图像。
示例性地，在本申请实施例中训练图像处理模型的训练数据可以包括原始图像、与原始图像对应样本图像。
例如，原始图像可以是指未经过图像信号处理管道的raw域图像；比如，拜耳格式的图像，样本图像可以是指经过图像修复处理后的输出图像，比如，可以是指相对于原始raw 域图像而言在纹理或者细节等一个或多个方面均得到提升后的联合去马赛克和去噪处理后的图像。
在采集到训练数据之后，数据采集设备160将这些训练数据存入数据库130，训练设备120基于数据库130中维护的训练数据训练得到目标模型/规则101(即本申请实施例中的图像处理模型)。训练设备120将训练数据输入图像处理模型，直到训练图像处理模型输出的预测图像与样本图像之间的差值满足预设条件(例如，预测图像与样本图像差值小于一定阈值，或者预测图像与样本图像的差值保持不变或不再减少)，从而完成目标模型/规则101的训练。
示例性地，本申请实施例中用于执行图像处理方法的图像处理模型可以实现端到端的训练，比如，图像处理模型可以通过输入图像与输入图像对应的联合去马赛克和去噪后的图像(例如，真值图像)实现端到端的训练。
在本申请提供的实施例中，该目标模型/规则101是通过训练图像处理模型得到的，即图像处理模型可以是指联合去马赛克和去噪的模型。需要说明的是，在实际的应用中，所述数据库130中维护的训练数据不一定都来自于数据采集设备160的采集，也有可能是从其他设备接收得到的。
另外需要说明的是，训练设备120也不一定完全基于数据库130维护的训练数据进行目标模型/规则101的训练，也有可能从云端或其他地方获取训练数据进行模型训练，上述描述不应该作为对本申请实施例的限定。还需要说明的是，数据库130中维护的训练数据中的至少部分数据也可以用于执行设备110对待处理处理进行处理的过程。
根据训练设备120训练得到的目标模型/规则101可以应用于不同的***或设备中，如应用于图5所示的执行设备110，所述执行设备110可以是终端，如手机终端，平板电脑，笔记本电脑，AR/VR，车载终端等，还可以是服务器或者云端等。
在图5中，执行设备110配置输入/输出(input/output，I/O)接口112，用于与外部设备进行数据交互，用户可以通过客户设备140向I/O接口212输入数据，所述输入数据在本申请实施例中可以包括：客户设备输入的待处理图像。
预处理模块113和预处理模块114用于根据I/O接口112接收到的输入数据(如待处理图像)进行预处理，在本申请实施例中，也可以没有预处理模块113和预处理模块114(也可以只有其中的一个预处理模块)，而直接采用计算模块111对输入数据进行处理。
在执行设备110对输入数据进行预处理，或者在执行设备110的计算模块111执行计算等相关的处理过程中，执行设备110可以调用数据存储***150中的数据、代码等以用于相应的处理，也可以将相应处理得到的数据、指令等存入数据存储***150中。
最后，I/O接口112将处理结果，如上述得到待处理raw域图像的联合去马赛克和去噪后的raw域图像，即将得到的输出图像返回给客户设备140，从而提供给用户。
值得说明的是，训练设备120可以针对不同的目标或称不同的任务，基于不同的训练数据生成相应的目标模型/规则101，该相应的目标模型/规则101即可以用于实现上述目标或完成上述任务，从而为用户提供所需的结果。
在图5中所示情况下，在一种情况下，用户可以手动给定输入数据，该手动给定可以通过I/O接口112提供的界面进行操作。
另一种情况下，客户设备140可以自动地向I/O接口112发送输入数据，如果要求客 户设备140自动发送输入数据需要获得用户的授权，则用户可以在客户设备140中设置相应权限。用户可以在客户设备140查看执行设备110输出的结果，具体的呈现形式可以是显示、声音、动作等具体方式。客户设备140也可以作为数据采集端，采集如图所示输入I/O接口112的输入数据及输出I/O接口112的输出结果作为新的样本数据，并存入数据库130。当然，也可以不经过客户设备140进行采集，而是由I/O接口112直接将如图所示输入I/O接口112的输入数据及输出I/O接口112的输出结果，作为新的样本数据存入数据库130。
值得注意的是，图5仅是本申请实施例提供的一种***架构的示意图，图中所示设备、器件、模块等之间的位置关系不构成任何限制。例如，在图5中，数据存储***150相对执行设备110是外部存储器，在其它情况下，也可以将数据存储***150置于执行设备110中。
如图5所示，根据训练设备120训练得到目标模型/规则101，该目标模型/规则101在本申请实施例中可以是图像处理模型，具体的，本申请实施例提供的图像处理模型可以是深度神经网络，卷积神经网络，或者，可以是深度卷积神经网络等。
下面结合图6重点对卷积神经网络的结构进行详细的介绍。如上文的基础概念介绍所述，卷积神经网络是一种带有卷积结构的深度神经网络，是一种深度学习(deep learning)架构，深度学习架构是指通过机器学习的算法，在不同的抽象层级上进行多个层次的学习。作为一种深度学习架构，卷积神经网络是一种前馈(feed-forward)人工神经网络，该前馈人工神经网络中的各个神经元可以对输入其中的图像作出响应。
本申请实施例中图像处理模型的结构可以如图6所示。在图6中，卷积神经网络200可以包括输入层210，卷积层/池化层220(其中，池化层为可选的)，全连接层230以及输出层240。其中，输入层210可以获取待处理图像，并将获取到的待处理图像交由卷积层/池化层220以及全连接层230进行处理，可以得到图像的处理结果。下面对图6中的CNN 200中内部的层结构进行详细的介绍。
卷积层/池化层220：
如图6所示卷积层/池化层220可以包括如示例221-226层，举例来说：在一种实现中，221层为卷积层，222层为池化层，223层为卷积层，224层为池化层，225为卷积层，226为池化层；在另一种实现方式中，221、222为卷积层，223为池化层，224、225为卷积层，226为池化层，即卷积层的输出可以作为随后的池化层的输入，也可以作为另一个卷积层的输入以继续进行卷积操作。
下面将以卷积层221为例，介绍一层卷积层的内部工作原理。
卷积层221可以包括很多个卷积算子，卷积算子也称为核，其在图像处理中的作用相当于一个从输入图像矩阵中提取特定信息的过滤器，卷积算子本质上可以是一个权重矩阵，这个权重矩阵通常被预先定义，在对图像进行卷积操作的过程中，权重矩阵通常在输入图像上沿着水平方向一个像素接着一个像素(或两个像素接着两个像素等，这取决于步长stride的取值)的进行处理，从而完成从图像中提取特定特征的工作。该权重矩阵的大小应该与图像的大小相关，需要注意的是，权重矩阵的纵深维度(depth dimension)和输入图像的纵深维度是相同的，在进行卷积运算的过程中，权重矩阵会延伸到输入图像的整个深度。因此，和一个单一的权重矩阵进行卷积会产生一个单一纵深维度的卷积化输出， 但是大多数情况下不使用单一权重矩阵，而是应用多个尺寸(行×列)相同的权重矩阵，即多个同型矩阵。每个权重矩阵的输出被堆叠起来形成卷积图像的纵深维度，这里的维度可以理解为由上面所述的“多个”来决定。
不同的权重矩阵可以用来提取图像中不同的特征，例如，一个权重矩阵用来提取图像边缘信息，另一个权重矩阵用来提取图像的特定颜色，又一个权重矩阵用来对图像中不需要的噪点进行模糊化等。该多个权重矩阵尺寸(行×列)相同，经过该多个尺寸相同的权重矩阵提取后的卷积特征图的尺寸也相同，再将提取到的多个尺寸相同的卷积特征图合并形成卷积运算的输出。
这些权重矩阵中的权重值在实际应用中需要经过大量的训练得到，通过训练得到的权重值形成的各个权重矩阵可以用来从输入图像中提取信息，从而使得卷积神经网络200进行正确的预测。
当卷积神经网络200有多个卷积层的时候，初始的卷积层(例如，221)往往提取较多的一般特征，一般特征也可以称之为低级别的特征；随着卷积神经网络200深度的加深，越往后的卷积层(例如，226)提取到的特征越来越复杂，比如，高级别的语义之类的特征，语义越高的特征越适用于待解决的问题。
池化层：
由于常常需要减少训练参数的数量，因此卷积层之后常常需要周期性的引入池化层，在如图6中220所示例的221-226各层，可以是一层卷积层后面跟一层池化层，也可以是多层卷积层后面接一层或多层池化层。在图像处理过程中，池化层的目的就是减少图像的空间大小。池化层可以包括平均池化算子和/或最大池化算子，以用于对输入图像进行采样得到较小尺寸的图像。平均池化算子可以在特定范围内对图像中的像素值进行计算产生平均值作为平均池化的结果。最大池化算子可以在特定范围内取该范围内值最大的像素作为最大池化的结果。
另外，就像卷积层中用权重矩阵的大小应该与图像尺寸相关一样，池化层中的运算符也应该与图像的大小相关。通过池化层处理后输出的图像尺寸可以小于输入池化层的图像的尺寸，池化层输出的图像中每个像素点表示输入池化层的图像的对应子区域的平均值或最大值。
全连接层230：
在经过卷积层/池化层220的处理后，卷积神经网络200还不足以输出所需要的输出信息。因为如前所述，卷积层/池化层220只会提取特征，并减少输入图像带来的参数。然而为了生成最终的输出信息(所需要的类信息或其他相关信息)，卷积神经网络200需要利用全连接层230来生成一个或者一组所需要的类的数量的输出。因此，在全连接层230中可以包括多层隐含层(如图6所示的231、232至23n)以及输出层240，该多层隐含层中所包含的参数可以根据具体的任务类型的相关训练数据进行预先训练得到，例如该任务类型可以包括图像增强，图像识别，图像分类，图像检测以及图像超分辨率重建等等。
在全连接层230中的多层隐含层之后，也就是整个卷积神经网络200的最后层为输出层240，该输出层240具有类似分类交叉熵的损失函数，具体用于计算预测误差，一旦整个卷积神经网络200的前向传播(如图6由210至240方向的传播为前向传播)完成，反向传播(如图6由240至210方向的传播为反向传播)就会开始更新前面提到的各层的权 重值以及偏差，以减少卷积神经网络200的损失，及卷积神经网络200通过输出层输出的结果和理想结果之间的误差。
需要说明的是，图6所示的卷积神经网络仅作为一种本申请实施例图像处理模型的结构示例，在具体的应用中，本申请实施例的图像处理方法所采用的卷积神经网络还可以以其他网络模型的形式存在。
本申请的实施例中，图像处理装置可以包括图6所示的卷积神经网络200，该图像处理装置可以对待处理raw域图像进行卷积处理即联合去马赛克和去噪处理，得到处理后的raw域图像。
图7是本申请实施例提供的一种芯片的硬件结构，该芯片包括神经网络处理器300(neural-network processing unit，NPU)。该芯片可以被设置在如图5所示的执行设备110中，用以完成计算模块111的计算工作。该芯片也可以被设置在如图5所示的训练设备120中，用以完成训练设备120的训练工作并输出目标模型/规则101。如图6所示的卷积神经网络中各层的算法均可在如图7所示的芯片中得以实现。
NPU 300作为协处理器挂载到主中央处理器(central processing unit，CPU)上，由主CPU分配任务。NPU 300的核心部分为运算电路303，控制器304控制运算电路303提取存储器(权重存储器或输入存储器)中的数据并进行运算。
在一些实现中，运算电路303内部包括多个处理单元(process engine,PE)。在一些实现中，运算电路303是二维脉动阵列；运算电路303还可以是一维脉动阵列或者能够执行例如乘法和加法这样的数学运算的其它电子线路。在一些实现中，运算电路303是通用的矩阵处理器。
举例来说，假设有输入矩阵A，权重矩阵B，输出矩阵C；运算电路303从权重存储器302中取矩阵B相应的数据，并缓存在运算电路303中每一个PE上；运算电路303从输入存储器301中取矩阵A数据与矩阵B进行矩阵运算，得到的矩阵的部分结果或最终结果，保存在累加器308(accumulator)中。
向量计算单元307可以对运算电路303的输出做进一步处理，如向量乘，向量加，指数运算，对数运算，大小比较等等。例如，向量计算单元307可以用于神经网络中非卷积/非FC层的网络计算，如池化(pooling)，批归一化(batch normalization)，局部响应归一化(local response normalization)等。
在一些实现种，向量计算单元能307将经处理的输出的向量存储到统一存储器306。例如，向量计算单元307可以将非线性函数应用到运算电路303的输出，例如累加值的向量，用以生成激活值。在一些实现中，向量计算单元307生成归一化的值、合并值，或二者均有。
在一些实现中，处理过的输出的向量能够用作到运算电路303的激活输入，例如用于在神经网络中的后续层中的使用。
统一存储器306用于存放输入数据以及输出数据。权重数据直接通过存储单元访问控制器305(direct memory access controller，DMAC)将外部存储器中的输入数据存入到输入存储器401和/或统一存储器406、将外部存储器中的权重数据存入权重存储器302，以及将统一存储器306中的数据存入外部存储器。
总线接口单元310(bus interface unit，BIU)，用于通过总线实现主CPU、DMAC和 取指存储器309之间进行交互。
与控制器304连接的取指存储器309(instruction fetch buffer)用于存储控制器304使用的指令；控制器304用于调用取指存储器309中缓存的指令，实现控制该运算加速器的工作过程。
一般地，统一存储器306，输入存储器301，权重存储器302以及取指存储器309均为片上(On-Chip)存储器，外部存储器为该NPU外部的存储器，该外部存储器可以为双倍数据率同步动态随机存储器(double data rate synchronous dynamic random access memory，DDR SDRAM)、高带宽存储器(high bandwidth memory，HBM)或其他可读可写的存储器。
其中，图6所示的卷积神经网络中各层的运算可以由运算电路303或向量计算单元307执行。
上文中介绍的图5中的执行设备110能够执行本申请实施例的图像处理方法的各个步骤，图6所示的CNN模型和图7所示的芯片也可以用于执行本申请实施例的图像处理方法的各个步骤。
图8所示是本申请实施例提供了一种***架构400。该***架构包括本地设备420、本地设备430以及执行设备410和数据存储***450，其中，本地设备420和本地设备430通过通信网络与执行设备410连接。
示例性地，执行设备410可以由一个或多个服务器实现。
可选的，执行设备410可以与其它计算设备配合使用。例如：数据存储器、路由器、负载均衡器等设备。执行设备410可以布置在一个物理站点上，或者分布在多个物理站点上。执行设备410可以使用数据存储***450中的数据，或者调用数据存储***450中的程序代码来实现本申请实施例的图像处理方法。
需要说明的是，上述执行设备410也可以称为云端设备，此时执行设备410可以部署在云端。
具体地，执行设备410可以执行以下过程：获取待处理原始raw域图像对应的通道图像，其中，所述通道图像包括至少两个第一通道图像；对所述第一通道图像进行图像修复处理，得到修复后的第一通道图像；根据所述修复后的第一通道图像对所述待处理raw域图像进行图像修复处理，得到联合去马赛克和去噪处理后的raw域图像。
在一种可能的实现方式中，本申请实施例的图像处理方法可以是在云端执行的离线方法，比如，可以由上述执行设备410中执行本申请实施例的图像处理方法。
在一种可能的实现方式中，本申请实施例的图像处理方法可以是由本地设备420或者本地设备430执行。
例如，用户可以操作各自的用户设备(例如，本地设备420和本地设备430)与执行设备410进行交互。每个本地设备可以表示任何计算设备，例如，个人计算机、计算机工作站、智能手机、平板电脑、智能摄像头、智能汽车或其他类型蜂窝电话、媒体消费设备、可穿戴设备、机顶盒、游戏机等。
每个用户的本地设备可以通过任何通信机制/通信标准的通信网络与执行设备410进行交互，通信网络可以是广域网、局域网、点对点连接等方式，或它们的任意组合。
在一种实现方式中，本地设备420、本地设备430可以从执行设备410获取到上述神 经网络模型的相关参数，将神经网络模型部署在本地设备420、本地设备430上，利用该神经网络模型进行图像处理等。
在另一种实现中，执行设备410上可以直接部署神经网络模型，执行设备410通过从本地设备420和本地设备430获取待处理raw域图像对应的通道图像，在神经网络模型中对待处理raw域图像进行图像修复处理，得到联合去马赛克和去噪后的raw域图像。
目前，基于深度学习的联合去马赛克和去噪的方法可以是获取原始图像的拜耳图像，基于单通道的拜耳图像采样成四通道的RGGB图像，此外还可以通过单通道的拜耳图像得到三通道的RGB掩膜图像即分别为红色掩膜图像、蓝色掩膜图像以及绿色掩膜图像，进而将RGGB图像经过卷积层输出的特征图与RGB掩膜图像进行通道并联融合，从而得到干净的RGB图像即联合去马赛克和去噪后的图像；但是，由于仅仅采用了RGB掩膜图像对原有图像进行约束，当在原始图像中的密集纹理区域通过RGB掩膜进行图像处理后在图像中依然可能存在混叠和彩色伪影现象，导致处理后的图像质量较低。
有鉴于此，本申请提出了一种图像处理方法以及图像处理装置，可以通过获取原始raw域图像对应的通道图像，通道图像中可以包括至少两个第一通道图像；对第一通道图像先进行图像修复处理，得到修复处理后的第一通道图像；根据修复后的第一通道图像对raw域图像进行图像修复处理，最终得到联合去马赛克与去噪处理后的raw域图像；在本申请实施例中的图像处理方法中可以预先修复raw域图像的通道图像，通过修复后的通道图像对raw域图像进行引导，从而避免彩色纹理的出现，使得处理后图像能够保留图像细节，提升图像质量。
本申请实施例提供的图像处理方法可部署在相关设备的计算节点上，通过软件算法能够有效的实现本申请实施例中的联合去马赛克和去噪的方法。下面结合图9对本申请实施例的***架构进行详细描述。
如图9所示，本申请实施例提供的图像处理方法可以应用于联合去马赛克和去噪***500，其中，联合去马赛克和去噪***500中可以包括主修复模块510、通道修复模块520、引导模块530；进一步，还可以包括密集图构造模块540以及预训练特征仓库550；下面对各个模块进行详细说明。
主修复模块510：用于对获取的原始raw域图像进行卷积操作，实现初步的联合去马赛克和去噪处理，得到raw域图像的图像特征；其中，raw域图像可以是指具有去马赛克或者去噪需求的拜耳格式的图像。
示例性地，主修复模块510可以对智能终端的信号处理器获取的色彩滤波阵列信号数据(例如，raw域图像)进行重组从而获取四通道图像；将四通道图像与随机噪声图输入至卷积神经网络进行卷积处理，得到初步修复的图像特征。
其中，上述四通道图像可以是指RGGB四通道图像；或者，也可以是指BGGR四通道图像；或者，也可以是指RYYB四通道图像。
可选地，在主修复模块510对raw域图像进行修复时还可以从密集构造模块540获取密集图，从而向卷积神经网络中输入密集图，其中，密集图构造模块540用于获取密集图，密集图可以用于指示待处理raw域图像中不同纹理区域的频率大小；在密集图中对于高频纹理区域响应较高，对于低频纹理区域响应较低，从而能够根据密集图区分出raw域图像中的不同频率大小的纹理区域。
需要说明的是，通过密集图可以使得卷积神经网络更好的识别图像中的高频区域与低频区域，通常图像中的高频区域是指图像中的边缘以及细节区域，相对于低频区域而言高频区域更难修复；若不在卷积神经网络中引入密集图，对于卷积神经网络而言图像各个区域均为均匀修复；输入密集图后卷积神经网络能够重点对于待处理raw域图像中的高频区域进行重点修复，从而实现区分待处理raw域图像中的纹理密集区域与纹理稀疏区域，实现不同密集区域的修复。
通道修复模块520：用于对raw域图像的四通道图像中的主要通道进行预先修复，从而得到修复后的通道图像，即初步修复后的通道图像。
在一个示例中，若四通道图像为RGGB通道图像或者BGGR通道图像，则通道修复模块520可以用于对两个G通道进行预先修复。
在另一个示例中，若四通道图像为RYYB通道图像，则通道修复模块520可以用于对两个Y通道进行预先修复。
应理解，通过修复模块520会选取四通道中包括信息量较大的通道图像进行预先修复，从而能够为后续引导模块530提供更多的关于待处理raw域图像中的信息，有利于后续引导模块530对待处理raw域图像的图像特征进行进一步的自适应卷积处理。
引导模块530：用于通过通道修复模块520输入的修复后的第一通道图像对主修复模块510输出的图像特征进行进一步的卷积处理，从而得到最终的输出图像即联合去马赛克和去噪处理后的图像。
可选地，联合去马赛克和去噪***500中还可以包括训练特征仓库550，训练特征仓库550可以用于存储样本图像以及样本图像对应的去马赛克和去噪修复的图像特征。
示例性地，在训练阶段可以通过从训练特征仓库550中获取训练联合去马赛克和去噪模型的训练图像。
下面结合图10至图20对本申请实施例提供的图像处理方法进行详细描述。
图10示出了本申请实施例提供的图像处理方法600的示意性流程图，该方法可以应用于图9所示的联合去马赛克和去噪***中；该方法可以由能够进行图像处理装置执行，例如，该方法可以由图8中的执行设备410执行，或者，也可以由本地设备420执行。其中，方法600包括步骤610至步骤630，下面分别对这些步骤进行详细的描述。
步骤610、获取待处理原始raw域图像对应的通道图像，其中，通道图像包括至少两个第一通道图像。
需要说明的是，待处理原始raw图像可以是传感器采集的raw域图像，其中，待处理raw域图像是指未经过图像信号处理器(image signal processor，ISP)处理的原始域图像。
例如，上述待处理raw域图像可以是指图像具有去马赛克和去噪需求的拜耳格式的图像，即图像中可能存在部分额外的信息，可以是噪声或者伪影等。
在一个示例中，待处理raw域图像对应的通道图像可以是指待处理raw域图像对应的四通道图像。
例如，可以是指RGGB四通道图像；或者，BGGR四通道图像；或者，RYYB四通道图像。
示例性地，当获取的待处理raw域图像的通道图像可以为RGGB四通道图像，或者BGGR四通道图像时，上述第一通道图像可以表示绿色G通道图像。
示例性地，当获取的待处理raw域图像的通道图像为RYYB四通道图像时，上述第一通道图像可以表示黄色Y通道图像。
步骤620、对第一通道图像进行图像修复处理，得到修复后的第一通道图像。
其中，上述对第一通道图像进行图像修复处理，得到修复后的第一通道图像可以是指对第一通道图像进行初步的图像修复处理，得到初步修复后的第一通道图像。
需要说明的是，图像修复处理是指恢复图像损失的部分并基于图像信息将它们重建的技术。通过图像修复处理可以试图估计原始图像信息，对破损区域进行修复和改善，从而提高图像的视觉质量。
应理解，上述对第一通道图像进行图像修复处理可以是指对第一通道图像进行卷积处理，其中，第一通道图像可以是指在待处理raw域图像对应的多个通道图像中具有图像信息最多的通道图像；其中，图像信息可以是指图像中的高频信息；比如，可以是指图像中的细节信息、纹理信息以及边缘信息。示例性地，对第一通道图像进行预先修复的具体流程可以参照后续图13所示。
在一种可能的实现方式中，为了提升第一通道图像的修复效果，可以将第一通道图像与随机噪声图像进行通道并联后输入至卷积神经网络中进行卷积处理。
示意性地，上述卷积神经网络可以是指残差密集连接网络(Residual-in-Residual Dense block，RRDB)或者其他网络模型。
步骤630、根据修复后的第一通道图像对待处理raw域图像进行图像修复处理，得到联合去马赛克与去噪处理后的raw域图像。
应理解，通过修复后的第一通道图像即初步修复的第一通道图像可以使得神经网络在对待处理raw域图像进行卷积处理时具有更多的图像信息，从而使得神经网络在对待处理raw域图像进行图像修复时能够最大程度的避免彩色纹理的出现。
进一步，为了使得神经网络在对待处理raw域图像进行卷积处理时，能够区分待处理raw域图像中的不同纹理区域即纹理密集区域与纹理稀疏区域还可以向神经网络中输入密集图。
可选地，上述图像处理方法还包括：获取待处理raw域图像的密集图，其中，密集图可以用于指示待处理raw域图像中不同频率的纹理区域；根据修复后的第一通道图像对待处理raw域图像进行图像修复处理，得到联合去马赛克与去噪处理后的raw域图像，包括：通过对密集图和待处理raw域图像进行卷积处理，得到第一图像特征；通过修复后的第一通道图像引导第一图像特征进行卷积处理，得到联合去马赛克和去噪后的raw域图像。
可选地，上述通过修复后的第一通道图像引导第一图像特征进行卷积处理，得到联合去马赛克和去噪后的图像，包括：根据修复后的第一通道图像的像素空间分布对第一图像特征进行卷积处理，得到联合去马赛克和去噪的raw域图像，其中，像素空间分布用于指示修复后的第一通道图像中不同像素点分布的关联性。
在一个示例中，上述通过修复后的第一通道图像引导第一图像特征进行卷积处理，得到联合去马赛克和去噪后的raw域图像还可以是通过修复后的第一通道图像与待处理raw域图像对应的多通道图像(例如，RGGB通道图像、GBBR通道图像或者RYYB通道图像)进行拼接处理，从而对待处理的raw域图像进行引导，得到联合去马赛克和去噪后的raw域图像。
应理解，上述第一图像特征可以是指对待处理raw域图像进行初步修复得到的图像特征，即第一图像特征中可能还存在部分噪声与伪影现象，并未得到真正的联合去马赛克和去噪后的图像。
需要说明的是，上述密集图可以用于指示待处理raw域图像中不同频率的纹理区域；在密集图中对于高频纹理区域响应较高，对于低频纹理区域响应较低，从而能够根据密集图区分出待处理raw域图像中的不同纹理区域。
示例性地，获取待处理raw域图像的密集图的具体流程可以参照后续图14所示；可以包括以下步骤：
步骤一：获取待处理raw域图像对应的多个通道图像，对多个通道图像的像素进行平均处理得到灰度图像Img。
例如，对RGGB四个通道图像进行平均处理，得到灰度图像。
步骤二：将灰度图像进行经过高斯滤波得到模糊化的灰度图像Img_blur。
步骤三：将灰度图像Img与模糊化的灰度图像Img_blur相减得到残差图Img_minus，其中，残差图像中像素值高区域可以表示纹理频率较高。
步骤四：将残差图Img_minus先进行高斯模糊，再进行归一化处理，从而得到密集图MD。
进一步地，在对密集图和待处理raw域图像进行卷积处理时，还可以向神经网络中输入随机噪声图，即可以对密集图、待处理raw域图像以及随机噪声图进行通道并联后输入至卷积神经网络，从而进一步提升神经网络对待处理raw域图像的图像修复效果。
可选地，所述通过对密集图和待处理raw域图像进行卷积处理，得到第一图像特征，包括：获取随机噪声图；对密集图、随机噪声图以及待处理raw域图像进行卷积处理，得到第一图像特征。
示例性地，上述对待处理raw域图像进行卷积处理时可以通过RRDB网络结构或者其他网络模型。
在一种可能的实现方式中，可以先对待处理raw域图像进行初步的图像修复，即对待处理raw域图像进行卷积处理，得到输出的图像特征，该图像特征并非真正的干净图像可能还存在部分噪声或者伪影等；进一步，可以根据修复后的通道图像对输出的图像特征进行自适应卷积处理从而能够实现完全去除图像特征中的噪声和马赛克得到最终的联合去马赛克和去噪后的raw域图像；raw域图像。其中，自适应卷积处理的具体处理流程可以参见后续图15所示。
需要说明的是，与传统卷积处理的不同空间位置的窗口共享参数相比，自适应卷积可以考虑到第一通道图像上不同像素点空间分布的关联性；此外，由于待处理raw域图像中不同区域的图像信息的差异性，因此，对于待处理raw域图像中的不同区域可以采用不同的卷积窗口，即对于图像中的不同区域卷积窗口中的参数可以不同；从而对联合去马赛克和去噪后的raw域图像在像素空间分布以及图片纹理信息等方面起到引导的作用。
示例性地，可以通过以下公式进行自适应卷积处理：
Figure PCTCN2020112619-appb-000011
其中，其中，o i表示联合去马赛克和去噪后的raw域图像对应的图像特征中位置i的 图像特征，Ω表示卷积窗口；W表示卷积的权重参数；P i,j表示权重W的索引；b表示卷积的偏移量；f i表示修复后的通道图像对应的图像特征中位置i的一个向量；f j表示修复后的通道图像对应的图像特征中位置j的一个向量，所述位置j与所述位置i具有相关性，即位置j可以是指位置i对应的卷积窗口中的的任意一个位置；G表示高斯函数，可以用于求两个向量之间的高斯距离。
本申请实施例提供的图像处理方法，可以通过获取待处理raw域图像对应的通道图像，通道图像中可以包括至少两个第一通道图像；对第一通道图像进行预先的图像修复处理，得到修复后的第一通道图像；根据修复后的第一通道图像可以引导待处理raw域图像进行图像修复处理，得到联合去马赛克与去噪处理后的raw域图像；在本申请实施例中的图像处理方法中可以预先修复待处理raw域图像的通道图像，通过修复后的通道图像引导待处理raw域图像进行图像修复，从而避免彩色纹理的出现，使得处理后图像能够保留图像细节，提升图像质量。
此外，在本申请实施例的图像处理方法中还可以通过输入待处理raw域图像的密集图使得能够引导神经网络在进行卷积处理时区分密集纹理区域和稀疏纹理区域，对不同纹理密集程度的区域进行恢复。
图11是本申请实施例提供的图像处理方法即图像的联合去马赛克和去噪方法的示意性流程图，该联合去马赛克和去噪方法700包括步骤701至步骤705，下面分别对这些步骤进行详细的描述。
步骤701、获取待处理raw域图像。
应理解，摄像头是可以包括传感器和镜头的模组，待处理原始raw图像可以是获取的传感器采集的raw域图像，其中，待处理raw域图像是指未经过图像信号处理器(image signal processor，ISP)处理的原始域图像。
例如，上述待处理raw域图像可以是具有去马赛克和去噪需求的拜耳格式的图像，即raw域图像中可能存在噪声或者伪影等现象。
例如，上述待处理raw域图像也可以是指智能终端中的相机传感器采集到的色彩滤波阵列数据。
步骤702、修复后的通道图像，即初步修复的通道图像。
进一步地，可以对待处理raw域图像(例如，拜耳图像)进行重组得到四通道图像或者其他数量的通道图像。
示例性地，以四通道图像进行举例说明，上述四通道图像可以是指RGGB四通道图像；或者，也可以是指BGGR四通道图像；或者，也可以是指RYYB四通道图像。
例如，对上述四通道图像中的主要通道图像进行预先修复，其中，主要通道可以是指在通道图像中具有图像信息最多的通道图像。
在一个示例中，若四通道图像为RGGB通道图像或者BGGR通道图像，则通道修复模块520可以用于对两个G通道进行预先修复。
在另一个示例中，若四通道图像为RYYB通道图像，则通道修复模块520可以用于对两个Y通道进行预先修复。
上述步骤702可以在图9所示的通道修复模块520中执行。
例如，图13本申请实施例提供的修复通道图像的示意图。如图13所示，可以从多个 通道图像中选择进行预先修复的通道图像，比如，对于RGGB通道图像或者BGGR通道图像，则可以选择两个G通道图像和进行预先修复；对选取的两个G通道图像进行图像修复处理即卷积处理，从而得到修复后的G通道图像。
可选地，可以将选取的两个G通道图像与随机噪声图进行通道并联处理，即尺度相同的两个G通道图与随机噪声图进行特征图合并操作；将通道并联处理后的图像特征输入至卷积神经网络进行卷积处理，接着对预先修复的通道图像特征进行上采样操作，从而得到修复后的通道图像；如图13所示，输出大小为2w*2h*1的预先修复的通道图像，在此过程中可以采用通道图像的真值图像进行监督从而使得通道图像进行初步图像修复。
应理解，通过在对通道图像进行预先修复时引入随机噪声图像可以提升通道图像的去噪效果。
示例性地，图13中所示的卷积神经网络可以是指残差密集连接网络(Residual-in-Residual Dense block，RRDB)或者其他网络模型。
步骤703、第一图像特征，即对待处理raw域图像进行初步修复后的图像特征。
应理解，上述初步修复图像可以是指对待处理raw域图像的各个通道图像均进行初步的图像修复处理，得到输出的第一图像特征，但是该第一图像特征中可能还存在部分噪声与伪影现象，并未得到真正的干净图像。
需要说明的是，上述步骤702与步骤703可以是先执行步骤702再执行步骤703；或者，也可以是先执行步骤703再执行步骤702，本申请对步骤702与步骤703的执行顺序不作任何限定。
例如，图12本申请实施例提供的初步修复图像的示意图。如图12所示，输入数据可以是指通过相机传感器采集的大小为2w*2h的待处理raw域图像(例如，拜耳图像)，通过对raw域图像进行重采样得到通道图像；比如，对拜耳图像按照2*2进行分组，然后分别从每个2*2的小块中从固定的位置(左上，右上，左下，右下)采样像素，生成四个通道图像(例如，RGGB、BGGR、RYYB等)；将四通道的RGGB图像输入卷积神经网络进行卷积处理，从而得到raw域图像的初步图像修复后的图像特征。
可选地，为了提升卷积神经网络对待处理raw域图像的去噪效果，上述初步图像修复图像的过程中还可以向卷积神经网络中输入随机噪声图。
进一步地，为了使得卷积神经网络在进行初步图像修复时能够区分待处理raw域图像中的不同纹理区域，即纹理密集区域与纹理稀疏区域还可以向卷积神经网络中输入密集图，其中，密集图可以用于指示待处理raw域图像中不同频率的纹理区域；在密集图中对于高频纹理区域响应较高，对于低频纹理区域响应较低，从而能够根据密集图区分出待处理raw域图像中的不同纹理区域。
需要说明的是，通过密集图可以使得卷积神经网络更好的识别图像中的高频区域与低频区域，通常图像中的高频区域是指图像中的边缘以及细节区域，相对于低频区域而言高频区域更难修复；若不在卷积神经网络中引入密集图，对于卷积神经网络而言图像各个区域均为均匀修复；输入密集图后卷积神经网络能够重点对于待处理raw域图像中的高频区域进行重点修复，从而实现区分待处理raw域图像中的纹理密集区域与纹理稀疏区域，实现不同纹理区域的图像修复。
例如，图14是本申请实施例提供的获取密集图的示意图。获取密集图的过程可以包 括以下步骤：
步骤一：获取待处理raw域图像对应的多个通道图像，对多个通道图像的像素进行平均处理得到灰度图像Img。
例如，对RGGB四个通道图像进行平均处理，得到灰度图像。
步骤二：将灰度图像进行经过高斯滤波得到模糊化的灰度图像Img_blur。
步骤三：将灰度图像Img与模糊化的灰度图像Img_blur相减得到残差图Img_minus，其中，残差图像中像素值高区域可以表示纹理频率较高。
步骤四：将残差图Img_minus先进行高斯模糊，再进行归一化处理，从而得到密集图M D。
步骤704、自适应卷积处理。
通过将步骤702得到修复后的通道图像即预先修复的通道图像作为引导图，将通过步骤703得到的图像特征作为被引导图，进行自适应卷积处理。
应理解，与传统卷积处理在不同空间位置的窗口共享参数相比，自适应卷积可以考虑到第一通道图像上不同像素点空间分布的关联性；此外，由于待处理raw域图像中不同区域的图像信息的差异性，因此，对于待处理raw域图像中的不同区域可以采用不同的卷积窗口，即对于图像中的不同区域卷积窗口中的参数可以不同；从而对联合去马赛克和去噪后的raw域图像在像素空间分布以及图片纹理信息等方面起到引导的作用。
还应理解，通过可以上述预先修复的通道图像的像素空间分布对初步修复的图像特征引导，使得初步修复的图像特征进行进一步的卷积处理，即使得初步修复的图像特征也能够在一定程度上满足与预先修复的通道图像中相同或者近似的像素空间分布，其中，像素空间分布可以用于指示不同像素点分布的关联性。
示例性地，可以通过以下公式进行自适应卷积处理：
Figure PCTCN2020112619-appb-000012
其中，其中，o i表示联合去马赛克和去噪后的raw域图像对应的图像特征中位置i的图像特征，Ω表示卷积窗口；W表示卷积的权重参数；P i,j表示权重W的索引；b表示卷积的偏移量；f i表示修复后的通道图像对应的图像特征中位置i的一个向量；f j表示修复后的通道图像对应的图像特征中位置j的一个向量，所述位置j与所述位置i具有相关性，即位置j可以是指位置i对应的卷积窗口中的的任意一个位置；G表示高斯函数，可以用于求两个向量之间的高斯距离。
示例性地，图15是本申请实施例提供的自适应卷积处理的示意图。如图15所示，首先可以对第一通道图像(例如，预先修复的G通道图像或者预先修复的Y通道图像)经过两层卷积进行升维处理，得到8通道的特征图；然后根据上述自适应卷积处理公式对初步修复的图像特征进行进一步的卷积处理，最终得到联合去马赛克和去噪后的raw域图像。
示例性地，在图15中卷积窗口的大小可以是指3*3；f i可以表示图像特征中的任意一个位置的向量；比如，f i表示卷积窗口中f 2,2位置处的向量时，f j可以表示卷积窗口中除f 2,2位置之外的任意一个向量，即对于一个f i可能同时存在多个相关的f j。
步骤705、输出联合去马赛克和去噪后的raw域图像。
可选地，可以进一步对联合去马赛克和去噪后的raw域图像进行后续的图像处理，从而得到标准的全彩图像即SRGB图像，其中，后续的图像处理可以是指对联合去马赛克和去噪后的raw域图像进行白平衡、颜色校正、色调映射等图像处理。
下面结合附图16对本申请实施例的图像处理方法进行详细的描述。该图像处理方法可以应用于具有显示屏和摄像头的电子设备。该电子设备具体可以是移动终端(例如，智能手机)，电脑，个人数字助理，可穿戴设备，车载设备，物联网设备或者其他能够进行图像显示的设备。
图16所示的图像处理方法800包括步骤810和820，下面分别对这些步骤进行详细的描述。
步骤810、检测到用户指示相机的操作。
例如，可以是检测到用户指示第一处理模式的操作；其中，第一处理模式可以是专业拍摄模式，或者，第一处理模式还可以是指AI ISP处理模式，即采用上述图10所示的图像处理方法对摄像头采集到的待处理raw域图像进行图像修复处理，或者，第一处理模式还可以是联合去马赛克和去噪的拍摄模式，从而使得获取图像质量较高的图像。
在一个示例中，参见图18中的(a)，拍摄界面上包括拍摄选项960，在电子设备检测到用户点击拍摄选项960后，参见图18中的(b)，电子设备显示拍摄模式界面。在电子设备检测到用户点击拍摄模式界面上用于指示专业拍摄模式961后，手机进入专业拍摄模式。
例如，可以是检测到用户用于指示拍摄的操作，该操作可以作为指示对获取的待处理raw域图像进行图像修复的操作。参见图19，可以检测到用户用于指示拍摄的操作970。
应理解，用户用于指示拍摄行为的操作可以包括按下电子设备的相机中的拍摄按钮，也可以包括用户设备通过语音指示电子设备进行拍摄行为，或者，也可以是指用户通过快捷键指示电子设备进行拍摄行为，还可以包括用户其它的指示电子设备进行拍摄行为。上述为举例说明，并不对本申请作任何限定。
在一种可能的实现方式中，上述检测到所述用户指示相机的操作可以是指用户通过快捷键直接进行拍摄的操作。
在一种可能的实现方式中，在步骤810之前还包括：检测到检测到用户用于打开相机的第一操作；响应于所述第一操作，在所述显示屏上显示拍摄界面。
在一个示例中，用户的拍摄行为可以包括用户打开相机的第一操作；响应于所述第一操作，在显示屏上显示拍摄界面。
如图17中的(a)示出了手机的一种图形用户界面(graphical user interface，GUI)，该GUI为手机的桌面910。当电子设备检测到用户点击桌面910上的相机应用(application，APP)的图标920的操作后，可以启动相机应用，显示如图17中的(b)所示的另一GUI，该GUI可以称为拍摄界面930。
示例性地，该拍摄界面930上可以包括取景框940；在预览状态下，该取景框940内可以实时显示预览图像。
示例性的，参见图17中的(b)，电子设备在启动相机后，取景框940内可以显示有预览图像，该预览图像为彩色图像；拍摄界面上还可以包括用于指示拍照模式的控件950，以及其它拍摄控件。需要注意的是，在本申请实施例中，为了区别于灰度图像部分，彩色 图像部分通过斜线填充来表示。
在一个示例中，用户的拍摄行为可以包括用户打开相机的第一操作；响应于所述第一操作，在显示屏上显示拍摄界面。
例如，电子设备可以检测到用户点击桌面上的相机应用(application，APP)的图标的第一操作后，可以启动相机应用，显示拍摄界面。在拍摄界面上可以包括取景框，可以理解的是，在拍照模式和录像模式下，取景框的大小可以不同。例如，取景框可以为拍照模式下的取景框。在录像模式下，取景框可以为整个显示屏。在预览状态下即可以是用户打开相机且未按下拍照/录像按钮之前，该取景框内可以实时显示预览图像。
步骤820、响应于所述操作，在所述显示屏中显示输出图像，其中，其中，所述输出图像是根据联合去马赛克和去噪后的原始raw域图像得到的，所述联合去马赛克和去噪后的raw域图像是针对所述摄像头采集到的待处理raw域图像进行图像修复处理后得到的raw域图像，修复后的第一通道图像用于所述待处理raw域图像的图像修复处理过程中，所述待处理raw域图像包括至少两个第一通道图像，所述修复后的第一通道图像是指对所述第一通道图像进行图像修复处理后的通道图像。
例如，输出图像可以是指对联合去马赛克和去噪后的raw域图像进行后续的图像处理得到的标准全彩图像，其中，后续的图像处理可以包括但不限于：对联合去马赛克和去噪后的raw域图像进行白平衡、颜色校正、色调映射等图像处理。例如，参见图20。
需要说明的是，上述对第一通道图像进行卷积处理可以是指对第一通道图像进行预先修复，其中，第一通道图像可以是指在待处理raw域图像对应的多个通道图像中具有图像信息最多的通道图像；其中，图像信息可以是指图像中的高频信息；比如，可以是指图像中的细节信息、纹理信息以及边缘信息。示例性地，对第一通道图像进行预先修复的具体流程可以参见图13所示，此处不再赘述。
例如，待处理raw域图像对应的通道图像可以是指RGGB四通道图像；或者，BGGR四通道图像；或者，RYYB四通道图像。
示例性地，当待处理raw域图像对应的通道图像为RGGB四通道图像，或者BGGR四通道图像时，上述第一通道图像可以表示绿色G通道图像。
示例性地，当待处理raw域图像对应的通道图像为RYYB四通道图像时，上述第一通道图像可以表示黄色Y通道图像。
应理解，通过修复后的第一通道图像，即预先修复的通道图像可以使得神经网络在进行卷积处理时具有更多的图像信息，从而使得神经网络在对待处理raw域图像进行修复时能够最大程度的避免彩色纹理的出现。
进一步，为了使得神经网络在对待处理raw域图像进行图像修复处理时，能够区分待处理raw域图像中的不同纹理区域，即纹理密集区域与纹理稀疏区域还可以向神经网络中输入密集图。
可选地，所述图像修复处理过程还包括：获取所述待处理raw域图像的密集图，其中，所述密集图用于指示所述待处理raw域图像中不同频率的纹理区域；通过对根据所述密集图和所述待处理raw域图像进行卷积处理，得到第一图像特征；通过所述修复后的第一通道图像引导所述第一图像特征进行卷积处理，得到所述联合去马赛克和去噪后的raw域图像。
需要说明的是，上述第一图像特征可以是指对待处理raw域图像进行初步修复得到的图像特征，即第一图像特征中可能还存在部分噪声与伪影现象，并未得到真正的联合去马赛克和去噪后的raw域图像。
在一个示例中，可以先对待处理raw域图像进行初步的图像修复，即对待处理raw域图像进行卷积操作，得到输出的图像特征，该图像特征并非真正的干净图像可能还存在部分噪声或者伪影等；进一步，可以根据修复后的第一通道图像对输出的图像特征进行自适应卷积处理从而能够实现完全去除图像中的噪声和马赛克得到最终的联合去马赛克和去噪后的图像；其中，与传统卷积操作在不同空间位置的窗口共享参数相比，自适应卷积考虑到所示第一通道图像上不同像素点空间分布的关联性；此外，由于raw域图像中不同区域的图像信息的差异性，因此，对于图像中的不同区域可以采用不同的卷积窗口，即对于图像中的不同区域卷积窗口中的参数可以不同；从而对联合去马赛克和去噪后的raw域图像在像素空间分布以及图片纹理信息等方面起到引导的作用。其中，自适应卷积处理的具体处理流程可以参见图15所示，此处不再赘述。
可选地，通过所述修复后的第一通道图像引导第一图像特征进行卷积处理，得到联合去马赛克和去噪后的raw域图像，包括：根据修复后的第一通道图像的像素空间分布对第一图像特征进行卷积处理，得到联合去马赛克和去噪后的raw域图像，其中，像素空间分布用于指示修复后的第一通道图像中不同像素点分布的关联性。
应理解，与传统卷积操作的不同空间位置的窗口共享参数相比，自适应卷积可以考虑到第一通道图像上不同像素点空间分布的关联性；此外，由于待处理raw域图像中不同区域的图像信息的差异性，因此，对于待处理raw域图像中的不同区域可以采用不同的卷积窗口，即对于图像中的不同区域卷积窗口中的参数可以不同；从而对联合去马赛克和去噪后的raw域图像在像素空间分布以及图片纹理信息等方面起到引导的作用。
还应理解，通过可以上述预先修复的通道图像的像素空间分布对初步修复的图像特征引导，使得初步修复的图像特征进行进一步的卷积处理，即使得初步修复的图像特征也能够在一定程度上满足与预先修复的通道图像中相同或者近似的像素空间分布，其中，像素空间分布可以用于指示不同像素点分布的关联性。
示例性地，可以通过以下公式进行自适应卷积处理：
Figure PCTCN2020112619-appb-000013
其中，其中，o i表示联合去马赛克和去噪后的raw域图像对应的图像特征中位置i的图像特征，Ω表示卷积窗口；W表示卷积的权重参数；P i,j表示权重W的索引；b表示卷积的偏移量；f i表示修复后的通道图像对应的图像特征中位置i的一个向量；f j表示修复后的通道图像对应的图像特征中位置j的一个向量，所述位置j与所述位置i具有相关性，即位置j可以是指位置i对应的卷积窗口中的的任意一个位置；G表示高斯函数，可以用于求两个向量之间的高斯距离。
可选地，通过对密集图和待处理raw域图像进行卷积处理，得到第一图像特征，包括：获取随机噪声图；对密集图、随机噪声图以及待处理raw域图像进行卷积处理，得到第一图像特征。
应理解，上述图8至图15所示的图像处理方法同样适用于图16所示的图像处理方法， 即上述图8至图15中相关内容的扩展、限定、解释和说明也适用于图16中相同的内容，此处不再赘述。
本申请实施例提供的图像处理方法，可以通过获取待处理raw域图像对应的通道图像，通道图像中可以包括至少两个第一通道图像；对第一通道图像进行预先的图像修复处理，得到修复后的第一通道图像；根据修复后的第一通道图像可以引导待处理raw域图像进行图像修复处理，得到联合去马赛克与去噪处理后的raw域图像；在本申请实施例中的图像处理方法中可以预先修复待处理raw域图像的通道图像，得到修复后的通道图像即初步修复的通道图像；基于初步修复的通道图像引导待处理raw域图像进行图像修复处理，从而避免彩色纹理的出现，使得处理后图像能够保留图像细节，提升图像质量。
此外，在本申请实施例的图像处理方法中还可以通过输入待处理raw域图像的密集图使得能够引导神经网络在进行卷积处理时区分密集纹理区域和稀疏纹理区域，对不同纹理密集程度的区域进行恢复。
表1是本申请实施例提供的在不同数据集中不同的模型进行联合去马赛克和去噪的测试结果。其中，进行测试时的噪声选取标准差为10的高斯噪声，测试结果通过峰值信噪比(Peak signal ro noise ratio，PSNR)，学习感知图像块相似度(Learned perceptual image patch similarity，LPIPS)和结构相似性(Structural similarity index，SSIM)；其中，测试的算法包括本申请实施例提供的图像处理方法、灵活图像信号处理算法(Flexible image signal processing，FlexISP)、连续能量最小化算法(Sequential energy minimization，SEM)、乘法器交替方向算法(Alternating direction method ofmultipliers，ADMM)、深度联合去马赛克和去噪算法(Deep joint demosaicking and denoising，DeepJoint)以及Kokkinos算法。通过表1所示的测试性能指标可以看出，本申请实施例提出的图像处理方法即联合去马赛克和去噪的方法在两个测试数据集中取得较好的效果。
表1
Figure PCTCN2020112619-appb-000014
应理解，上述举例说明是为了帮助本领域技术人员理解本申请实施例，而非要将本申请实施例限于所例示的具体数值或具体场景。本领域技术人员根据所给出的上述举例说明，显然可以进行各种等价的修改或变化，这样的修改或变化也落入本申请实施例的范围内。
上文结合图1至图20，详细描述了本申请实施例图像分类方法，下面将结合图21和 图23，详细描述本申请的装置实施例。应理解，本申请实施例中的图像处理装置可以执行前述本申请实施例的各种图像分处理类方法，即以下各种产品的具体工作过程，可以参考前述方法实施例中的对应过程。
图21是本申请实施例提供的图像处理装置的示意性框图。应理解，图像处理装置1000可以执行图10所示的图像处理方法。该图像处理装置1000包括：获取单元1010和处理单元1020。
其中，所述获取单元1010用于获取待处理原始raw域图像对应的通道图像，其中，所述通道图像包括至少两个第一通道图像；所述处理单元1020用于对所述第一通道图像进行图像修复处理，得到修复后的第一通道图像；根据所述修复后的第一通道图像对所述raw域图像进行图像修复处理，得到联合去马赛克与去噪处理后的raw域图像。
可选地，作为一个实施例，所述获取单元1010还用于：
获取所述待处理raw域图像的密集图，其中，所述密集图用于指示所述待处理raw域图像中不同频率的纹理区域；
所述处理单元1020具体用于：
通过对所述密集图和所述待处理raw域图像进行卷积处理，得到第一图像特征；
通过所述修复后的第一通道图像引导所述第一图像特征进行卷积处理，得到所述联合去马赛克和去噪后的raw域图像。
可选地，作为一个实施例，所述处理单元1020具体用于：
根据所述修复后的第一通道图像的像素空间分布对所述第一图像特征进行卷积处理，得到所述联合去马赛克和去噪的raw域图像，其中，所述像素空间分布用于指示所述修复后的第一通道图像中不同像素点分布的关联性。
可选地，作为一个实施例，所述获取单元1010还用于：
获取随机噪声图；
所述处理单元1020具体用于：
对所述密集图、所述随机噪声图以及所述待处理raw域图像进行卷积处理，得到所述第一图像特特征。
可选地，作为一个实施例，所述第一通道图像为绿色G通道图像。
可选地，作为一个实施例，所述第一通道图像为黄色Y通道图像。
图22是本申请实施例提供的图像处理装置的示意性框图。应理解，图像处理装置1100可以执行图16所示的图像处理方法；该图像处理装置1100包括：检测单元1110和处理单元1120。
其中，所述检测单元1110用于检测到用户指示相机的操作；所述处理单元1120用于响应于所述操作，在所述显示屏内显示输出图像，其中，所述输出图像是根据联合去马赛克和去噪后的原始raw域图像得到的，所述联合去马赛克和去噪后的raw域图像是针对所述摄像头采集到的待处理raw域图像进行图像修复处理后得到的raw域图像，修复后的第一通道图像用于所述待处理raw域图像的图像修复处理过程中，所述待处理raw域图像包括至少两个第一通道图像，所述修复后的第一通道图像是指对所述第一通道图像进行图像修复处理后的通道图像。
示意性地，上述图像处理装置1100可以是具有显示屏和摄像头的电子设备。
可选地，作为一个实施例，所述检测单元1110具体用于：
检测到所述用户指示第一处理模式的操作，所述第一处理模式用于对所述待处理raw域图像进行图像修复处理；或者，
检测到所述用户用于指示拍摄的操作。
可选地，作为一个实施例，所述图像处理装置还包括获取单元，所述获取单元用于：
获取所述待处理raw域图像的密集图，其中，所述密集图用于指示所述待处理raw域图像中不同频率的纹理区域；
所述处理单元1120具体用于：
通过对所述密集图和所述待处理raw域图像进行卷积处理，得到第一图像特征；
通过所述修复后的第一通道图像引导所述第一图像特征进行卷积处理，得到所述联合去马赛克和去噪后的raw域图像。
可选地，作为一个实施例，所述处理单元1120具体用于：
根据所述修复后的第一通道图像的像素空间分布对所述第一图像特征进行卷积处理，得到所述联合去马赛克和去噪的raw域图像，其中，所述像素空间分布用于指示所述修复后的第一通道图像中不同像素点分布的关联性。
可选地，作为一个实施例，所述获取单元还用于：
获取随机噪声图；
所述处理单元1120具体用于：
对所述密集图、所述随机噪声图以及所述待处理raw域图像进行卷积处理，得到所述第一图像特征。
可选地，作为一个实施例，所述第一通道图像为绿色G通道图像。
可选地，作为一个实施例，所述第一通道图像为黄色Y通道图像。
需要说明的是，上述图像处理装置1000与图像处理装置1100以功能单元的形式体现。这里的术语“单元”可以通过软件和/或硬件形式实现，对此不作具体限定。
例如，“单元”可以是实现上述功能的软件程序、硬件电路或二者结合。所述硬件电路可能包括应用特有集成电路(application specific integrated circuit，ASIC)、电子电路、用于执行一个或多个软件或固件程序的处理器(例如共享处理器、专有处理器或组处理器等)和存储器、合并逻辑电路和/或其它支持所描述的功能的合适组件。
因此，在本申请的实施例中描述的各示例的单元，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
图23是本申请实施例提供的图像处理装置的硬件结构示意图。如图23所示的图像处理装置1200(该装置1200具体可以是一种计算机设备)包括存储器1201、处理器1202、通信接口1203以及总线1204。其中，存储器1201、处理器1202、通信接口1203通过总线1204实现彼此之间的通信连接。
存储器1201可以是只读存储器(read only memory，ROM)，静态存储设备，动态存储设备或者随机存取存储器(random access memory，RAM)。存储器1201可以存储程序，当存储器1201中存储的程序被处理器1202执行时，处理器1202用于执行本申请实施例 的图像处理方法的各个步骤，例如，执行图10至图20所示的各个步骤。
应理解，本申请实施例所示的图像处理装置可以是服务器，例如，可以是云端的服务器，或者，也可以是配置于云端的服务器中的芯片；或者，本申请实施例所示的图像处理装置可以是智能终端，也可以是配置于智能终端中的芯片。
上述本申请实施例揭示的图像处理方法可以应用于处理器1202中，或者由处理器1202实现。处理器1202可能是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述图像处理方法的各步骤可以通过处理器1202中的硬件的集成逻辑电路或者软件形式的指令完成。例如，处理器1202可以是包含图7所示的NPU的芯片。
上述的处理器1202可以是中央处理器(central processing unit，CPU)、图形处理器(graphics processing unit，GPU)、通用处理器、数字信号处理器(digital signal processor，DSP)、专用集成电路(application specific integrated circuit，ASIC)、现成可编程门阵列(field programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本申请实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存取存储器(random access memory，RAM)、闪存、只读存储器(read-only memory，ROM)、可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器1201，处理器1202读取存储器1201中的指令，结合其硬件完成本申请实施中图21或图22所示的图像处理装置中包括的单元所需执行的功能，或者，执行本申请方法实施例的图10至图20所示的图像处理方法的各个步骤。
通信接口1203使用例如但不限于收发器一类的收发装置，来实现装置1200与其他设备或通信网络之间的通信。
总线1204可包括在图像处理装置1200各个部件(例如，存储器1201、处理器1202、通信接口1203)之间传送信息的通路。
应注意，尽管上述图像处理装置1200仅仅示出了存储器、处理器、通信接口，但是在具体实现过程中，本领域的技术人员应当理解，图像处理装置1200还可以包括实现正常运行所必须的其他器件。同时，根据具体需要本领域的技术人员应当理解，上述图像处理装置1200还可包括实现其他附加功能的硬件器件。此外，本领域的技术人员应当理解，上述图像处理装置1200也可仅仅包括实现本申请实施例所必须的器件，而不必包括图23中所示的全部器件。
本申请实施例还提供一种芯片，该芯片包括收发单元和处理单元。其中，收发单元可以是输入输出电路、通信接口；处理单元为该芯片上集成的处理器或者微处理器或者集成电路。该芯片可以执行上述方法实施例中的图像处理方法。
本申请实施例还提供一种计算机可读存储介质，其上存储有指令，该指令被执行时执行上述方法实施例中的图像处理方法。
本申请实施例还提供一种包含指令的计算机程序产品，该指令被执行时执行上述方法实施例中的图像处理方法。
还应理解，本申请实施例中，该存储器可以包括只读存储器和随机存取存储器，并向 处理器提供指令和数据。处理器的一部分还可以包括非易失性随机存取存储器。例如，处理器还可以存储设备类型的信息。
还应理解，本申请实施例中，该存储器可以包括只读存储器和随机存取存储器，并向处理器提供指令和数据。处理器的一部分还可以包括非易失性随机存取存储器。例如，处理器还可以存储设备类型的信息。
应理解，本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。
应理解，在本申请的各种实施例中，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请实施例的实施过程构成任何限定。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的***、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的***、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个***，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能模块可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(read-only memory，ROM)、随机存取存储器(random access memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖 在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (29)

  1. 一种图像处理方法，其特征在于，包括：
    获取待处理原始raw域图像对应的通道图像，其中，所述通道图像包括至少两个第一通道图像；
    对所述第一通道图像进行图像修复处理，得到修复后的第一通道图像；
    根据所述修复后的第一通道图像对所述待处理raw域图像进行图像修复处理，得到联合去马赛克和去噪处理后的raw域图像。
  2. 如权利要求1所述的图像处理方法，其特征在于，还包括：
    获取所述待处理raw域图像的密集图，其中，所述密集图用于指示所述待处理raw域图像中不同频率的纹理区域；
    所述根据所述修复后的第一通道图像对所述待处理raw域图像进行图像修复处理，得到联合去马赛克和去噪处理后的raw域图像，包括：
    通过对所述密集图和所述待处理raw域图像进行卷积处理，得到第一图像特征；
    通过所述修复后的第一通道图像引导所述第一图像特征进行卷积处理，得到所述联合去马赛克和去噪后的raw域图像。
  3. 如权利要求2所述的图像处理方法，其特征在于，所述通过所述修复后的第一通道图像引导所述第一图像特征进行卷积处理，得到所述联合去马赛克和去噪后的raw域图像，包括：
    根据所述修复后的第一通道图像的像素空间分布对所述第一图像特征进行卷积处理，得到所述联合去马赛克和去噪后的raw域图像，其中，所述像素空间分布用于指示所述修复后的第一通道图像中不同像素点分布的关联性。
  4. 如权利要求2或3所述的图像处理方法，其特征在于，还包括：
    获取随机噪声图；
    所述通过对所述密集图和所述待处理raw域图像进行卷积处理，得到第一图像特征，包括：
    对所述密集图、所述随机噪声图以及所述待处理raw域图像进行卷积处理，得到所述第一图像特征。
  5. 如权利要求1至4中任一项所述的图像处理方法，其特征在于，所述第一通道图像为绿色G通道图像。
  6. 如权利要求1至4中任一项所述的图像处理方法，其特征在于，所述第一通道图像为黄色Y通道图像。
  7. 一种图像处理方法，应用于具有显示屏和摄像头的电子设备，其特征在于，包括：
    检测到用户指示相机的操作；
    响应于所述操作，在所述显示屏内显示输出图像，其中，所述输出图像是根据联合去马赛克和去噪后的原始raw域图像得到的，所述联合去马赛克和去噪后的raw域图像是针对所述摄像头采集到的待处理raw域图像进行图像修复处理后得到的raw域图像，修复后的第一通道图像用于所述待处理raw域图像的图像修复处理过程中，所述待处理raw域图 像包括至少两个第一通道图像，所述修复后的第一通道图像是指对所述第一通道图像进行图像修复处理后的通道图像。
  8. 如权利要求7所述的图像处理方法，其特征在于，所述检测到用户指示相机的操作，包括：
    检测到所述用户指示第一处理模式的操作，所述第一处理模式用于指示对所述待处理raw域图像进行图像修复处理；或者，
    检测到所述用户用于指示拍摄的操作。
  9. 如权利要求7或8所述的图像处理方法，其特征在于，所述对待处理raw域图像进行图像修复处理的过程还包括：
    获取所述待处理raw域图像的密集图，其中，所述密集图用于指示所述待处理raw域图像中不同频率的纹理区域；
    通过对所述密集图和所述待处理raw域图像进行卷积处理，得到第一图像特征；
    通过所述修复后的第一通道图像引导所述第一图像特征进行卷积处理，得到所述联合去马赛克和去噪后的raw域图像。
  10. 如权利要求9所述的图像处理方法，其特征在于，所述通过所述修复后的第一通道图像引导所述第一图像特征进行卷积处理，得到所述联合去马赛克和去噪后的raw域图像，包括：
    根据所述修复后的第一通道图像的像素空间分布对所述第一图像特征进行卷积处理，得到所述联合去马赛克和去噪后的raw域图像，其中，所述像素空间分布用于指示所述修复后的第一通道图像中不同像素点分布的关联性。
  11. 如权利要求9或10所述的图像处理方法，其特征在于，还包括：
    获取随机噪声图；
    所述通过对所述密集图和所述待处理raw域图像进行卷积处理，得到第一图像特征，包括：
    对所述密集图、所述随机噪声图以及所述待处理raw域图像进行卷积处理，得到所述第一图像特征。
  12. 如权利要求7至11中任一项所述的图像处理方法，其特征在于，所述第一通道图像为绿色G通道图像。
  13. 如权利要求7至11中任一项所述的图像处理方法，其特征在于，所述第一通道图像为黄色Y通道图像。
  14. 一种图像处理装置，其特征在于，包括：
    获取单元，用于获取待处理原始raw域图像对应的通道图像，其中，所述通道图像包括至少两个第一通道图像；
    处理单元，用于对所述第一通道图像进行图像修复处理，得到修复后的第一通道图像；根据所述修复后的第一通道图像对所述待处理raw域图像进行图像修复处理，得到联合去马赛克和去噪处理后的raw域图像。
  15. 如权利要求14所述的图像处理装置，其特征在于，所述获取单元还用于：
    获取所述待处理raw域图像的密集图，其中，所述密集图用于指示所述待处理raw域图像中不同频率的纹理区域；
    所述处理单元具体用于：
    通过对所述密集图和所述待处理raw域图像进行卷积处理，得到第一图像特征；
    通过所述修复后的第一通道图像引导所述第一图像特征进行卷积处理，得到所述联合去马赛克和去噪后的raw域图像。
  16. 如权利要求15所述的图像处理装置，其特征在于，所述处理单元具体用于：
    根据所述修复后的第一通道图像的像素空间分布对所述第一图像特征进行卷积处理，得到所述联合去马赛克和去噪后的raw域图像，其中，所述像素空间分布用于指示所述修复后的第一通道图像中不同像素点分布的关联性。
  17. 如权利要求15或16所述的图像处理装置，其特征在于，所述获取单元还用于：
    获取随机噪声图；
    所述处理单元具体用于：
    对所述密集图、所述随机噪声图以及所述待处理raw域图像进行卷积处理，得到所述第一图像特征。
  18. 如权利要求14至17中任一项所述的图像处理装置，其特征在于，所述第一通道图像为绿色G通道图像。
  19. 如权利要求14至17中任一项所述的图像处理装置，其特征在于，所述第一通道图像为黄色Y通道图像。
  20. 一种图像处理装置，其特征在于，包括：
    检测单元，用于检测到用户指示相机的操作；
    处理单元，用于响应于所述操作，在所述图像处理装置的显示屏内显示输出图像，其中，所述输出图像是根据联合去马赛克和去噪后的原始raw域图像得到的，所述联合去马赛克和去噪后的raw域图像是针对所述图像处理装置的摄像头采集到的待处理raw域图像进行图像修复处理后得到的raw域图像，修复后的第一通道图像用于对所述待处理raw域图像进行图像修复处理的过程中，所述待处理raw域图像包括至少两个第一通道图像，所述修复后的第一通道图像是指对所述第一通道图像进行图像修复处理后的通道图像。
  21. 如权利要求20所述的图像处理装置，其特征在于，所述检测单元具体用于：
    检测到所述用户指示第一处理模式的操作，所述第一处理模式用于对所述待处理raw域图像进行图像修复处理；或者，
    检测到所述用户用于指示拍摄的操作。
  22. 如权利要求20或21所述的图像处理装置，其特征在于，还包括获取单元，所述获取单元用于：
    获取所述待处理raw域图像的密集图，其中，所述密集图用于指示所述待处理raw域图像中不同频率的纹理区域；
    所述处理单元具体用于：
    通过对所述密集图和所述待处理raw域图像进行卷积处理，得到第一图像特征；
    通过所述修复后的第一通道图像引导所述第一图像特征进行卷积处理，得到所述联合去马赛克和去噪后的raw域图像。
  23. 如权利要求22所述的图像处理装置，其特征在于，所述处理单元具体用于：
    根据所述修复后的第一通道图像的像素空间分布对所述第一图像特征进行卷积处理， 得到所述联合去马赛克和去噪后的raw域图像，其中，所述像素空间分布用于指示所述修复后的第一通道图像中不同像素点分布的关联性。
  24. 如权利要求22或23所述的图像处理装置，其特征在于，所述获取单元还用于：
    获取随机噪声图；
    所述处理单元具体用于：
    对所述密集图、所述随机噪声图以及所述待处理raw域图像进行卷积处理，得到所述第一图像特征。
  25. 如权利要求20至24中任一项所述的图像处理装置，其特征在于，所述第一通道图像为绿色G通道图像。
  26. 如权利要求20至24中任一项所述的图像处理装置，其特征在于，所述第一通道图像为黄色Y通道图像。
  27. 一种图像处理装置，其特征在于，包括：
    存储器，用于存储程序；
    处理器，用于执行所述存储器存储的程序，当所述处理器执行所述存储器存储的程序时，所述处理器用于执行权利要求1至6，或者7至13中任一项所述的图像处理方法。
  28. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质中存储有程序指令，当所述程序指令由处理器运行时，实现权利要求1至6，或者7至13中任一项所述的图像处理方法。
  29. 一种芯片，其特征在于，所述芯片包括处理器与数据接口，所述处理器通过所述数据接口读取存储器上存储的指令，以执行如权利要求1至6，或者7至13中任一项所述的图像处理方法。
PCT/CN2020/112619 2020-02-21 2020-08-31 图像处理方法以及图像处理装置 WO2021164234A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP20920655.6A EP4109392A4 (en) 2020-02-21 2020-08-31 IMAGE PROCESSING METHOD AND IMAGE PROCESSING DEVICE

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010107896.9A CN111402146B (zh) 2020-02-21 2020-02-21 图像处理方法以及图像处理装置
CN202010107896.9 2020-02-21

Publications (1)

Publication Number Publication Date
WO2021164234A1 true WO2021164234A1 (zh) 2021-08-26

Family

ID=71430435

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/112619 WO2021164234A1 (zh) 2020-02-21 2020-08-31 图像处理方法以及图像处理装置

Country Status (3)

Country Link
EP (1) EP4109392A4 (zh)
CN (1) CN111402146B (zh)
WO (1) WO2021164234A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114331916A (zh) * 2022-03-07 2022-04-12 荣耀终端有限公司 图像处理方法及电子设备
CN114596624A (zh) * 2022-04-20 2022-06-07 深圳市海清视讯科技有限公司 人眼状态检测方法、装置、电子设备及存储介质

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111402146B (zh) * 2020-02-21 2022-05-10 华为技术有限公司 图像处理方法以及图像处理装置
CN114514552A (zh) * 2020-09-16 2022-05-17 华为技术有限公司 电子装置和电子装置的图像处理方法
CN112102200B (zh) * 2020-09-21 2024-05-07 腾讯科技（深圳）有限公司 图像补全模型初始化方法、训练方法和图像补全方法
CN114338958B (zh) * 2020-09-30 2023-07-11 华为技术有限公司 一种图像处理的方法及相关设备
CN112862717B (zh) * 2021-02-10 2022-09-20 山东英信计算机技术有限公司 一种图像的去噪模糊化方法、***及介质
CN112967195B (zh) * 2021-03-04 2024-04-23 浙江大华技术股份有限公司 一种图像去噪方法、设备及计算机可读存储介质
CN113658060A (zh) * 2021-07-27 2021-11-16 中科方寸知微(南京)科技有限公司 基于分布学习的联合去噪去马赛克方法及***
CN115767287B (zh) * 2021-09-03 2023-10-27 荣耀终端有限公司 图像处理方法与电子设备
CN113808043A (zh) * 2021-09-16 2021-12-17 北京拙河科技有限公司 相机成像方法、装置、介质及设备
CN113837970B (zh) * 2021-09-30 2024-04-26 北京地平线信息技术有限公司 图像数据的脱敏方法和装置
CN116051386B (zh) * 2022-05-30 2023-10-20 荣耀终端有限公司 图像处理方法及其相关设备

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107578392A (zh) * 2017-09-25 2018-01-12 华北电力大学 一种基于残余插值的卷积神经网络去马赛克算法
CN110009590A (zh) * 2019-04-12 2019-07-12 北京理工大学 一种基于卷积神经网络的高质量彩色图像去马赛克方法
CN110378845A (zh) * 2019-06-17 2019-10-25 杭州电子科技大学 一种基于卷积神经网络的极端条件下的图像修复方法
US20190355098A1 (en) * 2018-05-18 2019-11-21 Gopro, Inc. Multiscale denoising of raw images with noise estimation
CN111402146A (zh) * 2020-02-21 2020-07-10 华为技术有限公司 图像处理方法以及图像处理装置

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR100647402B1 (ko) * 2005-11-01 2006-11-23 매그나칩 반도체 유한회사 이미지 센서의 화질개선장치 및 그 방법
US8837785B2 (en) * 2011-03-21 2014-09-16 Apple Inc. Red-eye removal using multiple recognition channels
CN103327220B (zh) * 2013-05-29 2016-09-07 湖南源信光电科技有限公司 低照度Bayer图像上以绿色通道为导向的去噪方法
KR20200015589A (ko) * 2017-05-29 2020-02-12 유니베르시태트 취리히 동적 비전 센서를 위한 블록-매칭 광학 흐름 및 스테레오 비전
CN109889800B (zh) * 2019-02-28 2021-09-10 深圳市商汤科技有限公司 图像增强方法和装置、电子设备、存储介质
CN109993707B (zh) * 2019-03-01 2023-05-12 华为技术有限公司 图像去噪方法和装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107578392A (zh) * 2017-09-25 2018-01-12 华北电力大学 一种基于残余插值的卷积神经网络去马赛克算法
US20190355098A1 (en) * 2018-05-18 2019-11-21 Gopro, Inc. Multiscale denoising of raw images with noise estimation
CN110009590A (zh) * 2019-04-12 2019-07-12 北京理工大学 一种基于卷积神经网络的高质量彩色图像去马赛克方法
CN110378845A (zh) * 2019-06-17 2019-10-25 杭州电子科技大学 一种基于卷积神经网络的极端条件下的图像修复方法
CN111402146A (zh) * 2020-02-21 2020-07-10 华为技术有限公司 图像处理方法以及图像处理装置

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
LIU LIN; JIA XU; LIU JIANZHUANG; TIAN QI: "Joint Demosaicing and Denoising With Self Guidance", 2020 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), IEEE, 13 June 2020 (2020-06-13), pages 2237 - 2246, XP033805260, DOI: 10.1109/CVPR42600.2020.00231 *
MICHAËL GHARBI ; GAURAV CHAURASIA ; SYLVAIN PARIS ; FRÉDO DURAND: "Deep joint demosaicking and denoising", ACM TRANSACTIONS ON GRAPHICS, ACM, NY, US, vol. 35, no. 6, 11 November 2016 (2016-11-11), US, pages 1 - 12, XP058306363, ISSN: 0730-0301, DOI: 10.1145/2980179.2982399 *
See also references of EP4109392A4

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114331916A (zh) * 2022-03-07 2022-04-12 荣耀终端有限公司 图像处理方法及电子设备
CN114596624A (zh) * 2022-04-20 2022-06-07 深圳市海清视讯科技有限公司 人眼状态检测方法、装置、电子设备及存储介质
CN114596624B (zh) * 2022-04-20 2022-08-05 深圳市海清视讯科技有限公司 人眼状态检测方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
CN111402146B (zh) 2022-05-10
CN111402146A (zh) 2020-07-10
EP4109392A1 (en) 2022-12-28
EP4109392A4 (en) 2024-01-03

Similar Documents

Publication Publication Date Title
WO2021164234A1 (zh) 图像处理方法以及图像处理装置
WO2020192483A1 (zh) 图像显示方法和设备
WO2021164731A1 (zh) 图像增强方法以及图像增强装置
US20220188999A1 (en) Image enhancement method and apparatus
WO2021063341A1 (zh) 图像增强方法以及装置
US11037278B2 (en) Systems and methods for transforming raw sensor data captured in low-light conditions to well-exposed images using neural network architectures
US20230214976A1 (en) Image fusion method and apparatus and training method and apparatus for image fusion model
WO2020177607A1 (zh) 图像去噪方法和装置
CN111914997B (zh) 训练神经网络的方法、图像处理方法及装置
CN113066017B (zh) 一种图像增强方法、模型训练方法及设备
CN112348747A (zh) 图像增强方法、装置及存储介质
CN111951195A (zh) 图像增强方法及装置
CN113011562A (zh) 一种模型训练方法及装置
WO2023151511A1 (zh) 模型训练方法、图像去摩尔纹方法、装置及电子设备
WO2024002211A1 (zh) 一种图像处理方法及相关装置
CN114627034A (zh) 一种图像增强方法、图像增强模型的训练方法及相关设备
WO2021042774A1 (zh) 图像恢复方法、图像恢复网络训练方法、装置和存储介质
CN115131256A (zh) 图像处理模型、图像处理模型的训练方法及装置
WO2022179606A1 (zh) 一种图像处理方法及相关装置
WO2022193132A1 (zh) 图像检测方法、装置和电子设备
WO2021189321A1 (zh) 一种图像处理方法和装置
CN115731115A (zh) 一种数据处理方法以及装置
CN115170456A (zh) 检测方法及相关设备
WO2023028866A1 (zh) 图像处理方法、装置和车辆
CN113066019A (zh) 一种图像增强方法及相关装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20920655

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2020920655

Country of ref document: EP

Effective date: 20220921