WO2020187029A1 - 图像处理方法及装置、神经网络的训练方法、存储介质 - Google Patents
图像处理方法及装置、神经网络的训练方法、存储介质 Download PDFInfo
- Publication number
- WO2020187029A1 WO2020187029A1 PCT/CN2020/077763 CN2020077763W WO2020187029A1 WO 2020187029 A1 WO2020187029 A1 WO 2020187029A1 CN 2020077763 W CN2020077763 W CN 2020077763W WO 2020187029 A1 WO2020187029 A1 WO 2020187029A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- sampling
- output
- processing
- sampling process
- level
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 544
- 238000012549 training Methods 0.000 title claims abstract description 104
- 238000003672 processing method Methods 0.000 title claims abstract description 80
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 72
- 238000005070 sampling Methods 0.000 claims abstract description 802
- 238000012545 processing Methods 0.000 claims abstract description 344
- 230000008569 process Effects 0.000 claims description 481
- 125000004122 cyclic group Chemical group 0.000 claims description 159
- 238000013527 convolutional neural network Methods 0.000 claims description 52
- 230000006870 function Effects 0.000 claims description 42
- 238000004458 analytical method Methods 0.000 claims description 21
- 230000015572 biosynthetic process Effects 0.000 claims description 19
- 238000003786 synthesis reaction Methods 0.000 claims description 19
- 239000010410 layer Substances 0.000 description 197
- 238000010586 diagram Methods 0.000 description 24
- 230000004913 activation Effects 0.000 description 20
- 230000011514 reflex Effects 0.000 description 8
- 230000000694 effects Effects 0.000 description 6
- 238000004422 calculation algorithm Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 5
- 238000010606 normalization Methods 0.000 description 5
- 238000013135 deep learning Methods 0.000 description 4
- 239000011159 matrix material Substances 0.000 description 4
- 210000002569 neuron Anatomy 0.000 description 4
- 238000005457 optimization Methods 0.000 description 4
- 241001465754 Metazoa Species 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000011176 pooling Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 239000012792 core layer Substances 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000001815 facial effect Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 229920006395 saturated elastomer Polymers 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20016—Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Definitions
- the embodiments of the present disclosure relate to an image processing method, an image processing device, a training method of a neural network, and a storage medium.
- CNN Convolutional Neural Network
- At least one embodiment of the present disclosure provides an image processing method, including: receiving a first characteristic image; and performing multi-scale cyclic sampling processing on the first characteristic image at least once;
- the multi-scale cyclic sampling processing includes nested first-level sampling processing and second-level sampling processing
- the first-level sampling processing includes first down-sampling processing, first up-sampling processing, and first residual linking Addition processing
- the first down-sampling processing performs down-sampling processing based on the input of the first-level sampling processing to obtain a first down-sampled output
- the first up-sampling processing performs up-sampling based on the first down-sampling output
- the first up-sampling output is obtained by processing
- the first residual link addition processing performs a first residual link addition on the input of the first-level sampling processing and the first up-sampling output, and then adds the first residual link
- the result of a residual link addition is used as the output of the first-level sampling process
- the second-level sampling process is nested between the first down-sampling process and the first up-sampling process, and receives the first
- the down-sampling output is used as the
- the size of the output of the first upsampling process is the same as the size of the input of the first downsampling process; the size of the output of the second upsampling process is The size is the same as the input size of the second downsampling process.
- the multi-scale cyclic sampling processing further includes a third-level sampling processing, and the third-level sampling processing is nested in the second down-sampling processing and the During the second up-sampling process, the second down-sampling output is received as the input of the third-level sampling process, and the output of the third-level sampling process is provided as the input of the second up-sampling process, so that the second up-sampling The processing performs up-sampling processing based on the second down-sampling output; the third-level sampling processing includes third down-sampling processing, third up-sampling processing, and third residual link addition processing, where the third down-sampling processing The sampling process performs down-sampling based on the input of the third-level sampling process to obtain a third down-sampled output, and the third up-sampling process performs up-sampling based on the third down-sampled output to obtain a third up-s
- the multi-scale cyclic sampling process includes the second-level sampling process that is sequentially executed multiple times, and the second-level sampling process receives the first-level sampling process for the first time.
- the one-shot output is used as the input of the first second-level sampling process, and each second-level sampling process except the first second-level sampling process receives the previous second-level sampling
- the processed output is used as the input of the second-level sampling process this time, and the output of the last second-level sampling process is used as the input of the first upsampling process.
- the at least one multi-scale cyclic sampling process includes the multi-scale cyclic sampling process performed sequentially multiple times, and each time the input of the multi-scale cyclic sampling process is The input of the first-level sampling processing in the multi-scale cyclic sampling processing this time, and the output of the first-level sampling processing in the multi-scale cyclic sampling processing each time is used as the multi-scale cyclic sampling this time Processing output; the first multi-scale cyclic sampling process receives the first feature image as the input of the first multi-scale cyclic sampling process, except for the first multi-scale cyclic sampling process.
- the multi-scale cyclic sampling process receives the output of the previous multi-scale cyclic sampling process as the input of this multi-scale cyclic sampling process, and the output of the last multi-scale cyclic sampling process is used as the at least one multi-scale cyclic sampling process.
- the output of the scale cycle sampling process is used as the at least one multi-scale cyclic sampling process.
- the multi-scale cyclic sampling processing further includes: performing the first down-sampling processing, the first up-sampling processing, the second down-sampling processing, and After the second up-sampling processing, perform instance standardization processing or layer standardization processing on the first down-sampling output, the first up-sampling output, the second down-sampling output, and the second up-sampling output, respectively .
- the image processing method provided by some embodiments of the present disclosure further includes: using a first convolutional neural network to perform the multi-scale cyclic sampling processing; wherein, the first convolutional neural network includes: a first element network for Perform the first-level sampling processing; a second meta network for performing the second-level sampling processing.
- the first meta-network includes: a first sub-network for performing the first down-sampling process; a second sub-network for performing the first down-sampling process An up-sampling process; the second meta-network includes: a third sub-network for performing the second down-sampling process; a fourth sub-network for performing the second up-sampling process.
- each of the first sub-network, the second sub-network, the third sub-network, and the fourth sub-network includes a convolutional layer, One of residual networks and dense networks.
- each of the first sub-network, the second sub-network, the third sub-network, and the fourth sub-network includes instance standardization A layer or layer standardization layer, the instance standardization layer is used to perform instance standardization processing, and the layer standardization layer is used to perform layer standardization processing.
- the image processing method provided by some embodiments of the present disclosure further includes: acquiring an input image; using an analysis network to convert the input image into the first feature image; and using a synthesis network to process the output of the at least one multi-scale cyclic sampling process Convert to output image.
- At least one embodiment of the present disclosure further provides a neural network training method, wherein the neural network includes: an analysis network, a first sub-neural network, and a synthesis network, and the analysis network processes the input image to obtain the first feature Image, the first sub-neural network performs multi-scale cyclic sampling processing on the first feature image at least once to obtain a second feature image, and the synthesis network processes the second feature image to obtain an output image;
- the training method includes: obtaining a training input image; using the analysis network to process the training input image to provide a first training feature image; using the first sub-neural network to perform an analysis on the first training feature image
- the at least one multi-scale cyclic sampling process is used to obtain a second training feature image; the synthesis network is used to process the second training feature image to obtain a training output image; based on the training output image, the loss function is used to calculate the The loss value of the neural network; and correcting the parameters of the neural network according to the loss value;
- the multi-scale cyclic sampling processing includes nested first-level sampling processing and second-level sampling processing
- the first-level sampling processing includes first down-sampling processing, first up-sampling processing, and first residual linking Addition processing
- the first down-sampling processing performs down-sampling processing based on the input of the first-level sampling processing to obtain a first down-sampled output
- the first up-sampling processing performs up-sampling based on the first down-sampling output
- the first up-sampling output is obtained by processing
- the first residual link addition processing performs a first residual link addition on the input of the first-level sampling processing and the first up-sampling output, and then adds the first residual link
- the result of a residual link addition is used as the output of the first-level sampling process
- the second-level sampling process is nested between the first down-sampling process and the first up-sampling process, and receives the first
- the down-sampling output is used as the
- the size of the output of the first upsampling process is the same as the size of the input of the first downsampling process; the size of the output of the second upsampling process The same size as the input of the second downsampling process.
- the multi-scale cyclic sampling process further includes a third-level sampling process, and the third-level sampling process is nested in the second down-sampling process and the second down-sampling process.
- the second down-sampling output is received as the input of the third-level sampling process, and the output of the third-level sampling process is provided as the input of the second up-sampling process, so that the second up-sampling process Up-sampling processing is performed based on the second down-sampling output;
- the third-level sampling processing includes third down-sampling processing, third up-sampling processing, and third residual link addition processing, where the third down-sampling Processing is performed based on the input of the third-level sampling process to perform down-sampling processing to obtain a third down-sampled output, and the third up-sampling process performs up-sampling based on the third down-sampled output to obtain a third up-sampled output, the The third residual link addition process performs a third residual link addition on the input of the third level sampling process and the third up-sampling output, and then uses the result of the third residual link addition as the The output of the third-level sampling process.
- the multi-scale cyclic sampling processing includes the second-level sampling processing that is sequentially executed multiple times, and the second-level sampling processing receives the first
- the down-sampling output is used as the input of the first second-level sampling process, and each second-level sampling process except the first second-level sampling process receives the previous second-level sampling process
- the output of is used as the input of the second-level sampling process this time, and the output of the last second-level sampling process is used as the input of the first upsampling process.
- the at least one multi-scale cyclic sampling processing includes the multi-scale cyclic sampling processing performed sequentially multiple times, and each time the input of the multi-scale cyclic sampling processing is used as the original
- the input of the first-level sampling process in the multi-scale cyclic sampling process, and the output of the first-level sampling process in the multi-scale cyclic sampling process is used as the multi-scale cyclic sampling process this time
- the output of the first multi-scale cyclic sampling process receives the first training feature image as the input of the first multi-scale cyclic sampling process, except for the first multi-scale cyclic sampling process every time
- the multi-scale cyclic sampling process receives the output of the previous multi-scale cyclic sampling process as the input of this multi-scale cyclic sampling process, and the output of the last multi-scale cyclic sampling process is used as the at least one multi-scale cyclic sampling process.
- the output of the scale cycle sampling process is performed sequentially multiple times, and each time the input of the multi-scale cycl
- the multi-scale cyclic sampling processing further includes: performing the first down-sampling processing, the first up-sampling processing, the second down-sampling processing, and After the second up-sampling processing, perform instance standardization processing or layer standardization processing on the first down-sampling output, the first up-sampling output, the second down-sampling output, and the second up-sampling output, respectively .
- the first sub-neural network includes: a first meta-network for performing the first-level sampling processing; a second meta-network for performing the first-level sampling process; Two-level sampling processing.
- the first meta-network includes: a first sub-network for performing the first downsampling process; a second sub-network for performing the first Up-sampling processing; the second meta-network includes: a third sub-network for performing the second down-sampling processing; a fourth sub-network for performing the second up-sampling processing.
- each of the first sub-network, the second sub-network, the third sub-network, and the fourth sub-network includes a convolutional layer, One of residual networks and dense networks.
- each of the first sub-network, the second sub-network, the third sub-network, and the fourth sub-network includes an instance standardization layer Or a layer standardization layer
- the instance standardization layer is used to perform instance standardization processing on the first down-sampling output, the first up-sampling output, the second down-sampling output, and the second up-sampling output, respectively
- the layer standardization layer is used to perform layer standardization processing on the first down-sampling output, the first up-sampling output, the second down-sampling output, and the second up-sampling output, respectively.
- At least one embodiment of the present disclosure further provides an image processing device, including: a memory for non-transitory storage of computer-readable instructions; and a processor for running the computer-readable instructions, the computer-readable instructions being The processor executes the image processing method provided by any embodiment of the present disclosure or the neural network training method provided by any embodiment of the present disclosure while running.
- At least one embodiment of the present disclosure further provides a storage medium that non-temporarily stores computer-readable instructions, and when the computer-readable instructions are executed by a computer, the instructions or instructions of the image processing method provided in any embodiment of the present disclosure can be executed. Instructions of the neural network training method provided by any embodiment of the present disclosure.
- Figure 1 is a schematic diagram of a convolutional neural network
- Figure 2A is a schematic diagram of a convolutional neural network
- Figure 2B is a schematic diagram of the working process of a convolutional neural network
- FIG. 3 is a flowchart of an image processing method provided by an embodiment of the disclosure.
- FIG. 4A is a schematic flow chart of a multi-scale cyclic sampling process corresponding to the image processing method shown in FIG. 3 according to an embodiment of the present disclosure
- FIG. 4B is a schematic flowchart diagram corresponding to the multi-scale cyclic sampling processing in the image processing method shown in FIG. 3 according to another embodiment of the present disclosure
- FIG. 4C is a schematic flowchart diagram corresponding to the multi-scale cyclic sampling processing in the image processing method shown in FIG. 3 according to still another embodiment of the present disclosure
- FIG. 4D is a schematic flowchart diagram corresponding to the multi-scale cyclic sampling processing in the image processing method shown in FIG. 3 according to another embodiment of the present disclosure
- FIG. 5 is a flowchart of an image processing method provided by another embodiment of the present disclosure.
- Fig. 6A is a schematic diagram of an input image
- FIG. 6B is a schematic diagram of an output image obtained by processing the input image shown in FIG. 6A according to an image processing method provided by an embodiment of the present disclosure
- FIG. 7A is a schematic structural diagram of a neural network provided by an embodiment of the disclosure.
- FIG. 7B is a flowchart of a neural network training method provided by an embodiment of the disclosure.
- FIG. 7C is a schematic structural block diagram of training the neural network shown in FIG. 7A corresponding to the training method shown in FIG. 7B according to an embodiment of the present disclosure
- FIG. 8 is a schematic block diagram of an image processing apparatus provided by an embodiment of the present disclosure.
- FIG. 9 is a schematic diagram of a storage medium provided by an embodiment of the disclosure.
- Image enhancement is one of the research hotspots in the field of image processing. Due to the limitations of various physical factors in the image acquisition process (for example, the size of the image sensor of the mobile phone camera is too small and other software and hardware limitations) and the interference of environmental noise, the image quality will be greatly reduced.
- the purpose of image enhancement is to improve the grayscale histogram of the image and the contrast of the image through image enhancement technology, thereby highlighting the detailed information of the image and improving the visual effect of the image.
- the use of deep neural networks for image enhancement is a technology emerging with the development of deep learning technology.
- low-quality photos input images
- the quality of the output images can be close to that of digital single-lens reflex cameras (Digital Single Lens Reflex Camera).
- DSLR Digital Single Lens Reflex Camera
- the quality of the photos taken For example, the Peak Signal to Noise Ratio (PSNR) index is commonly used to characterize image quality, where the higher the PSNR value, the closer the image is to the real photos taken by a digital single-lens reflex camera.
- PSNR Peak Signal to Noise Ratio
- Andrey Ignatov et al. proposed a method of convolutional neural network to achieve image enhancement, please refer to the literature, Andrey Ignatov, Nikolay Kobyshev, Kenneth Vanhoey, Radu Timofte, Luc Van Gool, DSLR-Quality Photos on Mobile Devices with Deep Convolutional Networks.arXiv:1704.02470v2[cs.CV], September 5, 2017. This document is hereby incorporated by reference in its entirety as a part of this application.
- This method mainly uses convolutional layers, batch normalization layers and residual connections to construct a single-scale convolutional neural network.
- the network can be used to input low-quality images (for example, low contrast, underexposure or exposure Excessive, the entire image is too dark or too bright, etc.) processed into a higher quality image.
- low-quality images for example, low contrast, underexposure or exposure Excessive, the entire image is too dark or too bright, etc.
- color loss, texture loss and content loss as the loss function in training can achieve better processing results.
- At least one embodiment of the present disclosure provides an image processing method, an image processing device, a neural network training method, and a storage medium.
- This image processing method proposes a multi-scale cyclic sampling method based on convolutional neural network. By repeatedly sampling at multiple scales to obtain higher image fidelity, the quality of the output image can be greatly improved, and it is suitable for image processing. Offline applications such as batch processing with high quality requirements.
- CNN Convolutional Neural Network
- FIG. 1 shows a schematic diagram of a convolutional neural network.
- the convolutional neural network can be used for image processing, which uses images as input and output, and replaces scalar weights with convolution kernels.
- FIG. 1 only shows a convolutional neural network with a 3-layer structure, which is not limited in the embodiment of the present disclosure.
- the convolutional neural network includes an input layer 101, a hidden layer 102, and an output layer 103.
- the input layer 101 has 4 inputs
- the hidden layer 102 has 3 outputs
- the output layer 103 has 2 outputs.
- the convolutional neural network finally outputs 2 images.
- the 4 inputs of the input layer 101 may be 4 images, or 4 feature images of 1 image.
- the three outputs of the hidden layer 102 may be characteristic images of the image input through the input layer 101.
- the convolutional layer has weights And bias Weights Represents the convolution kernel, bias Is a scalar superimposed on the output of the convolutional layer, where k is the label of the input layer 101, and i and j are the labels of the unit of the input layer 101 and the unit of the hidden layer 102, respectively.
- the first convolutional layer 201 includes a first set of convolution kernels (in Figure 1 ) And the first set of offsets (in Figure 1 ).
- the second convolution layer 202 includes a second set of convolution kernels (in Figure 1 ) And the second set of offsets (in Figure 1 ).
- each convolutional layer includes tens or hundreds of convolution kernels. If the convolutional neural network is a deep convolutional neural network, it may include at least five convolutional layers.
- the convolutional neural network further includes a first activation layer 203 and a second activation layer 204.
- the first activation layer 203 is located behind the first convolutional layer 201
- the second activation layer 204 is located behind the second convolutional layer 202.
- the activation layer (for example, the first activation layer 203 and the second activation layer 204) includes activation functions, which are used to introduce nonlinear factors into the convolutional neural network, so that the convolutional neural network can better solve more complex problems .
- the activation function may include a linear correction unit (ReLU) function, a sigmoid function (Sigmoid function), or a hyperbolic tangent function (tanh function).
- the ReLU function is an unsaturated nonlinear function
- the Sigmoid function and tanh function are saturated nonlinear functions.
- the activation layer can be used as a layer of the convolutional neural network alone, or the activation layer can also be included in the convolutional layer (for example, the first convolutional layer 201 can include the first activation layer 203, and the second convolutional layer 202 can be Including the second active layer 204).
- the first convolution layer 201 For example, in the first convolution layer 201, first, several convolution kernels in the first group of convolution kernels are applied to each input And several offsets in the first set of offsets In order to obtain the output of the first convolutional layer 201; then, the output of the first convolutional layer 201 can be processed by the first activation layer 203 to obtain the output of the first activation layer 203.
- the second convolutional layer 202 first, apply several convolution kernels in the second set of convolution kernels to the output of the input first activation layer 203 And several offsets in the second set of offsets In order to obtain the output of the second convolutional layer 202; then, the output of the second convolutional layer 202 can be processed by the second activation layer 204 to obtain the output of the second activation layer 204.
- the output of the first convolutional layer 201 can be a convolution kernel applied to its input Offset
- the output of the second convolutional layer 202 can be a convolution kernel applied to the output of the first activation layer 203 Offset The result of the addition.
- the convolutional neural network Before using the convolutional neural network for image processing, the convolutional neural network needs to be trained. After training, the convolution kernel and bias of the convolutional neural network remain unchanged during image processing. In the training process, each convolution kernel and bias are adjusted through multiple sets of input/output example images and optimization algorithms to obtain an optimized convolutional neural network model.
- FIG. 2A shows a schematic diagram of the structure of a convolutional neural network
- FIG. 2B shows a schematic diagram of the working process of a convolutional neural network.
- the main components of a convolutional neural network can include multiple convolutional layers, multiple downsampling layers, and fully connected layers.
- each of these layers refers to a corresponding processing operation, that is, convolution processing, downsampling processing, fully connected processing, etc.
- the described neural network also refers to the corresponding processing operation, the example standardization layer or layer standardization layer to be described below is similar to this, and the description is not repeated here.
- a complete convolutional neural network can be composed of these three layers.
- FIG. 2A only shows three levels of a convolutional neural network, namely the first level, the second level, and the third level.
- each level may include a convolution module and a downsampling layer.
- each convolution module may include a convolution layer.
- the processing process of each level may include: convolution and down-sampling of the input image.
- each convolution module may also include an instance normalization layer or a layer normalization layer, so that the processing process of each level may also include instance normalization processing or layer normalization processing.
- the instance standardization layer is used to perform instance standardization processing on the feature image output by the convolutional layer, so that the gray value of the pixel of the feature image changes within a predetermined range, thereby simplifying the image generation process and improving the effect of image enhancement.
- the predetermined range may be [-1, 1].
- the instance standardization layer performs instance standardization processing on each feature image according to its own mean and variance.
- the instance standardization layer can also be used to perform instance standardization processing on a single image.
- the instance standardization formula of the instance standardization layer can be expressed as follows:
- x tijk is the value of the t-th feature image, the i-th feature image, the j-th row, and the k-th column in the feature image set output by the convolutional layer.
- y tijk represents the result obtained after processing x tijk by the instance standardization layer.
- ⁇ 1 is a small integer to avoid zero denominator.
- the layer standardization layer is similar to the instance standardization layer, and is also used to perform layer standardization processing on the feature image output by the convolutional layer, so that the gray value of the pixel of the feature image changes within a predetermined range, thereby simplifying the image generation process and improving The effect of image enhancement.
- the predetermined range may be [-1, 1].
- the layer standardization layer performs layer standardization processing on each column of the characteristic image according to the mean value and variance of each column of each characteristic image, thereby realizing the layer standardization processing of the characteristic image.
- the layer standardization layer can also be used to perform layer standardization processing on a single image.
- the model of the feature image is expressed as (T, C, H, W). Therefore, the layer standardization formula of the layer standardization layer can be expressed as follows:
- x tijk is the value of the t-th feature image, the i-th feature image, the j-th row, and the k-th column in the feature image set output by the convolutional layer.
- y′ tijk represents the result obtained after processing x tijk by the layer standardization layer.
- ⁇ 2 is a small integer to avoid zero denominator.
- the convolutional layer is the core layer of the convolutional neural network.
- a neuron is only connected to some of the neurons in the adjacent layer.
- the convolutional layer can apply several convolution kernels (also called filters) to the input image to extract multiple types of features of the input image.
- Each convolution kernel can extract one type of feature.
- the convolution kernel is generally initialized in the form of a random decimal matrix. During the training process of the convolutional neural network, the convolution kernel will learn to obtain reasonable weights.
- the result obtained after applying a convolution kernel to the input image is called a feature map, and the number of feature images is equal to the number of convolution kernels.
- Each feature image is composed of some rectangularly arranged neurons, and the neurons of the same feature image share weights, and the shared weights here are the convolution kernels.
- the feature image output by the convolution layer of one level can be input to the convolution layer of the next next level and processed again to obtain a new feature image.
- the first-level convolutional layer may output a first-level feature image
- the first-level feature image is input to the second-level convolutional layer and processed again to obtain a second-level feature image.
- the convolutional layer can use different convolution kernels to convolve the data of a certain local receptive field of the input image, and the convolution result is input to the activation layer, which is calculated according to the corresponding activation function To get the characteristic information of the input image.
- the down-sampling layer is arranged between adjacent convolutional layers, and the down-sampling layer is a form of down-sampling.
- the down-sampling layer can be used to reduce the scale of the input image, simplify the calculation complexity, and reduce over-fitting to a certain extent; on the other hand, the down-sampling layer can also perform feature compression to extract the input image Main features.
- the down-sampling layer can reduce the size of feature images, but does not change the number of feature images.
- a 2 ⁇ 2 output image can be obtained, which means that 36 pixels on the input image are merged into the output image. 1 pixel.
- the last downsampling layer or convolutional layer can be connected to one or more fully connected layers, which are used to connect all the extracted features.
- the output of the fully connected layer is a one-dimensional matrix, which is a vector.
- FIG. 3 is a flowchart of an image processing method provided by an embodiment of the disclosure.
- the image processing method includes:
- Step S110 receiving a first characteristic image
- Step S120 Perform at least one multi-scale cyclic sampling process on the first feature image.
- the first feature image may include a feature image obtained after the input image is processed by one of a convolutional layer, a residual network, a dense network, etc. (for example, refer to FIG. 2B).
- the residual network maintains its input in a certain proportion in its output by means of, for example, adding residual connections.
- a dense network includes a bottleneck layer and a convolution layer.
- the bottleneck layer is used to reduce the dimensionality of the data to reduce the number of parameters in the subsequent convolution operation, such as the convolution kernel of the bottleneck layer.
- the convolution kernel of the convolution layer is a 3 ⁇ 3 convolution kernel; the present disclosure includes but is not limited to this.
- the input image is processed by convolution, down-sampling, etc. to obtain the first feature image.
- this embodiment does not limit the acquisition method of the first characteristic image.
- the first characteristic image may include a plurality of characteristic images, but is not limited thereto.
- the first feature image received in step S110 is used as the input of the multi-scale cyclic sampling process in step S120.
- the multi-scale cyclic sampling process may have various forms, including but not limited to the three forms shown in FIGS. 4A-4C which will be described below.
- FIG. 4A is a schematic flowchart diagram corresponding to the multi-scale cyclic sampling processing in the image processing method shown in FIG. 3 according to an embodiment of the present disclosure.
- the multi-scale cyclic sampling processing includes nested first-level sampling processing and second-level sampling processing.
- the input of the multi-scale cyclic sampling processing is used as the input of the first-level sampling processing
- the output of the first-level sampling processing is used as the output of the multi-scale cyclic sampling processing.
- the output of the multi-scale cyclic sampling process is called the second feature image.
- the size of the second feature image (the number of rows and columns of the pixel array) may be the same as the size of the first feature image.
- the first-level sampling process includes a first down-sampling process, a first up-sampling process, and a first residual link addition process that are sequentially executed.
- the first down-sampling process is performed based on the input of the first-level sampling process to obtain the first down-sampled output.
- the first down-sampling process can directly down-sample the input of the first-level sampling process to obtain the first down-sampling Output.
- the first up-sampling process is performed based on the first down-sampling output to perform up-sampling processing to obtain the first up-sampling output, for example, after the first down-sampling output is subjected to the second-level sampling process, the up-sampling process is performed to obtain the first up-sampling output That is, the first up-sampling process can indirectly perform up-sampling on the first down-sampling output.
- the first residual link addition process performs the first residual link addition on the input of the first level sampling process and the first upsampling output, and then uses the result of the first residual link addition as the output of the first level sampling process .
- the size of the output of the first up-sampling process is the same as the size of the input of the first-level sampling process (ie, the input of the first down-sampling process), which is added through the first residual link
- the size of the output of the first-level sampling process is the same as the size of the input of the first-level sampling process.
- the second-level sampling process is nested between the first down-sampling process and the first up-sampling process of the first-level sampling process, and the first down-sampling output is received as the input of the second-level sampling process. , Providing the output of the second-level sampling process as the input of the first up-sampling process, so that the first up-sampling process performs the up-sampling process based on the first down-sampling output.
- the second-level sampling process includes a second down-sampling process, a second up-sampling process, and a second residual link addition process that are sequentially executed.
- the second down-sampling process performs down-sampling based on the input of the second-level sampling process to obtain the second down-sampled output.
- the second down-sampling process can directly down-sample the input of the second-level sampling process to obtain the second down-sampling Output.
- the second up-sampling process performs up-sampling based on the second down-sampled output to obtain the second up-sampled output.
- the second up-sampling process may directly up-sample the second down-sampled output to obtain the second up-sampled output.
- the second residual link addition process performs a second residual link addition on the input of the second level sampling process and the second upsampling output, and then uses the result of the second residual link addition as the output of the second level sampling process .
- the size of the output of the second up-sampling process (ie, the second up-sampling output) is the same as the size of the input of the second-level sampling process (ie, the input of the second down-sampling process), so that it is added through the second residual link
- the size of the output of the second-level sampling process is the same as the size of the input of the second-level sampling process.
- the sampling processing of each level for example, the first-level sampling processing, the second-level sampling processing, and the embodiment shown in FIG. 4B will be The procedures of the third-level sampling processing, etc. introduced are similar, including down-sampling processing, up-sampling processing and residual link addition processing.
- the residual link addition processing may include correspondingly adding the values of each row and each column of the matrix of the two feature images, but it is not limited to this.
- “nested” means that an object includes another object that is similar or identical to the object, and the object includes but is not limited to a process or a network structure.
- the size of the output of the upsampling process (for example, the output of the upsampling process is a feature image) in the sampling process of each level and the input of the downsampling process (for example, The input of the down-sampling process is the feature image) with the same size, so after the residual link addition, the output of the sampling process of each level (for example, the output of the sampling process of each level can be the feature image) size and each The input of the sampling process of the levels (for example, the input of the sampling process of each level may be a feature image) has the same size.
- multi-scale cyclic sampling processing can be implemented by a convolutional neural network.
- the first convolutional neural network may be used to perform multi-scale cyclic sampling processing.
- the first convolutional neural network may include a nested first meta network and a second meta network, the first meta network is used to perform the first level of sampling processing, and the second meta network is used to perform the second Hierarchical sampling processing.
- the first meta-network may include a first sub-network and a second sub-network, the first sub-network is used to perform the first down-sampling process, and the second sub-network is used to perform the first up-sampling process.
- the second meta network is nested between the first sub network and the third sub network of the first meta network.
- the second meta-network may include a third sub-network and a fourth sub-network, the third sub-network is used to perform the second down-sampling process, and the fourth sub-network is used to perform the second up-sampling process.
- both the first meta network and the second meta network are similar to the aforementioned residual network.
- each of the first sub-network, the second sub-network, the third sub-network, and the fourth sub-network includes one of a convolutional layer, a residual network, a dense network, and the like.
- the first sub-network and the third sub-network may include a convolutional layer (down-sampling layer) with down-sampling function, and may also include one of residual networks and dense networks with down-sampling function;
- the and fourth sub-network may include a convolutional layer (up-sampling layer) with an up-sampling function, and may also include one of a residual network with an up-sampling function, a dense network, and the like.
- the first sub-network and the third sub-network may have the same structure or different structures; the second sub-network and the fourth sub-network may have the same structure or different structures; The disclosed embodiment does not limit this.
- Down-sampling is used to reduce the size of the feature image, thereby reducing the data amount of the feature image.
- down-sampling can be performed through the down-sampling layer, but is not limited to this.
- the down-sampling layer can use max pooling, average pooling, strided convolution, decimation, such as selecting fixed pixels, and demultiplexing output (demuxout, Split the input image into multiple smaller images) and other down-sampling methods to achieve down-sampling processing.
- Upsampling is used to increase the size of the feature image, thereby increasing the data volume of the feature image.
- upsampling can be performed through an upsampling layer, but is not limited to this.
- the up-sampling layer can adopt up-sampling methods such as strided transposed convolution and interpolation algorithms to implement up-sampling processing.
- the interpolation algorithm may include, for example, interpolation, bilinear interpolation, and bicubic interpolation (Bicubic Interprolation).
- the downsampling factor of the downsampling process at the same level corresponds to the upsampling factor of the upsampling process, that is, when the downsampling factor of the downsampling process is 1/y .
- the upsampling factor of the upsampling process is y, where y is a positive integer, and y is usually greater than 2.
- the parameters of the downsampling process at different levels may be the same or different; different levels
- the parameters of the upsampling process that is, the parameters of the network corresponding to the upsampling process
- the added parameters of the residual connections at different levels can be the same or different. This disclosure does not limit this.
- the multi-scale cyclic sampling processing may also include: first down-sampling processing, first up-sampling processing After the processing, the second down-sampling process, and the second up-sampling process, the first down-sampling output, the first up-sampling output, the second down-sampling output, and the second up-sampling output are respectively subjected to instance standardization processing or layer standardization processing.
- first down-sampling output, the first up-sampling output, the second down-sampling output, and the second up-sampling output can use the same standardization processing method (instance standardization processing or layer standardization processing), or different Standardized processing method, this disclosure does not limit this.
- the first sub-network, the second sub-network, the third sub-network and the fourth sub-network also include an instance standardization layer or a layer standardization layer, respectively, the instance standardization layer is used to perform instance standardization processing, and the layer standardization layer is used to execute the layer.
- Standardized processing can perform instance standardization processing according to the aforementioned instance standardization formula
- the layer standardization layer can perform layer standardization processing according to the aforementioned layer standardization formula, which is not limited in the present disclosure.
- first sub-network, the second sub-network, the third sub-network, and the fourth sub-network may include the same standardization layer (example standardization layer or layer standardization layer), or can include different standardization layers, the present disclosure There is no restriction on this.
- FIG. 4B is a schematic flowchart diagram corresponding to the multi-scale cyclic sampling processing in the image processing method shown in FIG. 3 according to another embodiment of the present disclosure.
- the multi-scale cyclic sampling process further includes a third-level sampling process.
- the other procedures of the multi-scale cyclic sampling processing shown in FIG. 5 are basically the same as the procedures of the multi-scale cyclic sampling processing shown in FIG. 4A, and the repetitions are not repeated here.
- the third-level sampling process is nested between the second down-sampling process and the second up-sampling process of the second-level sampling process, and the second down-sampling output is received as the input of the third-level sampling process.
- the second up-sampling process also indirectly up-sampling the second down-sampled output.
- the third-level sampling process includes a third down-sampling process, a third up-sampling process, and a third residual link addition process that are sequentially executed.
- the third down-sampling process is performed based on the input of the third-level sampling process to obtain the third down-sampled output.
- the third down-sampling process can directly down-sample the input of the third-level sampling process to obtain the third down-sampling Output.
- the third up-sampling process performs up-sampling based on the third down-sampled output to obtain the third up-sampled output.
- the third up-sampling process may directly up-sample the third down-sampled output to obtain the third up-sampled output.
- the third residual link addition process performs the third residual link addition on the input of the third level sampling process and the third upsampling output, and then uses the third residual link addition result as the output of the third level sampling process .
- the size of the output of the third up-sampling process ie, the third up-sampling output
- the size of the input of the third-level sampling process ie, the input of the third down-sampling process
- the size of the output of the third-level sampling process is the same as the size of the input of the third-level sampling process.
- multi-scale cyclic sampling processing may also include more levels of sampling processing, for example, it may also include a fourth level nested in the third level of sampling processing.
- the sampling processing, the fifth-level sampling processing nested in the fourth-level sampling processing, etc., the nesting method is similar to the second-level sampling processing and the third-level sampling processing described above. limit.
- FIG. 4C is a schematic flowchart diagram corresponding to the multi-scale cyclic sampling processing in the image processing method shown in FIG. 3 according to another embodiment of the present disclosure.
- the multi-scale cyclic sampling process includes a second-level sampling process that is sequentially executed multiple times.
- the other procedures of the multi-scale cyclic sampling processing shown in FIG. 5 are basically the same as the procedures of the multi-scale cyclic sampling processing shown in FIG. 4A, and the repetitions are not repeated here.
- the inclusion of two second-level sampling processing in FIG. 4C is exemplary.
- the multi-scale cyclic sampling processing may include two or more second-level sampling performed sequentially. deal with.
- the number of second-level sampling processing can be selected according to actual needs, and the present disclosure does not limit this.
- the inventor of the present application found that compared to using an image processing method with one or three second-level sampling processing, an image processing method with two second-level sampling processing is used for image enhancement processing. The effect is better, but this should not be seen as a limitation of the present disclosure.
- the first second-level sampling process receives the first down-sampling output as the input of the first second-level sampling process, and every second-level sampling process except the first second-level sampling process receives the previous one
- the output of the second-level sampling process is used as the input of this second-level sampling process
- the output of the last second-level sampling process is used as the input of the first upsampling process.
- the parameters of the downsampling process of the same level in different orders may be the same or different; the parameters of the upsampling process of the same level in different orders It can be the same or different; the added parameters of the residual connections of the same level in different orders can be the same or different. This disclosure does not limit this.
- the first-level sampling process can nest multiple second-level sampling processes that are executed in sequence; further, at least partially The second-level sampling process may nest one or more third-level sampling processes that are executed sequentially, and the number of third-level sampling processes nested in at least part of the second-level sampling process may be the same or different; further, The third-level sampling processing can nest the fourth-level sampling processing, and the specific nesting manner may be the same as the second-level sampling processing nesting the third-level sampling processing; and so on.
- FIGS. 4A-4C show a situation where the image processing method provided by an embodiment of the present disclosure includes a multi-scale cyclic sampling process.
- at least one multi-scale cyclic sampling processing includes one multi-scale cyclic sampling processing.
- the multi-scale cyclic sampling processing receives the first feature image as the input of the multi-scale cyclic sampling processing, and the input of the multi-scale cyclic sampling processing is used as the input of the first-level sampling processing in the multi-scale cyclic sampling processing.
- the output of the first-level sampling processing is used as the output of the multi-scale cyclic sampling processing, and the output of the multi-scale cyclic sampling processing is used as the output of the multi-scale cyclic sampling processing at least once.
- the present disclosure includes but is not limited to this.
- FIG. 4D is a schematic flow chart corresponding to the multi-scale cyclic sampling processing in the image processing method shown in FIG. 3 according to another embodiment of the present disclosure.
- at least one multi-scale cyclic sampling process includes multiple times of sequential execution of multi-scale cyclic sampling processing.
- at least one multi-scale cyclic sampling process may include two or three times.
- Multi-scale cyclic sampling processing executed sequentially, but not limited to this. It should be noted that in the embodiments of the present disclosure, the number of times of multi-scale cyclic sampling processing can be selected according to actual needs, and the present disclosure does not limit this.
- the inventor of the present application found that compared to the image processing method with one or three multi-scale cyclic sampling processing, the image processing method with two multi-scale cyclic sampling processing is used for image enhancement processing.
- the effect is better, but this should not be seen as a limitation of the present disclosure.
- each multi-scale cyclic sampling process is used as the input of the first-level sampling process in this multi-scale cyclic sampling process
- the output of the first-level sampling process in each multi-scale cyclic sampling process is used as the current multi-scale The output of cyclic sampling processing.
- the first multi-scale cyclic sampling process receives the first feature image as the input of the first multi-scale cyclic sampling process, and each multi-scale cycle except the first multi-scale cyclic sampling process
- the sampling process receives the output of the previous multi-scale cyclic sampling process as the input of this multi-scale cyclic sampling process, and the output of the last multi-scale cyclic sampling process is used as the output of at least one multi-scale cyclic sampling process.
- Fig. 5 is a flowchart of an image processing method provided by another embodiment of the present disclosure.
- the image processing method includes step S210 to step S250.
- steps S230 to S240 of the image processing method shown in FIG. 5 correspond to the same as steps S110 to S120 of the image processing method shown in FIG. 3, that is, the image processing method shown in FIG. Therefore, steps S230 to S240 of the image processing method shown in FIG. 5 can refer to the foregoing description of steps S110 to S120 of the image processing method shown in FIG. 3, and of course, you can also refer to FIG. 4A to The method of the embodiment shown in 4D, etc.
- steps S210 to S250 of the image processing method shown in FIG. 5 will be described in detail.
- Step S210 Obtain an input image.
- the input image may include photos captured by a camera of a smart phone, a camera of a tablet computer, a camera of a personal computer, a lens of a digital camera, a surveillance camera, or a web camera, etc., which may include images of people, animations, etc. Plant images or landscape images, etc., are not limited in this disclosure.
- the quality of the input image is lower than the quality of photos taken by a real digital single-lens reflex camera, that is, the input image is a low-quality image.
- the input image may include a 3-channel RGB image; in other examples, the input image may include a 3-channel YUV image.
- the input image includes an RGB image as an example, but the embodiment of the present disclosure is not limited to this.
- Step S220 Use the analysis network to convert the input image into a first feature image.
- the analysis network may be a convolutional neural network including one of a convolutional layer, a residual network, and a dense network.
- the analysis network can convert 3 channel RGB images (ie, input images) into multiple first feature images, such as 64 first feature images.
- RGB images ie, input images
- first feature images such as 64 first feature images.
- the present disclosure includes but is not limited to this.
- the embodiment of the present disclosure does not limit the structure and parameters of the analysis network, as long as it can convert the input image to the convolution feature dimension (ie, convert it to the first feature image).
- Step S230 Receive the first characteristic image
- Step S240 Perform at least one multi-scale cyclic sampling process on the first feature image.
- step S230 to step S240 reference may be made to the foregoing description of step S110 to step S120, which will not be repeated in this disclosure.
- Step S250 Use a synthesis network to convert the output of at least one multi-scale cyclic sampling process into an output image.
- the synthesis network may be a convolutional neural network including one of a convolutional layer, a residual network, a dense network, and the like.
- the output of at least one multi-scale cyclic sampling process can be referred to as the second feature image.
- the number of second feature images may be multiple, but is not limited to this.
- the synthesis network may convert multiple second feature images into output images.
- the output image may include 3 channel RGB images. The present disclosure includes but is not limited to this.
- FIG. 6A is a schematic diagram of an input image
- FIG. 6B is a result obtained by processing the input image shown in FIG. 6A according to an image processing method (for example, the image processing method shown in FIG. 5) provided by an embodiment of the present disclosure Schematic of the output image.
- an image processing method for example, the image processing method shown in FIG. 5
- the output image retains the content of the input image, but the contrast of the image is improved, and the problem of the input image being too dark is improved, so that the quality of the output image can be close to that of the input image.
- the output image is a high-quality image.
- the embodiment of the present disclosure does not limit the structure and parameters of the synthesis network, as long as it can convert the convolution feature dimension (ie, the second feature image) into an output image.
- the image processing method provided by the embodiments of the present disclosure can perform image enhancement processing on low-quality input images, and by repeatedly sampling at multiple scales to obtain higher image fidelity, the quality of output images can be greatly improved.
- the PSNR of the image output by the image enhancement method proposed by Andrey Ignatov et al. is 20.08, while the PSNR of the output image obtained based on the image processing method provided in the embodiment shown in FIG. 4C of the present disclosure can reach 23.35, that is The image obtained by the image processing method provided by the embodiments of the present disclosure may be closer to a real photo taken by a digital single-lens reflex camera.
- FIG. 7A is a schematic structural diagram of a neural network provided by an embodiment of the disclosure
- FIG. 7B is a flowchart of a neural network training method provided by an embodiment of the disclosure
- FIG. 7C is a schematic diagram of a neural network training method provided by an embodiment of the disclosure. This corresponds to the schematic block diagram of the training method shown in FIG. 7B for training the neural network shown in FIG. 7A.
- the neural network 300 includes an analysis network 310, a first sub-neural network 320, and a synthesis network 330.
- the analysis network 310 processes the input image to obtain the first feature image
- the first sub-neural network 320 performs at least one multi-scale cyclic sampling process on the first feature image to obtain the second feature image
- the synthesis network 330 performs the second feature image
- the image is processed to obtain an output image.
- the structure of the analysis network 310 can refer to the description of the analysis network in the aforementioned step S220, which is not limited in the present disclosure; the structure of the first sub-neural network 320 can refer to the aforementioned step S120 (that is, step S240) regarding the multi-scale loop
- the first sub-neural network may include but is not limited to the aforementioned first convolutional neural network, which is not limited in the present disclosure; for example, the synthesis network 330 may refer to the synthesis network in the aforementioned step S250 Description, this disclosure does not limit this.
- the input image and the output image can also refer to the description of the input image and the output image in the image processing method provided in the foregoing embodiment, which will not be repeated in this disclosure.
- the training method of the neural network includes step S410 to step S460.
- Step S410 Obtain training input images.
- the training input image may also include photos taken by the camera of a smart phone, a camera of a tablet computer, a camera of a personal computer, a lens of a digital camera, a surveillance camera, or a web camera. It may include images of people, images of animals and plants, or landscapes, etc., which is not limited in the present disclosure.
- the quality of the training input image is lower than the quality of photos taken by a real digital single-lens reflex camera, that is, the training input image is a low-quality image.
- the training input image may include 3 channel RGB images.
- Step S420 Use the analysis network to process the training input image to provide a first training feature image.
- the analysis network 310 may be a convolutional neural network including one of a convolutional layer, a residual network, and a dense network.
- the analysis network can convert 3 channel RGB images (ie, training input images) into multiple first training feature images, such as 64 first training feature images.
- RGB images ie, training input images
- first training feature images such as 64 first training feature images.
- Step S430 Use the first sub-neural network to perform multi-scale cyclic sampling processing on the first training feature image at least once to obtain a second training feature image.
- the multi-scale cyclic sampling process can be implemented as the multi-scale cyclic sampling process in any of the embodiments shown in FIGS. 4A-4D, but is not limited thereto.
- the multi-scale cyclic sampling processing in step S430 is implemented as the multi-scale cyclic sampling processing shown in FIG. 4A as an example for description.
- the multi-scale cyclic sampling process nests the first-level sampling process and the second-level sampling process.
- the input of the multi-scale cyclic sampling process (ie, the first training feature image) is used as the input of the first-level sampling process
- the output of the first-level sampling process is used as the output of the multi-scale cyclic sampling process (ie, the first Two training feature images).
- the size of the second training feature image may be the same as the size of the first training feature image.
- the first-level sampling process includes a first down-sampling process, a first up-sampling process, and a first residual link addition process that are sequentially executed.
- the first down-sampling process is performed based on the input of the first-level sampling process to obtain the first down-sampled output.
- the first down-sampling process can directly down-sample the input of the first-level sampling process to obtain the first down-sampling Output.
- the first up-sampling process is performed based on the first down-sampling output to perform up-sampling processing to obtain the first up-sampling output, for example, after the first down-sampling output is subjected to the second-level sampling process, the up-sampling process is performed to obtain the first up-sampling output That is, the first up-sampling process can indirectly perform up-sampling on the first down-sampling output.
- the first residual link addition process performs the first residual link addition on the input of the first level sampling process and the first upsampling output, and then uses the result of the first residual link addition as the output of the first level sampling process .
- the size of the output of the first up-sampling process is the same as the size of the input of the first-level sampling process (ie, the input of the first down-sampling process), which is added through the first residual link
- the size of the output of the first-level sampling process is the same as the size of the input of the first-level sampling process.
- the second-level sampling process is nested between the first down-sampling process and the first up-sampling process of the first-level sampling process, and the first down-sampling output is received as the input of the second-level sampling process. , Providing the output of the second-level sampling process as the input of the first up-sampling process, so that the first up-sampling process performs the up-sampling process based on the first down-sampling output.
- the second-level sampling process includes a second down-sampling process, a second up-sampling process, and a second residual link addition process that are sequentially executed.
- the second down-sampling process performs down-sampling based on the input of the second-level sampling process to obtain the second down-sampled output.
- the second down-sampling process can directly down-sample the input of the second-level sampling process to obtain the second down-sampling Output.
- the second up-sampling process performs up-sampling based on the second down-sampled output to obtain the second up-sampled output.
- the second up-sampling process may directly up-sample the second down-sampled output to obtain the second up-sampled output.
- the second residual link addition process performs a second residual link addition on the input of the second level sampling process and the second upsampling output, and then uses the result of the second residual link addition as the output of the second level sampling process .
- the size of the output of the second up-sampling process (ie, the second up-sampling output) is the same as the size of the input of the second-level sampling process (ie, the input of the second down-sampling process), so that it is added through the second residual link
- the size of the output of the second-level sampling process is the same as the size of the input of the second-level sampling process.
- the first sub-neural network 320 may be implemented as the aforementioned first convolutional neural network.
- the first sub-neural network 320 may include a nested first meta-network and a second meta-network, the first meta-network is used to perform the first-level sampling processing, and the second meta-network is used to perform the second-level sampling processing.
- the first meta-network may include a first sub-network and a second sub-network, the first sub-network is used to perform the first down-sampling process, and the second sub-network is used to perform the first up-sampling process.
- the second meta network is nested between the first sub network and the third sub network of the first meta network.
- the second meta-network may include a third sub-network and a fourth sub-network, the third sub-network is used to perform the second down-sampling process, and the fourth sub-network is used to perform the second up-sampling process.
- each of the first sub-network, the second sub-network, the third sub-network, and the fourth sub-network includes one of a convolutional layer, a residual network, a dense network, and the like.
- the first sub-network and the third sub-network may include one of a convolutional layer (down-sampling layer) with down-sampling function, a residual network, a dense network, etc.
- the second and fourth sub-networks may include One of the convolutional layer (upsampling layer), residual network, dense network, etc. of the upsampling function.
- the first sub-network and the third sub-network may have the same structure or different structures; the second sub-network and the fourth sub-network may have the same structure or different structures; There is no restriction on this publicly.
- the multi-scale cyclic sampling processing may further include: first down-sampling processing, first up-sampling processing, second down-sampling processing, and After the second up-sampling processing, the first down-sampling output, the first up-sampling output, the second down-sampling output, and the second up-sampling output are respectively subjected to instance standardization processing or layer standardization processing.
- first down-sampling output, the first up-sampling output, the second down-sampling output, and the second up-sampling output can use the same standardization processing method (instance standardization processing or layer standardization processing), or different Standardized processing method, this disclosure does not limit this.
- the first sub-network, the second sub-network, the third sub-network and the fourth sub-network also include an instance standardization layer or a layer standardization layer, respectively, the instance standardization layer is used to perform instance standardization processing, and the layer standardization layer is used to execute the layer.
- Standardized processing can perform instance standardization processing according to the aforementioned instance standardization formula
- the layer standardization layer can perform layer standardization processing according to the aforementioned layer standardization formula, which is not limited in the present disclosure.
- first sub-network, the second sub-network, the third sub-network, and the fourth sub-network may include the same standardization layer (example standardization layer or layer standardization layer), or can include different standardization layers, the present disclosure There is no restriction on this.
- step S430 for more implementation methods and more details of the multi-scale cyclic sampling processing in step S430, please refer to the foregoing step S120 (ie, step S240) and the multi-scale cyclic sampling processing in the embodiment shown in FIGS. 4A-4D. This disclosure will not repeat the description. It should also be noted that when the multi-scale cyclic sampling processing in step S430 is implemented in other forms, the first sub-neural network 320 should be changed accordingly to implement other forms of multi-scale cyclic sampling processing, which will not be discussed in this disclosure. Repeat.
- the number of second training feature images may be multiple, but is not limited thereto.
- Step S440 Use the synthetic network to process the second training feature image to obtain a training output image.
- the synthesis network 330 may be a convolutional neural network including one of a convolutional layer, a residual network, a dense network, and the like.
- the synthesis network may convert multiple second training feature images into training output images.
- the training output image may include 3 channel RGB images, and the present disclosure includes but is not limited to this.
- Step S450 Based on the training output image, calculate the loss value of the neural network through the loss function.
- the parameters of the neural network 300 include the parameters of the analysis network 310, the parameters of the first sub-neural network 320, and the parameters of the synthesis network 330.
- the initial parameter of the neural network 300 may be a random number, for example, the random number conforms to a Gaussian distribution, which is not limited in the embodiment of the present disclosure.
- the loss function of this embodiment can refer to the loss function in the literature provided by Andrey Ignatov et al.
- the loss function can include a color loss function, a texture loss function, and a content loss function; accordingly, the specific process of calculating the loss value of the parameters of the neural network 300 through the loss function can also refer to this Description in the literature.
- the embodiment of the present disclosure does not limit the specific form of the loss function, which includes but is not limited to the form of the loss function in the above-mentioned documents.
- Step S460 Correct the parameters of the neural network according to the loss value.
- the training process of the neural network 300 may also include an optimization function (not shown in FIG. 7C).
- the optimization function may calculate the error value of the parameters of the neural network 300 according to the loss value calculated by the loss function, and according to the error value The parameters of the neural network 300 are corrected.
- the optimization function may use a stochastic gradient descent (SGD) algorithm, a batch gradient descent (BGD) algorithm, etc., to calculate the error value of the parameters of the neural network 300.
- SGD stochastic gradient descent
- BGD batch gradient descent
- the training method of the neural network may further include: judging whether the training of the neural network satisfies a predetermined condition, if the predetermined condition is not met, repeating the above training process (ie, step S410 to step S460); if the predetermined condition is met, stopping the above During the training process, a trained neural network is obtained.
- the foregoing predetermined condition is that the loss values corresponding to two consecutive (or more) training output images no longer decrease significantly.
- the foregoing predetermined condition is that the number of training times or training cycles of the neural network reaches a predetermined number. This disclosure does not limit this.
- the training output image output by the trained neural network 300 retains the content of the training input image, but the quality of the training output image can be close to the quality of photos taken by a real digital single-lens reflex camera, that is, the training output image is high Quality image.
- the above-mentioned embodiments only schematically illustrate the training process of the neural network.
- the training process of each sample image may include multiple iterations to correct the parameters of the neural network.
- the training phase also includes fine-tune the parameters of the neural network to obtain more optimized parameters.
- the neural network training method provided by the embodiments of the present disclosure can train the neural network used in the image processing method of the embodiments of the present disclosure, and the neural network trained by the training method can perform image enhancement on low-quality input images Processing, by repeatedly sampling at multiple scales to obtain higher image fidelity, the quality of the output image can be greatly improved, and it is suitable for offline applications such as batch processing that require high image quality.
- FIG. 8 is a schematic block diagram of an image processing device provided by an embodiment of the present disclosure.
- the image processing apparatus 500 includes a memory 510 and a processor 520.
- the memory 510 is used to non-temporarily store computer readable instructions
- the processor 520 is used to run the computer readable instructions.
- the image processing method provided by the embodiments of the present disclosure is executed.
- the memory 510 and the processor 520 may directly or indirectly communicate with each other.
- components such as the memory 510 and the processor 520 may communicate through a network connection.
- the network may include a wireless network, a wired network, and/or any combination of a wireless network and a wired network.
- the network may include a local area network, the Internet, a telecommunication network, the Internet of Things (Internet of Things) based on the Internet and/or a telecommunication network, and/or any combination of the above networks, etc.
- the wired network may, for example, use twisted pair, coaxial cable, or optical fiber transmission for communication, and the wireless network may use, for example, a 3G/4G/5G mobile communication network, Bluetooth, Zigbee, or WiFi.
- the present disclosure does not limit the types and functions of the network here.
- the processor 520 may control other components in the image processing apparatus to perform desired functions.
- the processor 520 may be a central processing unit (CPU), a tensor processor (TPU), or a graphics processor GPU, and other devices with data processing capabilities and/or program execution capabilities.
- the central processing unit (CPU) can be an X86 or ARM architecture.
- the GPU can be directly integrated on the motherboard alone or built into the north bridge chip of the motherboard.
- the GPU can also be built into the central processing unit (CPU).
- the memory 510 may include any combination of one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and/or nonvolatile memory.
- Volatile memory may include random access memory (RAM) and/or cache memory (cache), for example.
- the non-volatile memory may include, for example, read only memory (ROM), hard disk, erasable programmable read only memory (EPROM), portable compact disk read only memory (CD-ROM), USB memory, flash memory, etc.
- one or more computer instructions may be stored in the memory 510, and the processor 520 may execute the computer instructions to implement various functions.
- the computer-readable storage medium may also store various application programs and various data, such as training input images, and various data used and/or generated by the application programs.
- one or more steps in the image processing method described above may be executed.
- one or more steps in the neural network training method described above may be executed.
- the image processing device provided by the above-mentioned embodiments of the present disclosure is exemplary rather than restrictive. According to actual application requirements, the image processing device may also include other conventional components or structures, for example, to realize image processing. For necessary functions of the processing device, those skilled in the art can set other conventional components or structures according to specific application scenarios, which are not limited in the embodiments of the present disclosure.
- FIG. 9 is a schematic diagram of a storage medium provided by an embodiment of the disclosure.
- the storage medium 600 non-transitory stores computer-readable instructions 601.
- the non-transitory computer-readable instructions 601 are executed by a computer (including a processor), any of the embodiments of the present disclosure can be executed. Instructions for the image processing method.
- one or more computer instructions may be stored on the storage medium 600.
- Some computer instructions stored on the storage medium 600 may be, for example, instructions for implementing one or more steps in the foregoing image processing method.
- the other computer instructions stored on the storage medium may be, for example, instructions for implementing one or more steps in the above-mentioned neural network training method.
- the storage medium may include the storage components of a tablet computer, the hard disk of a personal computer, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM), optical disk read only memory (CD -ROM), flash memory, or any combination of the above storage media, can also be other suitable storage media.
- RAM random access memory
- ROM read only memory
- EPROM erasable programmable read only memory
- CD -ROM optical disk read only memory
- flash memory or any combination of the above storage media, can also be other suitable storage media.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Biophysics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
Description
Claims (19)
- 一种图像处理方法,包括:接收第一特征图像;以及对所述第一特征图像进行至少一次多尺度循环采样处理;其中,所述多尺度循环采样处理包括嵌套的第一层级采样处理和第二层级采样处理,所述第一层级采样处理包括第一下采样处理、第一上采样处理和第一残差链接相加处理,其中,所述第一下采样处理基于第一层级采样处理的输入进行下采样处理得到第一下采样输出,所述第一上采样处理基于所述第一下采样输出进行上采样处理得到第一上采样输出,所述第一残差链接相加处理将所述第一层级采样处理的输入和所述第一上采样输出进行第一残差链接相加,然后将所述第一残差链接相加的结果作为第一层级采样处理的输出;所述第二层级采样处理嵌套在所述第一下采样处理和所述第一上采样处理之间,接收所述第一下采样输出作为第二层级采样处理的输入,提供第二层级采样处理的输出作为第一上采样处理的输入,以使得所述第一上采样处理基于所述第一下采样输出进行上采样处理;所述第二层级采样处理包括第二下采样处理、第二上采样处理和第二残差链接相加处理,其中,所述第二下采样处理基于所述第二层级采样处理的输入进行下采样处理得到第二下采样输出,所述第二上采样处理基于所述第二下采样输出进行上采样处理得到第二上采样输出,所述第二残差链接相加处理将所述第二层级采样处理的输入和所述第二上采样输出进行第二残差链接相加,然后将所述第二残差链接相加的结果作为所述第二层级采样处理的输出。
- 根据权利要求1所述的图像处理方法,其中,所述第一上采样处理的输出的尺寸与所述第一下采样处理的输入的尺寸相同;所述第二上采样处理的输出的尺寸与所述第二下采样处理的输入的尺寸相同。
- 根据权利要求1或2所述的图像处理方法,其中,所述多尺度循环采样处理还包括第三层级采样处理,所述第三层级采样处理嵌套在所述第二下采样处理和所述第二上采样处理之间,接收所述第二下采样输出作为第三层级采样处理的输入,提供第三层 级采样处理的输出作为第二上采样处理的输入,以使得所述第二上采样处理基于所述第二下采样输出进行上采样处理;所述第三层级采样处理包括第三下采样处理、第三上采样处理和第三残差链接相加处理,其中,所述第三下采样处理基于所述第三层级采样处理的输入进行下采样处理得到第三下采样输出,所述第三上采样处理基于所述第三下采样输出进行上采样处理得到第三上采样输出,所述第三残差链接相加处理将所述第三层级采样处理的输入和所述第三上采样输出进行第三残差链接相加,然后将所述第三残差链接相加的结果作为所述第三层级采样处理的输出。
- 根据权利要求1或2所述的图像处理方法,其中,所述多尺度循环采样处理包括多次依次执行的所述第二层级采样处理,第一次所述第二层级采样处理接收所述第一下采样输出作为第一次所述第二层级采样处理的输入,除第一次所述第二层级采样处理之外的每次所述第二层级采样处理接收前一次所述第二层级采样处理的输出作为本次所述第二层级采样处理的输入,最后一次所述第二层级采样处理的输出作为所述第一上采样处理的输入。
- 根据权利要求1-4任一项所述的图像处理方法,其中,所述至少一次多尺度循环采样处理包括多次依次执行的所述多尺度循环采样处理,每次所述多尺度循环采样处理的输入作为本次所述多尺度循环采样处理中的所述第一层级采样处理的输入,每次所述多尺度循环采样处理中的所述第一层级采样处理的输出作为本次所述多尺度循环采样处理的输出;第一次所述多尺度循环采样处理接收所述第一特征图像作为第一次所述多尺度循环采样处理的输入,除第一次所述多尺度循环采样处理之外的每次所述多尺度循环采样处理接收前一次所述多尺度循环采样处理的输出作为本次所述多尺度循环采样处理的输入,最后一次所述多尺度循环采样处理的输出作为所述至少一次多尺度循环采样处理的输出。
- 根据权利要求1-5任一项所述的图像处理方法,其中,所述多尺度循环采样处理还包括:在所述第一下采样处理、所述第一上采样处理、所述第二下采样处理和所述第二上采样处理之后,分别对所述第一下采样输出、所述第一上采样输出、 所述第二下采样输出和所述第二上采样输出进行实例标准化处理或层标准化处理。
- 根据权利要求1-6任一项所述的图像处理方法,还包括:使用第一卷积神经网络进行所述多尺度循环采样处理;其中,所述第一卷积神经网络包括:第一元网络,用于执行所述第一层级采样处理;第二元网络,用于执行所述第二层级采样处理。
- 根据权利要求7所述的图像处理方法,其中,所述第一元网络包括:第一子网络,用于执行所述第一下采样处理;第二子网络,用于执行所述第一上采样处理;所述第二元网络包括:第三子网络,用于执行所述第二下采样处理;第四子网络,用于执行所述第二上采样处理。
- 根据权利要求8所述的图像处理方法,其中,所述第一子网络、所述第二子网络、所述第三子网络和所述第四子网络中每一个包括卷积层、残差网络、密集网络之一。
- 根据权利要求9所述的图像处理方法,其中,所述第一子网络、所述第二子网络、所述第三子网络和所述第四子网络中的每一个都包括实例标准化层或层标准化层,所述实例标准化层用于执行实例标准化处理,所述层标准化层用于执行层标准化处理。
- 根据权利要求1-10任一项所述的图像处理方法,还包括:获取输入图像;使用分析网络将输入图像转换为所述第一特征图像;以及:使用合成网络将所述至少一次多尺度循环采样处理的输出转换为输出图像。
- 一种神经网络的训练方法,其中,所述神经网络包括:分析网络、第一子神经网络和合成网络,所述训练方法包括:获取训练输入图像;使用所述分析网络对所述训练输入图像进行处理以提供第一训练特征图像;使用所述第一子神经网络对所述第一训练特征图像进行至少一次多尺度循环采样处理以得到第二训练特征图像;使用所述合成网络对所述第二训练特征图像进行处理以得到训练输出图像;基于所述训练输出图像,通过损失函数计算所述神经网络的损失值;以及根据所述损失值对所述神经网络的参数进行修正;其中,所述多尺度循环采样处理包括嵌套的第一层级采样处理和第二层级采样处理,所述第一层级采样处理包括第一下采样处理、第一上采样处理和第一残差链接相加处理,其中,所述第一下采样处理基于第一层级采样处理的输入进行下采样处理得到第一下采样输出,所述第一上采样处理基于所述第一下采样输出进行上采样处理得到第一上采样输出,所述第一残差链接相加处理将所述第一层级采样处理的输入和所述第一上采样输出进行第一残差链接相加,然后将所述第一残差链接相加的结果作为第一层级采样处理的输出;所述第二层级采样处理嵌套在所述第一下采样处理和所述第一上采样处理之间,接收所述第一下采样输出作为第二层级采样处理的输入,提供第二层级采样处理的输出作为第一上采样处理的输入,以使得所述第一上采样处理基于所述第一下采样输出进行上采样处理;所述第二层级采样处理包括第二下采样处理、第二上采样处理和第二残差链接相加处理,其中,所述第二下采样处理基于所述第二层级采样处理的输入进行下采样处理得到第二下采样输出,所述第二上采样处理基于所述第二下采样输出进行上采样处理得到第二上采样输出,所述第二残差链接相加处理将所述第二层级采样处理的输入和所述第二上采样输出进行第二残差链接相加,然后将所述第二残差链接相加的结果作为所述第二层级采样处理的输出。
- 根据权利要求12所述的训练方法,其中,所述第一上采样处理的输出的尺寸与所述第一下采样处理的输入的尺寸相同;所述第二上采样处理的输出的尺寸与所述第二下采样处理的输入的尺寸相同。
- 根据权利要求12或13所述的训练方法,其中,所述多尺度循环采样 处理还包括第三层级采样处理,所述第三层级采样处理嵌套在所述第二下采样处理和所述第二上采样处理之间,接收所述第二下采样输出作为第三层级采样处理的输入,提供第三层级采样处理的输出作为第二上采样处理的输入,以使得所述第二上采样处理基于所述第二下采样输出进行上采样处理;所述第三层级采样处理包括第三下采样处理、第三上采样处理和第三残差链接相加处理,其中,所述第三下采样处理基于所述第三层级采样处理的输入进行下采样处理得到第三下采样输出,所述第三上采样处理基于所述第三下采样输出进行上采样处理得到第三上采样输出,所述第三残差链接相加处理将所述第三层级采样处理的输入和所述第三上采样输出进行第三残差链接相加,然后将所述第三残差链接相加的结果作为所述第三层级采样处理的输出。
- 根据权利要求12或13所述的训练方法,其中,所述多尺度循环采样处理包括多次依次执行的所述第二层级采样处理,第一次所述第二层级采样处理接收所述第一下采样输出作为第一次所述第二层级采样处理的输入,除第一次所述第二层级采样处理之外的每次所述第二层级采样处理接收前一次所述第二层级采样处理的输出作为本次所述第二层级采样处理的输入,最后一次所述第二层级采样处理的输出作为所述第一上采样处理的输入。
- 根据权利要求12-15任一项所述的训练方法,其中,所述至少一次多尺度循环采样处理包括多次依次执行的所述多尺度循环采样处理,每次所述多尺度循环采样处理的输入作为本次所述多尺度循环采样处理中的所述第一层级采样处理的输入,每次所述多尺度循环采样处理中的所述第一层级采样处理的输出作为本次所述多尺度循环采样处理的输出;第一次所述多尺度循环采样处理接收所述第一训练特征图像作为第一次所述多尺度循环采样处理的输入,除第一次所述多尺度循环采样处理之外的每次所述多尺度循环采样处理接收前一次所述多尺度循环采样处理的输出作为本次所述多尺度循环采样处理的输入,最后一次所述多尺度循环采样处理的输出作为所述至少一次多尺度循环采样处理的输出。
- 根据权利要求12-16任一项所述的训练方法,其中,所述多尺度循环 采样处理还包括:在所述第一下采样处理、所述第一上采样处理、所述第二下采样处理和所述第二上采样处理之后,分别对所述第一下采样输出、所述第一上采样输出、所述第二下采样输出和所述第二上采样输出进行实例标准化处理或层标准化处理。
- 一种图像处理装置,包括:存储器,用于非暂时性存储计算机可读指令;以及处理器,用于运行所述计算机可读指令,所述计算机可读指令被所述处理器运行时执行根据权利要求1-11任一项所述的图像处理方法或根据权利要求12-17任一项所述的神经网络的训练方法。
- 一种存储介质,非暂时性地存储计算机可读指令,当所述计算机可读指令由计算机执行时能够执行根据权利要求1-11任一项所述的图像处理方法的指令或根据权利要求12-17任一项所述的神经网络的训练方法的指令。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910209662.2 | 2019-03-19 | ||
CN201910209662.2A CN111724309B (zh) | 2019-03-19 | 2019-03-19 | 图像处理方法及装置、神经网络的训练方法、存储介质 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2020187029A1 true WO2020187029A1 (zh) | 2020-09-24 |
Family
ID=72519587
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2020/077763 WO2020187029A1 (zh) | 2019-03-19 | 2020-03-04 | 图像处理方法及装置、神经网络的训练方法、存储介质 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN111724309B (zh) |
WO (1) | WO2020187029A1 (zh) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114501012A (zh) * | 2021-12-31 | 2022-05-13 | 浙江大华技术股份有限公司 | 图像滤波、编解码方法以及相关设备 |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113538287B (zh) * | 2021-07-29 | 2024-03-29 | 广州安思创信息技术有限公司 | 视频增强网络训练方法、视频增强方法及相关装置 |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107767408A (zh) * | 2017-11-09 | 2018-03-06 | 京东方科技集团股份有限公司 | 图像处理方法、处理装置和处理设备 |
WO2018102748A1 (en) * | 2016-12-01 | 2018-06-07 | Berkeley Lights, Inc. | Automated detection and repositioning of micro-objects in microfluidic devices |
CN109360151A (zh) * | 2018-09-30 | 2019-02-19 | 京东方科技集团股份有限公司 | 图像处理方法及***、分辨率提升方法、可读存储介质 |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10303977B2 (en) * | 2016-06-28 | 2019-05-28 | Conduent Business Services, Llc | System and method for expanding and training convolutional neural networks for large size input images |
CN107730474B (zh) * | 2017-11-09 | 2022-02-22 | 京东方科技集团股份有限公司 | 图像处理方法、处理装置和处理设备 |
-
2019
- 2019-03-19 CN CN201910209662.2A patent/CN111724309B/zh active Active
-
2020
- 2020-03-04 WO PCT/CN2020/077763 patent/WO2020187029A1/zh active Application Filing
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2018102748A1 (en) * | 2016-12-01 | 2018-06-07 | Berkeley Lights, Inc. | Automated detection and repositioning of micro-objects in microfluidic devices |
CN107767408A (zh) * | 2017-11-09 | 2018-03-06 | 京东方科技集团股份有限公司 | 图像处理方法、处理装置和处理设备 |
CN109360151A (zh) * | 2018-09-30 | 2019-02-19 | 京东方科技集团股份有限公司 | 图像处理方法及***、分辨率提升方法、可读存储介质 |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114501012A (zh) * | 2021-12-31 | 2022-05-13 | 浙江大华技术股份有限公司 | 图像滤波、编解码方法以及相关设备 |
CN114501012B (zh) * | 2021-12-31 | 2024-06-11 | 浙江大华技术股份有限公司 | 图像滤波、编解码方法以及相关设备 |
Also Published As
Publication number | Publication date |
---|---|
CN111724309A (zh) | 2020-09-29 |
CN111724309B (zh) | 2023-07-14 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2021073493A1 (zh) | 图像处理方法及装置、神经网络的训练方法、合并神经网络模型的图像处理方法、合并神经网络模型的构建方法、神经网络处理器及存储介质 | |
WO2020239026A1 (zh) | 图像处理方法及装置、神经网络的训练方法、存储介质 | |
WO2020200030A1 (zh) | 神经网络的训练方法、图像处理方法、图像处理装置和存储介质 | |
US11461639B2 (en) | Image processing method, image processing device, and training method of neural network | |
US11551333B2 (en) | Image reconstruction method and device | |
WO2020177651A1 (zh) | 图像分割方法和图像处理装置 | |
US10678508B2 (en) | Accelerated quantized multiply-and-add operations | |
CN111402130B (zh) | 数据处理方法和数据处理装置 | |
EP3923233A1 (en) | Image denoising method and apparatus | |
CN111914997B (zh) | 训练神经网络的方法、图像处理方法及装置 | |
CN112446834A (zh) | 图像增强方法和装置 | |
CN113011562A (zh) | 一种模型训练方法及装置 | |
WO2021169160A1 (zh) | 图像归一化处理方法及装置、存储介质 | |
CN113095470A (zh) | 神经网络的训练方法、图像处理方法及装置、存储介质 | |
WO2020187029A1 (zh) | 图像处理方法及装置、神经网络的训练方法、存储介质 | |
CN109754357B (zh) | 图像处理方法、处理装置以及处理设备 | |
CN113096023B (zh) | 神经网络的训练方法、图像处理方法及装置、存储介质 | |
CN113076966B (zh) | 图像处理方法及装置、神经网络的训练方法、存储介质 | |
WO2023029559A1 (zh) | 一种数据处理方法以及装置 | |
WO2022133814A1 (en) | Omni-scale convolution for convolutional neural networks | |
CN113256556A (zh) | 一种图像选择方法及装置 | |
WO2022183325A1 (zh) | 视频块处理方法及装置、神经网络的训练方法和存储介质 | |
US20240233074A9 (en) | Image processing method and device, training method of neural network, image processing method based on combined neural network model, constructing method of combined neural network model, neural network processor, and storage medium | |
WO2023028866A1 (zh) | 图像处理方法、装置和车辆 | |
CN117392491A (zh) | 一种数据处理方法及其装置 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 20773582 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 20773582 Country of ref document: EP Kind code of ref document: A1 |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 07.02.2022) |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 20773582 Country of ref document: EP Kind code of ref document: A1 |