CN110400275B - Color correction method based on full convolution neural network and characteristic pyramid - Google Patents
Color correction method based on full convolution neural network and characteristic pyramid Download PDFInfo
- Publication number
- CN110400275B CN110400275B CN201910659156.3A CN201910659156A CN110400275B CN 110400275 B CN110400275 B CN 110400275B CN 201910659156 A CN201910659156 A CN 201910659156A CN 110400275 B CN110400275 B CN 110400275B
- Authority
- CN
- China
- Prior art keywords
- neural network
- image
- layer
- light source
- source information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 110
- 238000012937 correction Methods 0.000 title claims abstract description 67
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000012545 processing Methods 0.000 claims abstract description 29
- 238000012549 training Methods 0.000 claims abstract description 20
- 238000004364 calculation method Methods 0.000 claims description 7
- 238000010606 normalization Methods 0.000 claims description 6
- 238000011176 pooling Methods 0.000 claims description 4
- 230000006870 function Effects 0.000 description 44
- 230000004913 activation Effects 0.000 description 36
- 230000000694 effects Effects 0.000 description 5
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 238000010191 image analysis Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20016—Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a color correction method based on a full convolution neural network and a characteristic pyramid, which relates to the technical field of image color correction, and is characterized in that color correction, cutting and data enhancement processing are sequentially carried out on an image to obtain corresponding color cast pictures, and a training data set is formed by a plurality of color cast pictures; training a preset full convolution neural network and a characteristic pyramid model by utilizing a training data set, optimizing and adjusting network parameters of the preset full convolution neural network and the characteristic pyramid model based on a preset loss function and an optimizer until the network converges, and outputting the trained full convolution neural network and the trained characteristic pyramid model; the method comprises the steps of estimating light source information of an image to be corrected by using a trained model, and performing color correction on the image to be corrected according to the estimated light source information.
Description
Technical Field
The invention relates to the technical field of image color correction, in particular to a color correction method based on a full convolution neural network and a characteristic pyramid.
Background
The collection and reproduction of real colors have important value in the fields of medicine, art and the like, and the color information of images is an important basis for analyzing certain professional images. The color presented on the surface of the object is closely related to various links such as light source characteristics, illumination conditions, acquisition equipment, display equipment, printing equipment and the like. Color correction is a key technology for color reproduction and color consistency presentation, and is currently applied in a plurality of image processing fields such as medical images, mural images, license images and the like. The research on the color correction technology capable of truly reflecting the color of the observed object has important significance.
Color correction is a key preprocessing step in image processing, and may cause image color distortion due to different illumination, different viewing angles and different acquisition devices, so many methods for image color correction are proposed in succession, and in the case of unknown light source information, the methods for color correction can be classified into the following three categories:
1. mapping-based color correction methods, such as matrix methods, polynomial regression methods, look-up tables, and the like;
2. color correction methods based on image analysis, such as gray world color correction, maximum brightness color correction, and the like;
3. a color correction method based on a full convolution neural network;
however, the above color correction method has the following problems:
1. mapping-based color correction methods require a large amount of standard patch data as a tool;
2. the color correction method based on image analysis is simple in calculation, but has certain limitation, and sometimes the real color of an object cannot be correctly reproduced;
3. the color correction method based on the full convolution neural network trains a limited neural network structure, so that light source information can not be accurately estimated sometimes, and the color correction effect is not ideal.
Disclosure of Invention
The invention aims to: in order to solve the problem that the color correction effect is not ideal because the training of the existing color correction method based on the full convolution neural network is limited by the neural network structure and sometimes the light source information cannot be accurately estimated, the invention provides the color correction method based on the full convolution neural network and the characteristic pyramid.
The invention specifically adopts the following technical scheme for realizing the purpose:
a color correction method based on a full convolution neural network and a characteristic pyramid is characterized by comprising the following steps:
carrying out color correction, cutting and data enhancement processing on the image in sequence to obtain corresponding color cast pictures, and forming a training data set by a plurality of color cast pictures;
constructing a preset full convolution neural network and a characteristic pyramid model for estimating image light source information, training the preset full convolution neural network and the characteristic pyramid model by utilizing a training data set, optimizing and adjusting network parameters of the preset full convolution neural network and the characteristic pyramid model based on a preset loss function and an optimizer until the network converges, and outputting the trained full convolution neural network and the trained characteristic pyramid model;
estimating light source information of the image to be corrected by using the trained full-convolution neural network and the characteristic pyramid model, and performing color correction on the image to be corrected by using the estimated light source information;
the preset full convolution neural network and the characteristic pyramid model are as follows:
the input layer is connected with a first layer of neural network comprising two convolution layers, and the output end of the first layer of neural network is connected with the input end of a second layer of neural network;
the second layer of neural network comprises two branches and an adder A, wherein one branch comprises a convolution layer, the other branch comprises two depth separable convolution layers, the output ends of the two branches are connected with the input end of the adder A, and the output end of the adder A is connected with the input end of the third layer of neural network;
the third layer of neural network comprises two branches and an adder B, wherein one branch comprises a convolution layer, the other branch comprises two depth separable convolution layers, the output ends of the two branches are connected with the input end of the adder B, and the output end of the adder B is connected with the input end of the fourth layer of neural network;
the fourth layer of neural network comprises two branches and an adder C, wherein one branch comprises a convolutional layer, the other branch comprises two depth separable convolutional layers, the output ends of the two branches are connected with the input end of the adder C, and the output end of the adder C is connected with the input end of the fifth layer of neural network;
the fifth layer of neural network comprises five branches and a splicer, wherein the first branch comprises a global pooling unit and a convolutional layer, the second branch comprises a convolutional layer, the third branch comprises a depth-separable convolutional layer, the fourth branch comprises a depth-separable convolutional layer, the fifth branch comprises a depth-separable convolutional layer, the output ends of the five branches are all connected with the input end of the splicer, and the output end of the splicer is connected with the input end of the sixth layer of neural network;
the sixth layer of neural network comprises two convolutional layers, and the output end of the sixth layer of neural network is connected with the seventh layer of neural network comprising a normalization model.
Further, the color correction of the image specifically includes: and converting the image into an 8-bit image data format, and processing the 8-bit image data by using the real light source information ground _ truth to obtain a standard image.
Further, the processing of the 8-bit image data by using the real light source information group _ route includes:
reading the 8-bit image data into an RGB format [ R, G, B ], and setting the corresponding real light source information group _ judge to [ real _ R, real _ G, real _ B ], then:
Gain_R＝max(ground_truth)/real_R
Gain_G＝max(ground_truth)/real_G
Gain_B＝max(ground_truth)/real_B
R′＝min(R*Gain_R,255)
G′＝min(G*Gain_G,255)
B′＝min(B*Gain_B,255)
image＝[R′,G′,B′]
wherein, Gain _ R, Gain _ G, Gain _ B is Gain coefficients of R, G, B three channels respectively, and R ', G ', B ' are corresponding values of R, G, B three channels of the image respectively;
carrying out color correction on the image by utilizing gamma correction to obtain an image', wherein the calculation formula is as follows:
wherein gamma is a gamma correction coefficient;
and converting the image' image into a unit8 format to obtain a standard picture.
Further, the image is cut specifically as follows: randomly rotating the standard picture within the range of [ -30 degrees, 30 degrees ], and then cutting the rotated standard picture to obtain a cut picture.
Further, the data enhancement processing on the image specifically includes: and randomly generating false light source information fake _ group _ truth, and performing data enhancement on the cut picture by using the generated false light source information fake _ group _ truth to obtain a color cast picture.
Further, randomly generating false light source information fake _ ground _ truth includes:
randomly generating fake _ group _ truth 'with the format of [ fakeR', fakeG ', fakeB' ], then:
max_fake＝max(fake_ground_truth′)
wherein max _ fake is the maximum value in fake _ group _ truth ', and fgtR', fgtG 'and fgtB' are gain coefficients corresponding to R, G, B three channels of the fake light source information fake _ group _ truth, respectively.
Further, performing data enhancement on the cut picture by using the generated false light source information fake _ ground _ truth, including:
and simulating and generating a picture image _ g ' by using a gamma correction inverse function, wherein the cut picture is st _ image and is in a format of [ stR ', stG ', stB ' ], and the picture image _ g ' is as follows:
if the format of the picture image _ g 'is [ Rg', Gg ', Bg' ], then [ Rf, Gf, Bf ] of the color cast picture fake _ image is:
Rf＝min(Rg′/fgtR′,255)
Gf＝min(Gg′/fgtG′,255)
Bf＝min(Bg′/fgtB′,255)
and converting Rf, Gf and Bf into a unit8 format to obtain a color cast picture fake _ image.
Further, the Loss function is based on cosine similarity, and the calculation formula of the Loss function Loss is as follows:
and g _ pre is image light source information estimated by a preset full convolution neural network and a characteristic pyramid model.
Furthermore, a plurality of subgraphs with the same size are intercepted at equal intervals from the image to be corrected, the light source information of each subgraph is respectively estimated by using a full convolution neural network and a characteristic pyramid model, the median of the light source information corresponding to each color channel is taken from all the light source information as the final estimated light source information, and the image to be corrected is subjected to color correction based on the final estimated light source information to obtain a corrected image.
The invention has the following beneficial effects:
1. the method estimates the light source information through the full convolution neural network and the multi-scale characteristic pyramid, performs color correction on the image to be corrected by utilizing gamma correction based on the estimated light source information, and has good correction effect, good correction effect on the single-light-source color cast picture and good universality.
2. The fifth-layer neural network of the preset full-convolution neural network and the characteristic pyramid model adopts five branches, and multi-scale convolution extraction is carried out on the characteristics by utilizing the five branches, so that the light source information of the picture can be accurately estimated, and the later correction effect is ensured.
Drawings
FIG. 1 is a schematic flow chart of the method of example 1 of the present invention.
Fig. 2 is a schematic structural diagram of a preset full convolution neural network and a feature pyramid model in embodiment 3 of the present invention.
Fig. 3 is a schematic diagram of an image to be corrected according to embodiment 3 of the present invention.
Fig. 4 is a schematic diagram of a correction picture obtained in embodiment 3 of the present invention.
Detailed Description
For a better understanding of the present invention by those skilled in the art, the present invention will be described in further detail below with reference to the accompanying drawings and the following examples.
Example 1
As shown in fig. 1, the present embodiment provides a color correction method based on a full convolution neural network and a feature pyramid, including the following steps:
carrying out color correction, cutting and data enhancement processing on the image in sequence to obtain corresponding color cast pictures, and forming a training data set by a plurality of color cast pictures;
constructing a preset full convolution neural network and a characteristic pyramid model for estimating image light source information, training the preset full convolution neural network and the characteristic pyramid model by utilizing a training data set, optimizing and adjusting network parameters of the preset full convolution neural network and the characteristic pyramid model based on a preset loss function and an optimizer until the network converges, and outputting the trained full convolution neural network and the trained characteristic pyramid model;
and estimating light source information of the image to be corrected by using the trained full-convolution neural network and the characteristic pyramid model, and performing color correction on the image to be corrected by using the estimated light source information.
Example 2
The embodiment is further optimized based on embodiment 1, and specifically includes:
the images used for training in this example are from the public Dataset-Shi's Re-processing of Gehler's Raw Dataset;
the color correction of the image specifically comprises the following steps: converting an image in the public data set from a RAW format into an 8-bit image data format, processing the 8-bit image data by utilizing real light source information ground _ route to obtain a standard picture, and processing the 8-bit image data by utilizing the real light source information ground _ route, wherein the processing method comprises the following steps:
reading the 8-bit image data into an RGB format [ R, G, B ], and setting the corresponding real light source information group _ judge to [ real _ R, real _ G, real _ B ], then:
Gain_R＝max(ground_truth)/real_R
Gain_G＝max(ground_truth)/real_G
Gain_B＝max(ground_truth)/real_B
R′＝min(R*Gain_R,255)
G′＝min(G*Gain_G,255)
B′＝min(B*Gain_B,255)
image＝[R′,G′,B′]
wherein, Gain _ R, Gain _ G, Gain _ B is Gain coefficients of R, G, B three channels respectively, and R ', G ', B ' are corresponding values of R, G, B three channels of the image respectively;
carrying out color correction on the image by utilizing gamma correction to obtain an image', wherein the calculation formula is as follows:
wherein gamma is a gamma correction coefficient, and in this embodiment, gamma is 1/2.2;
then, converting the image' image into a unit8 format to obtain a standard picture;
then randomly rotating the standard picture within the range of [ -30 degrees, 30 degrees ], and cutting the rotated standard picture to obtain a cut picture with the size of 512 x 512;
then randomly generating false light source information fake _ ground _ route, and performing data enhancement on the cut picture by using the generated false light source information fake _ ground _ route to obtain a color cast picture;
randomly generating false light source information fake _ ground _ truth, comprising:
randomly generating fake _ group _ truth ' with a format of [ fakeR ', fakeG ', fakeB ' ], wherein the fakeR ', fakeG ', and fakeB ' are random values in a range of 1-2.5, and then:
max_fake＝max(fake_ground_truth′)
wherein max _ fake is the maximum value in fake _ group _ truth ', and fgtR ', fgtG ' are gain coefficients corresponding to R, G, B three channels of the false light source information fake _ group _ truth respectively;
performing data enhancement on the cut picture by using the generated false light source information fake _ ground _ truth, wherein the data enhancement comprises the following steps:
and simulating and generating a picture image _ g ' by using a gamma correction inverse function, wherein the cut picture is st _ image and is in a format of [ stR ', stG ', stB ' ], and the picture image _ g ' is as follows:
in the formula, the gamma is 1/2.2;
if the format of the picture image _ g 'is [ Rg', Gg ', Bg' ], then [ Rf, Gf, Bf ] of the color cast picture fake _ image is:
Rf＝min(Rg′/fgtR′,255)
Gf＝min(Gg′/fgtG′,255)
Bf＝min(Bg′/fgtB′,255)
converting Rf, Gf and Bf into a unit8 format to obtain a color cast picture fake _ image, and turning the color cast picture fake _ image vertically and horizontally under the condition of false light source information fake _ group _ truth to obtain color cast pictures fake _ image with different angles in order to expand training data; and in order to facilitate model training, normalization processing can be performed on the color cast picture fake _ image, namely dividing the RGB value of the color cast picture fake _ image by 255 to scale the RGB value to the range from 0 to 1, and forming a training data set by the normalized color cast picture fake _ image so as to train a preset full convolution neural network and a characteristic pyramid model.
Example 3
The embodiment is further optimized based on the embodiment 2, and specifically includes:
as shown in fig. 2, the constructed preset full-convolution neural network and the feature pyramid model have seven layers, and the specific structure is as follows:
the input layer is connected with a first layer of neural network comprising two convolutional layers, wherein the first convolutional layer is provided with 32 convolutional kernels, the size of each convolutional kernel is 3 x 3, the step length is 2, and the activation function is Relu; the second convolutional layer has 64 convolutional kernels, each convolutional kernel size is 3 x 3, step size is 1, activation function is Relu; the output end of the second convolution layer is connected with the input end of the second layer of neural network;
the second layer of neural network comprises two branches and an adder a, wherein one branch comprises a convolutional layer with 128 convolutional kernels, each convolutional kernel having a size of 1 x 1, step size of 2, and activation function of Relu, and the other branch comprises two depth separable convolutional layers, wherein the first depth separable convolutional layer has 128 convolutional kernels, each convolutional kernel having a size of 3 x 3, step size of 1, and activation function of Relu; the second depth separable convolutional layer has 128 convolutional kernels, each convolutional kernel size is 3 x 3, step size is 2, activation function is Relu; the output ends of the two branches are connected with the input end of the adder A, and the output end of the adder A is connected with the input end of the third layer of neural network;
the third layer of the neural network comprises two branches and an adder B, wherein one branch comprises a convolutional layer with 256 convolutional kernels, each convolutional kernel having a size of 1 x 1, a step size of 2 and an activation function of Relu, and the other branch comprises two depth separable convolutional layers, wherein the first depth separable convolutional layer has 256 convolutional kernels, each convolutional kernel having a size of 3 x 3, a step size of 1 and an activation function of Relu; the second depth separable convolutional layer has 256 convolutional kernels, each convolutional kernel size is 3 x 3, step size is 2, activation function is Relu; the output ends of the two branches are connected with the input end of the adder B, and the output end of the adder B is connected with the input end of the fourth-layer neural network;
the fourth layer of neural network comprises two branches and an adder C, wherein one branch comprises a convolutional layer with 256 convolutional kernels, each convolutional kernel having a size of 1 x 1, a step size of 2, and an activation function of Relu, and the other branch comprises two depth separable convolutional layers, wherein the first depth separable convolutional layer has 256 convolutional kernels, each convolutional kernel having a size of 3 x 3, a step size of 1, and an activation function of Relu; the second depth separable convolutional layer has 256 convolutional kernels, each convolutional kernel size is 3 x 3, step size is 2, activation function is Relu; the output ends of the two branches are connected with the input end of the adder C, and the output end of the adder C is connected with the input end of the fifth-layer neural network;
the fifth layer neural network comprises five branches and a splicer, wherein the first branch comprises global pooling units and a convolutional layer having 32 convolutional kernels, each convolutional kernel having a size of 1 × 1 and an activation function of Relu, the second branch comprises a convolutional layer having 32 convolutional kernels, each convolutional kernel having a size of 1 × 1 and an activation function of Relu, the third branch comprises a depth separable convolutional layer having 32 convolutional kernels, each convolutional kernel having a size of 3 × 3, a step size of 1, an expansion rate of 6 and an activation function of Relu, the fourth branch comprises a depth separable convolutional layer having 32 convolutional kernels, each convolutional kernel having a size of 3 × 3, a step size of 1, an expansion rate of 12 and an activation function of Relu, and the fifth branch comprises a depth separable convolutional layer having 32 convolutional kernels, each convolutional kernel having a size of 3 × 3, a step size of 1, an expansion rate of 18, and a splicer, The activation function is a depth separable convolution layer of Relu, the output ends of the five branches are connected with the input end of the splicer, and the output end of the splicer is connected with the input end of the sixth layer of neural network;
the sixth layer of neural network comprises two convolutional layers, wherein the first convolutional layer has 64 convolutional kernels, each convolutional kernel has the size of 3 x 3, the step size is 2, and the activation function is Relu; the second convolution layer has 3 convolution kernels, each convolution kernel size is 3 x 3, step size is 1, activation function is Relu; and the output end of the sixth layer of neural network is connected with a seventh layer of neural network comprising a normalization model.
Training a preset full convolution neural network and a characteristic pyramid model by utilizing a normalized color cast picture fake _ image in a training data set, wherein the process is as follows:
inputting data with a normalized RGB value of the color cast picture fake image into a model input layer, entering a first layer of neural network, performing first convolutional layer processing with 32 convolutional kernels, the size of each convolutional kernel being 3 x 3, the step length being 2 and the activation function being Relu, performing second convolutional layer processing with 64 convolutional kernels, the size of each convolutional kernel being 3 x 3, the step length being 1 and the activation function being Relu to obtain the output characteristics of the first layer, and inputting the output characteristics into a second layer of neural network;
then, the output characteristics of the first branch are processed by two branches of a second layer of neural network, the output characteristics of the first branch are obtained by processing the output characteristics of the first layer by a convolutional layer with 128 convolutional kernels, the size of each convolutional kernel is 1 x 1, the step length is 2 and the activation function is Relu, and the output characteristics of the second branch are processed by a deep separable convolutional layer with 128 convolutional kernels, the size of each convolutional kernel is 3 x 3, the step length is 1 and the activation function is Relu; then, processing by a depth separable convolutional layer with 128 convolutional kernels, each convolutional kernel with the size of 3 x 3, the step length of 2 and the activation function of Relu to obtain the output characteristic of a second branch; the output characteristics of the two branches are added through an adder A to obtain the output characteristics of the second layer, and the output characteristics are input into the third layer of neural network;
processing by two branches of a third layer of neural network, wherein the output characteristic of the first branch is processed by a convolutional layer with 256 convolutional kernels, the size of each convolutional kernel is 1 x 1, the step length is 2 and the activation function is Relu to obtain the output characteristic of the first branch, and the output characteristic of the second branch is processed by a deep separable convolutional layer with 256 convolutional kernels, the size of each convolutional kernel is 3 x 3, the step length is 1 and the activation function is Relu to obtain the output characteristic of the second layer; then processing the two branches of the depth separable convolution layer with 256 convolution kernels, the size of each convolution kernel is 3 x 3, the step length is 2 and the activation function is Relu to obtain the output characteristic of the second branch; the output characteristics of the two branches are added through an adder B to obtain the output characteristics of a third layer, and the output characteristics are input into a fourth layer neural network;
processing by two branches of a fourth layer of neural network, wherein the output characteristics of the third layer of the first branch are processed by 256 convolution kernels, the size of each convolution kernel is 1 x 1, the step length is 2 and the convolution layer with the Relu as an activation function to obtain the output characteristics of the first branch, and the output characteristics of the third layer of the second branch are processed by 256 convolution kernels, the size of each convolution kernel is 3 x 3, the step length is 1 and the depth separable convolution layer with the Relu as the activation function; then processing the two branches of the depth separable convolution layer with 256 convolution kernels, the size of each convolution kernel is 3 x 3, the step length is 2 and the activation function is Relu to obtain the output characteristic of the second branch; the output characteristics of the two branches are added through an adder C to obtain the output characteristics of a fourth layer, and the output characteristics are input into a fifth layer neural network;
processing by five branches of a fifth layer neural network, wherein the output characteristics of the fourth layer are subjected to global pooling processing by the first branch, then are subjected to convolutional layer processing by 32 convolutional kernels, the size of each convolutional kernel is 1 x 1, and the activation function is Relu, and then the output characteristics of the first branch are obtained by up-sampling; the second branch processes the output characteristics of the fourth layer through the convolution layers with 32 convolution kernels, the size of each convolution kernel is 1 x 1, and the activation function is Relu to obtain the output characteristics of the second branch; the third branch processes the output characteristic of the fourth layer through 32 convolution kernels, wherein the size of each convolution kernel is 3 x 3, the step size is 1, the expansion rate is 6 and the activation function is Relu depth separable convolution layer to obtain the output characteristic of the third branch; the fourth branch processes the output characteristic of the fourth layer through 32 convolution kernels, each convolution kernel is 3 x 3 in size, the step length is 1, the expansion rate is 12 and the depth separable convolution layer with the Relu as an activation function to obtain the output characteristic of the fourth branch; the fifth branch processes the output characteristics of the fourth layer by 32 convolution kernels, each convolution kernel is 3 x 3 in size, the step length is 1, the expansion rate is 18 and the depth separable convolution layer with the Relu as an activation function to obtain the output characteristics of the fifth branch, then the output characteristics of the five branches of the layer are spliced by a splicer to obtain the output characteristics of the fifth layer, and the output characteristics are input into a sixth layer of neural network;
performing convolutional layer processing by 64 convolutional kernels, wherein the size of each convolutional kernel is 3 x 3, the step size is 2, and the activation function is Relu; then, processing the convolution layers with 3 convolution kernels, each convolution kernel being 3 × 3 in size, the step length being 1 and the activation function being Relu to obtain the output characteristics of a sixth layer, and inputting the output characteristics into a seventh layer of neural network;
the output end of the sixth layer of neural network is connected with a seventh layer of neural network comprising a normalization model, the seventh layer of neural network firstly normalizes the output characteristics of the sixth layer in the 3 rd dimension, then sums the output characteristics of the sixth layer after the L2 norm normalization in the 1 st dimension and the 2 nd dimension, and finally normalizes the obtained sum to obtain the output characteristics of the seventh layer, namely the estimated image light source information;
then, calculating Loss by using cosine similarity, wherein the calculation formula of a Loss function Loss is as follows:
the ground _ truth is real light source information, and the g _ pre is image light source information estimated by a preset full convolution neural network and a characteristic pyramid model;
in the embodiment, the optimizer selects Adam, and the learning rate of Adam is 1 e-5;
because the training of the preset full convolution neural network and the characteristic pyramid model is a back propagation process, the network parameters are optimized and adjusted by utilizing the optimizer Adam to calculate the gradient through Loss function Loss back propagation until the network converges, and the trained full convolution neural network and the trained characteristic pyramid are output;
intercepting 121 sub-images with the size of 512 × 512 at equal intervals from the image to be corrected as shown in fig. 3, respectively estimating light source information of each sub-image by using a full convolution neural network and a feature pyramid model, and taking a median of the light source information corresponding to each color channel from all the light source information as final estimated light source information, so as to obtain the final estimated light source information as follows:
[0.6395844, 0.64725494, 0.41653872], color correction is performed on the image to be corrected by the gamma correction method described in embodiment 2 based on the final estimated illuminant information, and the corrected picture shown in FIG. 4 is finally obtained.
The above description is only a preferred embodiment of the present invention, and not intended to limit the present invention, the scope of the present invention is defined by the appended claims, and all structural changes that can be made by using the contents of the description and the drawings of the present invention are intended to be embraced therein.
Claims (9)
1. A color correction method based on a full convolution neural network and a characteristic pyramid is characterized by comprising the following steps:
carrying out color correction, cutting and data enhancement processing on the image in sequence to obtain corresponding color cast pictures, and forming a training data set by a plurality of color cast pictures;
constructing a preset full convolution neural network and a characteristic pyramid model for estimating image light source information, training the preset full convolution neural network and the characteristic pyramid model by utilizing a training data set, optimizing and adjusting network parameters of the preset full convolution neural network and the characteristic pyramid model based on a preset loss function and an optimizer until the network converges, and outputting the trained full convolution neural network and the trained characteristic pyramid model;
estimating light source information of the image to be corrected by using the trained full-convolution neural network and the characteristic pyramid model, and performing color correction on the image to be corrected by using the estimated light source information;
the preset full convolution neural network and the characteristic pyramid model are as follows:
the input layer is connected with a first layer of neural network comprising two convolution layers, and the output end of the first layer of neural network is connected with the input end of a second layer of neural network;
the second layer of neural network comprises two branches and an adder A, wherein one branch comprises a convolution layer, the other branch comprises two depth separable convolution layers, the output ends of the two branches are connected with the input end of the adder A, and the output end of the adder A is connected with the input end of the third layer of neural network;
the third layer of neural network comprises two branches and an adder B, wherein one branch comprises a convolution layer, the other branch comprises two depth separable convolution layers, the output ends of the two branches are connected with the input end of the adder B, and the output end of the adder B is connected with the input end of the fourth layer of neural network;
the fourth layer of neural network comprises two branches and an adder C, wherein one branch comprises a convolutional layer, the other branch comprises two depth separable convolutional layers, the output ends of the two branches are connected with the input end of the adder C, and the output end of the adder C is connected with the input end of the fifth layer of neural network;
the fifth layer of neural network comprises five branches and a splicer, wherein the first branch comprises a global pooling unit and a convolutional layer, the second branch comprises a convolutional layer, the third branch comprises a depth-separable convolutional layer, the fourth branch comprises a depth-separable convolutional layer, the fifth branch comprises a depth-separable convolutional layer, the output ends of the five branches are all connected with the input end of the splicer, and the output end of the splicer is connected with the input end of the sixth layer of neural network;
the sixth layer of neural network comprises two convolutional layers, and the output end of the sixth layer of neural network is connected with the seventh layer of neural network comprising a normalization model.
2. The color correction method based on the full convolution neural network and the feature pyramid as claimed in claim 1, wherein the color correction of the image specifically comprises: and converting the image into an 8-bit image data format, and processing the 8-bit image data by using the real light source information ground _ truth to obtain a standard image.
3. The method of claim 2, wherein the processing of 8-bit image data with real light source information group _ route comprises:
reading the 8-bit image data into an RGB format [ R, G, B ], and setting the corresponding real light source information group _ judge to [ real _ R, real _ G, real _ B ], then:
Gain_R＝max(ground_truth)/real_R
Gain_G＝max(ground_truth)/real_G
Gain_B＝max(ground_truth)/real_B
R′＝min(R*Gain_R,255)
G′＝min(G*Gain_G,255)
B′＝min(B*Gain_B,255)
image＝[R′,G′,B′]
wherein, Gain _ R, Gain _ G, Gain _ B is Gain coefficients of R, G, B three channels respectively, and R ', G ', B ' are corresponding values of R, G, B three channels of the image respectively;
carrying out color correction on the image by utilizing gamma correction to obtain an image', wherein the calculation formula is as follows:
wherein gamma is a gamma correction coefficient;
and converting the image' image into a unit8 format to obtain a standard picture.
4. The color correction method based on the full convolution neural network and the feature pyramid as claimed in claim 2 or 3, wherein the cropping of the image is specifically as follows: randomly rotating the standard picture within the range of [ -30 degrees, 30 degrees ], and then cutting the rotated standard picture to obtain a cut picture.
5. The color correction method based on the full convolution neural network and the feature pyramid as claimed in claim 4, wherein the data enhancement processing on the image specifically comprises: and randomly generating false light source information fake _ group _ truth, and performing data enhancement on the cut picture by using the generated false light source information fake _ group _ truth to obtain a color cast picture.
6. The color correction method based on the full convolution neural network and the feature pyramid as claimed in claim 5, wherein randomly generating false light source information fake _ ground _ truth comprises:
randomly generating fake _ group _ truth 'with the format of [ fakeR', fakeG ', fakeB' ], then:
max_fake＝max(fake_ground_truth′)
wherein max _ fake is the maximum value in fake _ group _ truth ', and fgtR', fgtG 'and fgtB' are gain coefficients corresponding to R, G, B three channels of the fake light source information fake _ group _ truth, respectively.
7. The color correction method based on the full-convolution neural network and the feature pyramid as claimed in claim 6, wherein the data enhancement of the clipped picture using the generated false light source information fake _ ground _ route includes:
and simulating and generating a picture image _ g ' by using a gamma correction inverse function, wherein the cut picture is st _ image and is in a format of [ stR ', stG ', stB ' ], and the picture image _ g ' is as follows:
if the format of the picture image _ g 'is [ Rg', Gg ', Bg' ], then [ Rf, Gf, Bf ] of the color cast picture fake _ image is:
Rf＝min(Rg′/fgtR′,255)
Gf＝min(Gg′/fgtG′,255)
Bf＝min(Bg′/fgtB′,255)
and converting Rf, Gf and Bf into a unit8 format to obtain a color cast picture fake _ image.
8. The color correction method based on the full convolution neural network and the feature pyramid as claimed in claim 1, wherein the Loss function is based on cosine similarity, and the calculation formula of the Loss function Loss is:
and g _ pre is image light source information estimated by a preset full convolution neural network and a characteristic pyramid model.
9. The color correction method based on the full convolution neural network and the feature pyramid as claimed in claim 1, wherein a plurality of subgraphs with the same size are cut from the image to be corrected at equal intervals, the light source information of each subgraph is respectively estimated by using the full convolution neural network and the feature pyramid model, the median of the light source information corresponding to each color channel is taken from all the light source information as the final estimated light source information, and the image to be corrected is subjected to color correction based on the final estimated light source information to obtain the corrected image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910659156.3A CN110400275B (en) | 2019-07-22 | 2019-07-22 | Color correction method based on full convolution neural network and characteristic pyramid |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910659156.3A CN110400275B (en) | 2019-07-22 | 2019-07-22 | Color correction method based on full convolution neural network and characteristic pyramid |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110400275A CN110400275A (en) | 2019-11-01 |
| CN110400275B true CN110400275B (en) | 2021-06-22 |
Family
ID=68324831
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910659156.3A Active CN110400275B (en) | 2019-07-22 | 2019-07-22 | Color correction method based on full convolution neural network and characteristic pyramid |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110400275B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111027562B (en) * | 2019-12-06 | 2023-07-18 | 中电健康云科技有限公司 | Optical character recognition method based on multiscale CNN and RNN combined with attention mechanism |
| EP3889882B1 (en) * | 2020-03-31 | 2023-06-07 | Siemens Healthcare GmbH | Image normalization increasing robustness of machine learning applications for medical images |
| CN112101383B (en) * | 2020-09-15 | 2022-10-21 | 遵义师范学院 | Color cast image identification method |
| CN112581373B (en) * | 2020-12-14 | 2022-06-10 | 北京理工大学 | Image color correction method based on deep learning |
| CN112884682B (en) * | 2021-01-08 | 2023-02-21 | 福州大学 | Stereo image color correction method and system based on matching and fusion |
| CN112884693A (en) * | 2021-03-25 | 2021-06-01 | 维沃移动通信（深圳）有限公司 | Training method and device of image processing model and white balance processing method and device |
| CN113159204A (en) * | 2021-04-28 | 2021-07-23 | 深圳市捷顺科技实业股份有限公司 | License plate recognition model generation method, license plate recognition method and related components |
| CN117649661B (en) * | 2024-01-30 | 2024-04-12 | 青岛超瑞纳米新材料科技有限公司 | Carbon nanotube preparation state image processing method |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015189629A3 (en) * | 2014-06-13 | 2016-03-17 | Bangor University | Improvements in and relating to the display of images |
| CN108388905A (en) * | 2018-03-21 | 2018-08-10 | 合肥工业大学 | A kind of Illuminant estimation method based on convolutional neural networks and neighbourhood context |
| CN109886273A (en) * | 2019-02-26 | 2019-06-14 | 四川大学华西医院 | A kind of CMR classification of image segmentation system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9633282B2 (en) * | 2015-07-30 | 2017-04-25 | Xerox Corporation | Cross-trained convolutional neural networks using multimodal images |
-
2019
- 2019-07-22 CN CN201910659156.3A patent/CN110400275B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015189629A3 (en) * | 2014-06-13 | 2016-03-17 | Bangor University | Improvements in and relating to the display of images |
| CN108388905A (en) * | 2018-03-21 | 2018-08-10 | 合肥工业大学 | A kind of Illuminant estimation method based on convolutional neural networks and neighbourhood context |
| CN109886273A (en) * | 2019-02-26 | 2019-06-14 | 四川大学华西医院 | A kind of CMR classification of image segmentation system |
Non-Patent Citations (3)
| Title |
|---|
| Recurrent Spatial Pyramid CNN for Optical Flow Estimation;Ping Hu et al;《IEEE TRANSACTIONS ON MULTIMEDIA》;20181031;第20卷(第10期);第2814-2823页 * |
| Underwater in image de-scattering and enhancing using dehazened and hwd;Pan-wang Pan et al;《Journal of Marine Science and Technology》;20180131;第26卷(第4期);第531-540页 * |
| 基于BP神经网络的颜色补偿模型;丁淑芝 等;《计算机工程与设计》;20101231;第31卷(第10期);第2294-2296页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110400275A (en) | 2019-11-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110400275B (en) | Color correction method based on full convolution neural network and characteristic pyramid | |
| CN112288658B (en) | Underwater image enhancement method based on multi-residual joint learning | |
| CN111292264B (en) | Image high dynamic range reconstruction method based on deep learning | |
| CN110232661B (en) | Low-illumination color image enhancement method based on Retinex and convolutional neural network | |
| CN109871845B (en) | Certificate image extraction method and terminal equipment | |
| CN111161360B (en) | Image defogging method of end-to-end network based on Retinex theory | |
| Li et al. | Insulator defect detection for power grid based on light correction enhancement and YOLOv5 model | |
| CN113052868B (en) | Method and device for training matting model and image matting | |
| CN110189260B (en) | Image noise reduction method based on multi-scale parallel gated neural network | |
| CN108022223A (en) | A kind of tone mapping method based on the processing fusion of logarithmic mapping function piecemeal | |
| CN112508812A (en) | Image color cast correction method, model training method, device and equipment | |
| CN109544467B (en) | Method for enhancing contrast of color image based on LAB model | |
| CN108447059A (en) | It is a kind of to refer to light field image quality evaluating method entirely | |
| Zhang et al. | Deep tone mapping network in HSV color space | |
| CN107545570A (en) | A kind of reconstructed image quality evaluation method of half reference chart | |
| CN112419179A (en) | Method, device, equipment and computer readable medium for repairing image | |
| Wang et al. | Multiscale supervision-guided context aggregation network for single image dehazing | |
| CN105069767B (en) | Based on the embedded Image Super-resolution reconstructing method of representative learning and neighborhood constraint | |
| TWI426779B (en) | Object color correction method, system and its computer program products | |
| CN111951171A (en) | HDR image generation method and device, readable storage medium and terminal equipment | |
| CN111553856A (en) | Image defogging method based on depth estimation assistance | |
| CN113112422B (en) | Image processing method, device, electronic equipment and computer readable medium | |
| CN113706400A (en) | Image correction method, image correction device, microscope image correction method, and electronic apparatus | |
| CN102663683B (en) | Image enhancement method and image enhancement system | |
| CN111754412B (en) | Method and device for constructing data pair and terminal equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |