CN110163927B - Single image re-coloring method based on neural network - Google Patents

Single image re-coloring method based on neural network Download PDF

Info

Publication number
CN110163927B
CN110163927B CN201910412100.8A CN201910412100A CN110163927B CN 110163927 B CN110163927 B CN 110163927B CN 201910412100 A CN201910412100 A CN 201910412100A CN 110163927 B CN110163927 B CN 110163927B
Authority
CN
China
Prior art keywords
neural network
image
coloring
training
recoloring
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910412100.8A
Other languages
Chinese (zh)
Other versions
CN110163927A (en
Inventor
王怡婷
厉旭杰
濮济
林选
王艳丹
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wenzhou University
Original Assignee
Wenzhou University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wenzhou University filed Critical Wenzhou University
Priority to CN201910412100.8A priority Critical patent/CN110163927B/en
Publication of CN110163927A publication Critical patent/CN110163927A/en
Application granted granted Critical
Publication of CN110163927B publication Critical patent/CN110163927B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
    • G06F18/24133—Distances to prototypes
    • G06F18/24137—Distances to cluster centroïds
    • G06F18/2414—Smoothing the distance, e.g. radial basis function networks [RBFN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/001—Texturing; Colouring; Generation of texture or colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/40—Filling a planar surface by adding surface attributes, e.g. colour or texture
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a single image recoloring method based on a neural network, which comprises the steps of inputting a color image to be processed, and performing line coloring on the color image to be processed by a user to obtain a colored image; extracting the pixel characteristic value of the area where the coloring line is located and classifying the corresponding coloring line into a data set, and randomly sampling data in the data set to serve as training data for training a neural network; constructing an image recoloring classification neural network, and training the neural network; the characteristic values of pixels in the image to be recolored one by one are used as the input of a neural network, and the likelihood probability that each pixel belongs to a coloring line is obtained; and calculating a final image recoloring result according to the likelihood probability that each pixel output by the neural network belongs to the coloring line. The invention avoids the defect that a neural network needs large-scale training samples in the training stage, achieves near-real-time interaction performance, and obtains the re-coloring effect of the high-quality image by only inputting a small amount of user coloring lines by a user.

Description

Single image re-coloring method based on neural network
Technical Field
The invention relates to a method for re-coloring a color image, in particular to a method for re-coloring a single image based on a neural network.
Background
Image Recoloring (Recoloring) refers to the process of modifying and adjusting the color appearance of an image. Traditional recoloring techniques are done by pure hand or with the assistance of computer software, which requires a significant amount of time for manual work. Therefore, many researchers at home and abroad are continuously exploring new methods to improve the efficiency and effect of the recoloring technology in recent years. Although the re-coloring technology has been greatly developed at present at home and abroad, further improvement is still needed at present, and the problems are mainly reflected in the treatment effect and efficiency. In terms of effect, the existing image recoloring method often generates a color penetration phenomenon, and the goal of image recoloring is to generate a satisfactory high-quality image recoloring effect by using as few user interactions as possible. In terms of efficiency, when the image size reaches a certain scale, the solution is very time-consuming, and the image size is further increased due to the memory limitation of a computer, and some re-coloring methods even cannot obtain re-coloring results.
Image recoloring methods are generally classified into color migration based techniques and painted line propagation based techniques. The color of the image is changed by utilizing the mapping relation between the image to be recolorized and the reference image based on the color migration technology, the effect of the method depends heavily on the selection of the reference image, and the search for the reference image with similar appearance is a difficult task, so that the application of the method is limited. See: e.reinhard, m.ashikhmin, b.gooch, and p.shirley.color transfer between images.ieee comput.graph.appl.,2001,21 (5): 34-41. Because the selection of the reference image is very difficult, some methods directly adopt the adjustment of the color palette to edit the image color. See: Q.Zhang, C.Xiao, H.Sun and F.Tang.Pattern-Based Image correlation Using Color calculation, IEEE Transactions on Image processing, 2017,26 (4): 1952-1964. Huang H Z et al propose a data-driven method for automatically finding a matching reference image from a database for automatically recoloring a photograph to enhance the appearance of the photograph or to change the emotional response of the viewer to the photograph, which can generate multiple recoloring results for a new input image to be selected by the user, but which requires feature matching with the pictures in the database one by one, which consumes a lot of time. Reference is made to Huang H Z, zhang S H, martin R R, et al, learning Natural Colors for Image Recolluting [ J ]. Computer Graphics Forum,2014,33 (7): 299-308. The method can better reflect the interactivity of the user and obtain a high-quality recoloring effect. In recent years, many scholars begin to research and adopt convolutional neural network training to solve the problem of reference image selection, the convolutional neural network can effectively extract information in images, the method usually utilizes the existing large-scale scene classification database to train a convolutional neural network model, and after the convolutional neural network model is trained, the input image to be recolored only needs to be subjected to one-time forward propagation, so that the recolored effect can be quickly obtained. The method well solves the problem of reference image selection based on the color migration technology, but the method can fully automatically recolor the image without user interaction, so that the obtained recolor effect is not the result the user wants. See: satoshi Iizuka, edgar Simo-Serra, and Hiroshi Ishikawa let rest be color! Joint end-to-end learning of global and local image documents for automatic image analysis with a hierarchical organization, ACM Transformations On Graphics (TOG), 2016,35 (4): 110.
To solve the problem of user interactivity, richard Zhang et al propose a deep learning method based on user-guided image recoloring. The method maps images directly, as well as sparse local user "cues," to outputs using a Convolutional Neural Network (CNN). The convolutional neural network propagates user edits from low-level cues and high-level semantic information obtained from large-scale data, the method recommends possible colors based on the input image and current user input in order to guide the user to make efficient input selections, and the coloring is performed in a feed-forward process and can be used in real-time. See: richard Zhang, jun-Yan Zhu, phillip Isola, xinyang Geng, angela S Lin, tianhe Yu, and Alexei A Efros.real-time user-defined image registration with left deleted documents.2017, arXiv preprint arXiv:1705.02999.
However, these methods are based on training samples at image level, and usually training a deep learning neural network requires a sufficient amount of training images and several days or even months to train the deep learning neural network. In the pixel-level image recoloring method, the convolutional neural network usually adopts an image patch as a feature vector, and the generated recoloring result has color penetration in an edge area, which seriously affects the image recoloring effect. Such methods therefore require the use of edge-preserving image filters at a later stage to improve the effect of further improving image recoloring.
At present, two main problems exist in the image recoloring method: first, most image segmentation methods are time consuming, and the image size is further increased due to computer memory limitations, and some recoloring methods may not even obtain recoloring results; secondly, the mainstream image segmentation method based on the deep learning neural network is to build training samples at an image level, and usually, a sufficient amount of training images are required to train a deep learning neural network, and several days or even several months are required to train the deep learning neural network. And these methods do not interact well with the user.
Disclosure of Invention
The invention aims to overcome the defects and shortcomings of the prior art, and provides a neural network-based single image recoloring method, which utilizes a deep learning neural network to recoloring an image as a pixel-level multi-value classification problem, and trains the deep learning neural network by using normalized RGB color values as feature vectors.
In order to achieve the purpose, the technical scheme of the invention comprises the following steps:
(1) Inputting a color image to be processed, and then carrying out line coloring on the color image to be processed by a user to obtain a colored image;
(2) Extracting the pixel characteristic value of the area where the coloring line is located and the corresponding coloring line to be classified into a data set, and randomly sampling data in the data set to be used as training data for training a neural network;
(3) Constructing an image recoloring classification neural network and training the neural network;
(4) After the neural network is trained, the characteristic values of pixels one by one in the image to be recolored are used as the input of the neural network, and the likelihood probability that each pixel belongs to the coloring line is obtained;
(5) And calculating a final image recoloring result according to the likelihood probability that each pixel output by the neural network belongs to the colored line.
The step (1) is further set as follows: the user selects the painting brush of corresponding colour, and the line coloring is gone up to the user on pending color image, and the colour value of painted lines represents the colour value after the regional image recoloring that is covered by the painting brush, and wherein white painted lines is special painted lines, and when the colour value of painted lines was white, the colour value after the regional image recoloring that the presentation was covered by the painting brush remained unchanged.
The method is further provided with the following steps:
extracting the pixel characteristic value of the area where the coloring line is located and classifying the corresponding coloring line into a data set, and randomly sampling the data set in order to reduce the training time of a neural network; the sampled data is used as training data for training a neural network
Figure BDA0002063137500000041
Wherein:
i is an index value;
z i the method comprises the steps that a coloring line classification vector corresponding to the ith training sample is obtained, the value of a target classification in the vector is equal to 1, and the values of other classifications are equal to 0;
c i input for the ith training sample: c. C i ＝[(R,G,B)] i (R, G, B) represents a normalized RGB vector of the color image;
M 1 to train the number of samples, M 1 Ceil (β M), the sampling rate β set to 0.1, ceil representing the rounding-up operation, M being the number of pixels covered by the painted line.
The further setting is that the step (3) of constructing the image recoloring classification neural network specifically comprises the following steps:
the image recoloring classification neural network consists of 3 groups of full connection layers and ReLU activation functions, and finally 1 group of full connection layers and softmax activation functions, wherein the connection sequence of the whole neural network is as follows: FC1-ReLU-FC2-ReLU-FC3-ReLU-FC4-softmax; the FC1 layer, the FC2 layer, the FC3 layer and the FC4 layer are full connection layers, the number of input neurons of the FC1 layer is 3, and the number of output neurons is 64; the number of input neurons of the FC2 layer is 64, and the number of output neurons is 128; the number of input neurons of the FC3 layer is 128, the number of output neurons is 128, the number of input neurons of the FC4 layer is 128, the number of output neurons is K, and K is the number of types of coloring lines; reLU is a nonlinear activation function, reLU (x) = max (0,x);
the whole neural network specifically operates as follows:
J(c)＝f softmax (f FC4 (f ReLU (f FC3 (f ReLU (f FC2 (f ReLU (f FC1 (c))))))))
wherein:
c = (R, G, B) normalized RGB vector for pixel in training set;
f ReLU corresponding to the ReLU activation function;
f softmax corresponding to the softmax activation function;
f FC1 -f FC4 corresponding to the full connection layer.
The step (3) of training the neural network further comprises:
the loss function between the output of the training sample through the fully connected deep learning neural network and the real training sample output is as follows:
Figure BDA0002063137500000051
wherein:
i, k are index values;
M 1 is the number of training samples;
k is the number of the types of the coloring lines;
l is a cross entropy loss function;
a i an inactivated output at the softmax level for the ith training sample;
z i for the classification vector of the ith training sample in the training set, the target classification z in the vector ik Equal to 1, other classes z ik Equal to 0;
P ik calculating the likelihood probability that the ith training sample is predicted to belong to the classification k for the output of softmax;
Figure BDA0002063137500000052
and optimizing the loss function E to obtain a minimum extreme value, so as to obtain parameters of the deep learning neural network.
Further setting is that the step (4) comprises:
after the image recoloring classification neural network is trained, the characteristic values of pixels one by one in the color image to be processed
Figure BDA0002063137500000053
As the input of the neural network, wherein N is the number of the pixels of the whole image; c. C i ＝[(R,G,B)] i The trained neural network acts as a multi-valued classifier to generate likelihood probabilities that each pixel belongs to a colored line.
Further, the step (5) comprises the following steps:
the softmax layer of the neural network outputs the likelihood probability that each sample belongs to the colored line, and the final image recoloring result is calculated as follows:
Figure BDA0002063137500000061
wherein:
i is a pixel index value;
k is the number of the types of the coloring lines;
P ik predicting a likelihood probability for the ith sample as belonging to class k;
S k color values of the kth colored line; when the colored line is a special white colored line, S k The value of (A) is the original color value of the image under the pixel covered by the coloring line;
y is the final image recoloring result.
According to the method, a user only needs to input a small number of coloring lines on the image to be recolored, then the pixel characteristic values of the area where the coloring lines are located and the corresponding coloring lines are extracted and classified into the data set, and the defect that a neural network needs to train a sample in a large scale in a training stage is overcome, so that the method can achieve near-real-time interaction performance, and meanwhile, the user only needs to input a small number of user coloring lines to obtain a high-quality image recolored effect.
The invention has the beneficial effects that:
1. according to the method, a user only needs to input a small number of coloring lines on the image to be re-colored, then the pixel characteristic values of the area where the coloring lines are located and the corresponding coloring lines are extracted and classified into the data set, and the defect that a neural network needs to train samples in a large scale in a training stage is overcome.
2. The method can achieve near-real-time interaction performance, uses as few user interactions as possible, and simultaneously obtains the high-quality image recoloring effect.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is within the scope of the present invention for those skilled in the art to obtain other drawings based on the drawings without inventive exercise.
FIG. 1 is a flow chart of the present invention;
FIG. 2 is a neural network model of the present invention;
FIG. 3 compares the effect of the sampling rate β and the number of iterations t on the image recoloring effect;
FIG. 4 is a comparison of the present invention with the current mainstream image recoloring effect;
FIG. 5 is a diagram of the results of the invention for recoloring multiple sets of images.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the accompanying drawings.
The single image segmentation method based on the fully-connected deep learning neural network described in this embodiment includes the following five steps, see fig. 1:
s1: inputting a color image to be processed, and then, carrying out line coloring on the color image to be processed by a user to obtain a colored image;
the user selects the painting brush of corresponding colour, and the user carries out line coloring on the pending colour image, and the colour value of colouring the lines represents the colour value after the regional image that is covered by the painting brush recoloring, and wherein white colouring lines is special colouring lines, and when the colour value of colouring lines was white, it remained unchanged to represent the colour value after the regional image that is covered by the painting brush recoloring.
S2: extracting the pixel characteristic value of the area where the coloring line is located and the corresponding coloring line to be classified into a data set, and randomly sampling data in the data set to be used as training data for training a neural network;
extracting the pixel characteristic value of the area where the coloring line is located and classifying the corresponding coloring line into a data set, and randomly sampling the data set in order to reduce the training time of a neural network; the sampled data is used as training data for training a neural network
Figure BDA0002063137500000071
Wherein:
i is an index value;
z i the method comprises the steps that a coloring line classification vector corresponding to the ith training sample is obtained, the value of a target classification in the vector is equal to 1, and the values of other classifications are equal to 0;
c i input for the ith training sample: c. C i ＝[(R,G,B)] i (R, G, B) represents a normalized RGB vector of the color image;
M 1 for the number of training samples, M 1 Ceil (β M), the sampling rate β set to 0.1, ceil representing the rounding-up operation, M being the number of pixels covered by the painted line.
S3: constructing an image recoloring classification neural network and training the neural network;
the invention relates to a single image recoloring method based on a neural network, wherein the model architecture of the neural network is shown in figure 2; the number of input and output neurons of each fully-connected layer of the neural network is shown in table 1, FC1-FC3 each fully-connected layer is followed by a ReLU activation function, and the last fully-connected layer FC4 is followed by a softmax activation function
TABLE 1 number of input and output neurons of each fully-connected layer of the neural network of the method
Type (B) FC1 FC2 FC3 FC4
Number of input neurons 3 64 128 128
Number of output neurons 64 128 128 K
The image recoloring classification neural network consists of 3 groups of full connection layers and ReLU activation functions, and finally 1 group of full connection layers and softmax activation functions, wherein the connection sequence of the whole neural network is as follows: FC1-ReLU-FC2-ReLU-FC3-ReLU-FC4-softmax; the FC1 layer, the FC2 layer, the FC3 layer and the FC4 layer are full connection layers, the number of input neurons of the FC1 layer is 3, and the number of output neurons is 64; the number of input neurons of the FC2 layer is 64, and the number of output neurons is 128; the number of input neurons of the FC3 layer is 128, the number of output neurons is 128, the number of input neurons of the FC4 layer is 128, the number of output neurons is K, and K is the number of types of coloring lines; reLU is a nonlinear activation function, reLU (x) = max (0,x);
the whole neural network specifically operates as follows:
J(c)＝f softmax (f FC4 (f ReLU (f FC3 (f ReLU (f FC2 (f ReLU (f FC1 (c))))))))
wherein:
c = (R, G, B) normalized RGB vector for pixel in training set;
f ReLU corresponding to the ReLU activation function;
f softmax corresponding to the softmax activation function;
f FC1 -f FC4 corresponding to the full connection layer.
The loss function between the output of the training sample through the fully-connected deep learning neural network and the real training sample output is as follows:
Figure BDA0002063137500000091
wherein:
i, k are index values;
M 1 is the number of training samples;
k is the number of the types of the coloring lines;
l is a cross entropy loss function;
a i the inactivated output of the ith training sample at the softmax layer;
z i for the classification vector of the ith training sample in the training set, the target classification z in the vector ik Equal to 1, other classes z ik Equal to 0;
P ik calculating the likelihood probability that the ith training sample is predicted to belong to the classification k for the output of softmax;
Figure BDA0002063137500000092
and optimizing the loss function E to obtain a minimum extreme value, so as to obtain parameters of the deep learning neural network.
S4: after the neural network is trained, the characteristic values of pixels one by one in the image to be recolored are used as the input of the neural network, and the likelihood probability that each pixel belongs to the coloring line is obtained;
after the image recoloring classification neural network is trained, the characteristic values of pixels one by one in the color image to be processed
Figure BDA0002063137500000101
As the input of the neural network, wherein N is the number of the pixels of the whole image; c. C i ＝[(R,G,B)] i The trained neural network acts as a multi-valued classifier to generate likelihood probabilities that each pixel belongs to a colored line.
S5: calculating a final image recoloring result according to the likelihood probability that each pixel output by the neural network belongs to a coloring line;
the softmax layer of the neural network outputs the likelihood probability that each sample belongs to the colored line, and the final image recoloring result is calculated as follows:
Figure BDA0002063137500000102
wherein:
i is a pixel index value;
k is the number of the types of the coloring lines;
P ik predicting a likelihood probability for the ith sample as belonging to class k;
S k color values of the kth colored line; when the colored line is a special white colored line, S k The value of (A) is the original color value of the image under the pixel covered by the coloring line;
y is the final image recoloring result.
The method uses a deep learning neural network to recolor the image as a pixel-level multi-value classification problem, and uses normalized RGB color values as feature vectors to train the deep learning neural network. According to the method, a user only needs to input a small number of coloring lines on the image to be recolored, then the pixel characteristic values of the area where the coloring lines are located and the corresponding coloring lines are extracted and classified into the data set, and the defect that a neural network needs large-scale training samples in a training stage is overcome, so that the method can achieve near-real-time interaction performance, use as few user interactions as possible, and meanwhile obtain a high-quality image recolored effect. Inputting a color image to be processed, and then coloring lines of the color image to be processed by a user to obtain a colored image; extracting the pixel characteristic value of the area where the coloring line is located and the corresponding coloring line to be classified into a data set, and randomly sampling data in the data set to be used as training data for training a neural network; constructing an image recoloring classification neural network and training the neural network; after the neural network is trained, the characteristic values of pixels one by one in the image to be recolored are used as the input of the neural network, and the likelihood probability that each pixel belongs to the coloring line is obtained; and calculating a final image recoloring result according to the likelihood probability that each pixel output by the neural network belongs to the colored line.
The invention uses python 3.7 and chainer 5.3.0 libraries to realize a single image re-coloring method based on a neural network, and all experiments are carried out on machines of NVIDIA GeForce RTX 2080Ti GPU and Intel I9-7900X CPU. The method only extracts the pixel characteristic value of the area where the colored line is located and classifies the corresponding colored line into the data set, but the training of the neural network still needs tens of seconds, and in order to reduce the training time, the method provides an effective sampling strategy. Therefore, two parameters influence the re-coloring effect of the method, one is the sampling rate beta of the training set, and the other is the iteration number t of the neural network. Fig. 3 compares the effect of the sampling rate β and the number of iterations t on the image re-coloring effect, and as shown, the image re-coloring effect almost converges when the sampling rate β =0.1 and the number of iterations t =2. Here we see that a higher sampling rate β and number of iterations t does not produce significantly better recolouration. In experiments, we have determined that typically 10% of the user stroke pixels β =0.1 is sufficient to train our neural network for image recoloring. Therefore, the sampling rate of the image coloring adopted by the method is beta =0.1, and the iteration number is t =2.
Fig. 4 is a diagram showing the effect of the present invention compared with the current mainstream image recoloring method, wherein white strokes indicate that the color of the area should be kept unchanged after recoloring the image, and other color strokes indicate the final color after recoloring the image. Fig. 4 compares the effect comparison results of the method of the present invention with the main stream repigmentation method DeepProp, global optimization method and local optimization method. As can be seen from the figure, both the local optimization and global optimization methods have their own limitations. The local optimization method (see: ant Levin, dani Lischinski, and Yair Weiss. Visualization using optimization. In ACM transformations on graphics (tog), volume 23, pages 689-694.ACM, 2004.) considers only local spatial neighborhood propagation within a local window (3 × 3). The partial image approach provides good local control for the user, but it performs poorly when the color constraints provided are recoloring relatively far. Therefore, their approach requires the addition of more user interaction to achieve high quality results. The global optimization method (refer to Przemyslaw Musialski, ming Cui, jieping Ye, anshuman Razdan, and Peter Wonka. A frame work for interactive image color editing. The Visual Computer,29 (11): 1173-1186, 2013.) can realize global color propagation, and a user can generate a high-quality image re-coloring effect only by inputting a small number of coloring lines, but the method needs to consume a large amount of memory, and even cannot generate a correct result when the image reaches a certain size. DeepProp (reference: yuki Endo, satoshi Izuka, yoshihiro Kanamori, and Jun Mitani. Deepprop: extracting deep defects from a single image for an image re-coloring. In Computer Graphics Forum, volume 35, pages 189-201.Wiley Online library, 2016.) uses a convolutional neural network to produce an image re-coloring effect, but because of the use of an image patch as a feature vector, deepProp loses precision near the re-colored edge, requiring further refinement of the image re-coloring result using an edge-preserving filter to improve the quality of image re-coloring. The method can generate the high-quality image recoloring effect only by inputting a small amount of user recoloring lines, and does not need to perform the perfecting step at the later stage; meanwhile, only the pixel characteristic value of the area where the coloring line is located and the corresponding coloring line are extracted and classified into a data set, and an effective sampling strategy is provided, so that the training of the neural network achieves near-real-time performance, and the mini-batch training is adopted, and the consumption of the memory is small.
Table 2 operating efficiency of the method of the invention (sampling rate β =0.1, iteration number t = 2)
Figure BDA0002063137500000121
In order to test the operation efficiency of the method of the present invention, table 2 shows the operation time of three groups of images in fig. 4, the third column shows the number of pixel points of the painted lines, and the fourth column shows the operation time of the method of the present invention, as can be seen from table 2, because the method of the present invention only uses the pixel characteristic values of the areas where the painted lines are located and the corresponding painted lines as the training set, and adopts an effective sampling strategy; therefore, the time and memory requirements of neural network training are greatly reduced, and the image recoloring can be completed within 1-2 seconds.
FIG. 5 is a diagram showing the result of the present invention re-coloring a plurality of groups of images, and it can be seen from FIG. 5 that the method of the present invention can obtain a high quality image re-coloring effect with only a small amount of user input for various images.
It will be understood by those skilled in the art that all or part of the steps in the method for implementing the above embodiments may be implemented by relevant hardware instructed by a program, and the program may be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc.
The above disclosure is only for the purpose of illustrating the preferred embodiments of the present invention, and it is therefore to be understood that the invention is not limited by the scope of the appended claims.

Claims (5)

1. A method for single image recoloring based on neural networks, comprising the steps of:
(1) Inputting a color image to be processed, and then, carrying out line coloring on the color image to be processed by a user to obtain a colored image;
(2) Extracting the pixel characteristic value of the area where the coloring line is located and the corresponding coloring line to be classified into a data set, and randomly sampling data in the data set to be used as training data for training a neural network;
(3) Constructing an image recoloring classification neural network, and training the neural network;
(4) After the neural network is trained, the characteristic values of pixels in the image to be recolorized one by one are used as the input of the neural network, and the likelihood probability that each pixel belongs to a coloring line is obtained;
(5) Calculating a final image recoloring result according to the likelihood probability that each pixel output by the neural network belongs to a coloring line;
the step (3) of constructing the image recoloring classification neural network specifically comprises the following steps:
the image recoloring classification neural network consists of 3 groups of full connection layers and ReLU activation functions, and finally 1 group of full connection layers and softmax activation functions, wherein the connection sequence of the whole neural network is as follows: FC1-ReLU-FC2-ReLU-FC3-ReLU-FC4-softmax; the FC1 layer, the FC2 layer, the FC3 layer and the FC4 layer are full connection layers, the number of input neurons of the FC1 layer is 3, and the number of output neurons is 64; the number of input neurons of the FC2 layer is 64, and the number of output neurons is 128; the number of input neurons of the FC3 layer is 128, the number of output neurons of the FC3 layer is 128, the number of input neurons of the FC4 layer is 128, the number of output neurons of the FC4 layer is K, and K is the number of types of colored lines; reLU is a nonlinear activation function, reLU (x) = max (0,x);
the whole neural network specifically operates as follows:
J(c)＝f softmax (f FC4 (f ReLU (f FC3 (f ReLU (f FC2 (f ReLU (f FC1 (c))))))))
wherein:
c = (R, G, B) normalized RGB vector for pixel in training set;
f ReLU corresponding to the ReLU activation function;
f softmax corresponding to the softmax activation function;
f FC1 -f FC4 corresponding to the full connection layer;
the training of the neural network in the step (3) is specifically as follows:
the loss function between the output of the training sample through the fully-connected deep learning neural network and the real training sample output is as follows:
Figure FDA0003983467670000021
wherein:
i, k are index values;
M 1 is the number of training samples;
k is the number of the types of the coloring lines;
l is a cross entropy loss function;
a i the inactivated output of the ith training sample at the softmax layer;
z i for the classification vector of the ith training sample in the training set, the target classification z in the vector ik Equal to 1, other classes z ik Equal to 0;
P ik calculating the likelihood probability that the ith training sample is predicted to belong to the classification k for the output of softmax;
Figure FDA0003983467670000022
/>
and optimizing the loss function E to obtain a minimized extreme value, and obtaining parameters of the deep learning neural network.
2. The neural network-based single image recoloring method of claim 1, wherein: the step (1) is specifically as follows: the user selects the painting brush of corresponding colour, and the line coloring is gone up to the user on pending color image, and the colour value of painted lines represents the colour value after the regional image recoloring that is covered by the painting brush, and wherein white painted lines is special painted lines, and when the colour value of painted lines was white, the colour value after the regional image recoloring that the presentation was covered by the painting brush remained unchanged.
3. The neural network-based single image recoloring method of claim 1, wherein: the step (2) of extracting the pixel characteristic values of the areas where the colored lines are located and the corresponding colored lines are classified into a data set, and randomly sampling data in the data set as training data for training a neural network specifically comprises the following steps:
extracting pixel characteristic value of area where coloring line is located and corresponding coloring lineThe bars are classified into data sets, and the data sets are randomly sampled in order to reduce the training time of the neural network; the sampled data is used as training data for training a neural network
Figure FDA0003983467670000031
Wherein:
i is an index value;
z i the method comprises the steps that a coloring line classification vector corresponding to the ith training sample is obtained, the value of a target classification in the vector is equal to 1, and the values of other classifications are equal to 0;
c i input for the ith training sample: c. C i ＝[(R,G,B)] i (R, G, B) represents a normalized RGB vector of the color image;
M 1 for the number of training samples, M 1 Ceil (β M), with the sampling rate β set to 0.1, ceil representing the rounding-up operation, M being the number of pixels covered by the painted line.
4. The method for re-coloring single image based on neural network as claimed in claim 1, wherein after training the neural network in step (4), the method takes the feature values of the pixels in the image to be re-colored as the input of the neural network, and obtaining the likelihood probability that each pixel belongs to the colored line specifically comprises:
after the image recoloring classification neural network is trained, the characteristic values of pixels one by one in the color image to be processed
Figure FDA0003983467670000032
As the input of the neural network, wherein N is the number of the pixels of the whole image; c. C i ＝[(R,G,B)] i The trained neural network acts as a multi-valued classifier to generate likelihood probabilities that each pixel belongs to a colored line.
5. The method for single image recoloring based on neural network as claimed in claim 4, wherein the step (5) calculates the final image recoloring result according to the likelihood probability that each pixel output by the neural network belongs to the colored line, specifically:
the softmax layer of the neural network outputs the likelihood probability that each sample belongs to the colored line, and the final image recoloring result is calculated as follows:
Figure FDA0003983467670000033
wherein:
i is a pixel index value;
k is the number of the types of the coloring lines;
P ik predicting a likelihood probability for the ith sample as belonging to class k;
S k color values of the kth colored line; when the colored line is a special white colored line, S k The value of (A) is the original color value of the image under the pixel covered by the coloring line;
y is the final image recoloring result.
CN201910412100.8A 2019-05-17 2019-05-17 Single image re-coloring method based on neural network Active CN110163927B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910412100.8A CN110163927B (en) 2019-05-17 2019-05-17 Single image re-coloring method based on neural network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910412100.8A CN110163927B (en) 2019-05-17 2019-05-17 Single image re-coloring method based on neural network

Publications (2)

Publication Number Publication Date
CN110163927A CN110163927A (en) 2019-08-23
CN110163927B true CN110163927B (en) 2023-04-07

Family

ID=67631270

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910412100.8A Active CN110163927B (en) 2019-05-17 2019-05-17 Single image re-coloring method based on neural network

Country Status (1)

Country Link
CN (1) CN110163927B (en)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102096823A (en) * 2011-02-12 2011-06-15 厦门大学 Face detection method based on Gaussian model and minimum mean-square deviation
CN106991370A (en) * 2017-02-28 2017-07-28 中科唯实科技（北京）有限公司 Pedestrian retrieval method based on color and depth

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9792492B2 (en) * 2015-07-07 2017-10-17 Xerox Corporation Extracting gradient features from neural networks
US10916001B2 (en) * 2016-11-28 2021-02-09 Adobe Inc. Facilitating sketch to painting transformations

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102096823A (en) * 2011-02-12 2011-06-15 厦门大学 Face detection method based on Gaussian model and minimum mean-square deviation
CN106991370A (en) * 2017-02-28 2017-07-28 中科唯实科技（北京）有限公司 Pedestrian retrieval method based on color and depth

Also Published As

Publication number Publication date
CN110163927A (en) 2019-08-23

Similar Documents

Publication Publication Date Title
CN109711413B (en) Image semantic segmentation method based on deep learning
CN115049936B (en) High-resolution remote sensing image-oriented boundary enhanced semantic segmentation method
WO2022252272A1 (en) Transfer learning-based method for improved vgg16 network pig identity recognition
CN112669325B (en) Video semantic segmentation method based on active learning
CN109685072B (en) Composite degraded image high-quality reconstruction method based on generation countermeasure network
CN109410168B (en) Modeling method of convolutional neural network for determining sub-tile classes in an image
CN110163884B (en) Single image segmentation method based on full-connection deep learning neural network
CN108305253B (en) Pathological image classification method based on multiple-time rate deep learning
CN113128620B (en) Semi-supervised domain self-adaptive picture classification method based on hierarchical relationship
CN112287941B (en) License plate recognition method based on automatic character region perception
CN111105422B (en) Reticulocyte classification counting model construction method and application
CN108428238A (en) A kind of detection method general based on the polymorphic type task of depth network
CN111340720B (en) Color matching woodcut style conversion algorithm based on semantic segmentation
CN116310305A (en) Coding and decoding structure semantic segmentation model based on tensor and second-order covariance attention mechanism
Shen et al. Digital forensics for recoloring via convolutional neural network
CN114882278A (en) Tire pattern classification method and device based on attention mechanism and transfer learning
CN114743133A (en) Lightweight small sample video classification and identification method and system
CN110942463B (en) Video target segmentation method based on generation countermeasure network
CN110163927B (en) Single image re-coloring method based on neural network
CN108428234A (en) Interactive Segmentation performance optimization method based on image segmentation result evaluation
Fang et al. Stylized-colorization for line arts
CN115018729B (en) Content-oriented white box image enhancement method
Madkour et al. Automatic face and hijab segmentation using convolutional network
CN113222811A (en) Face attribute migration method based on image mask
CN113112397A (en) Image style migration method based on style and content decoupling

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant