CN114494523A - Line draft automatic coloring model training method and device under limited color space, electronic equipment and storage medium - Google Patents

Line draft automatic coloring model training method and device under limited color space, electronic equipment and storage medium Download PDF

Info

Publication number
CN114494523A
CN114494523A CN202210084599.6A CN202210084599A CN114494523A CN 114494523 A CN114494523 A CN 114494523A CN 202210084599 A CN202210084599 A CN 202210084599A CN 114494523 A CN114494523 A CN 114494523A
Authority
CN
China
Prior art keywords
color
coloring
image
module
style
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202210084599.6A
Other languages
Chinese (zh)
Other versions
CN114494523B (en
Inventor
陈缘
赵洋
曹力
李琳
谢文军
刘晓平
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hefei University of Technology
Original Assignee
Hefei University of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hefei University of Technology filed Critical Hefei University of Technology
Priority to CN202210084599.6A priority Critical patent/CN114494523B/en
Publication of CN114494523A publication Critical patent/CN114494523A/en
Application granted granted Critical
Publication of CN114494523B publication Critical patent/CN114494523B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/40—Filling a planar surface by adding surface attributes, e.g. colour or texture
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/23—Clustering techniques
    • G06F18/232—Non-hierarchical techniques
    • G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
    • G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a line draft automatic coloring model training method and device under a limited color space, electronic equipment and a storage medium, and belongs to the technical field of image processing. The method comprises the steps of obtaining a corresponding main color matrix as a main color vector according to the line draft type, conducting multi-level coding on the given main color vector to obtain a corresponding color code, constructing a limited color space, configuring a coloring model in advance to color the line draft in the limited color space, constructing different coloring models aiming at different drawing styles, further coloring the line draft, obtaining a coloring image of a specific drawing style, and meeting the requirement of personalized coloring.

Description

Line draft automatic coloring model training method and device under limited color space, electronic equipment and storage medium
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to a line draft automatic coloring model training method and device under a limited color space, electronic equipment and a storage medium.
Background
The traditional line draft coloring generally refers to a process of manually drawing reasonable and aesthetic colors on a line draft by an artist, such as coloring a sketch of a picture by a painter, coloring a partial mirror sketch in an animation production process, and the like. However, these purely manual processes are not only highly demanding for the painter, but also a time-consuming and labor-intensive process. Therefore, the automatic coloring of the line draft has an important application prospect.
In recent years, the fields of computer vision, image processing and the like are rapidly developed, particularly, deep learning methods can intelligently solve some problems, but related coloring applications are mainly to color gray-scale images and color line drawings with reference images or manual interaction. For example, a method of coloring a line draft image is disclosed; the method comprises the following steps: acquiring a first line draft image and a first reference image, wherein the first reference image comprises image information which is referred to when the first line draft image is colored; and coloring the first line manuscript image based on the first reference image by using a model based on machine learning training.
Although these methods can also achieve good coloring effect, they still need manual participation and cannot automatically and efficiently complete the coloring task of line art. And the problem that the line draft is colored completely and automatically by a computer is a difficult task, and unreasonable and not-according-to-human-aesthetic coloring results are easy to appear. For example, a one-sided red flag line manuscript may be colored, and a deep network may be colored in various colors without sufficient semantic analysis capability, and does not necessarily meet the specified requirements. In practice, there are many applications with limited color space, such as drawing in a given style, artistic creation, specific cartoon animation, etc. all often have a fixed color style prior. Therefore, the method is difficult to learn the painting style, and the coloring effect is greatly different from the artificial coloring result.
Disclosure of Invention
Aiming at the problem of poor coloring effect of the prior art on images with specified painting styles, the invention provides a line draft automatic coloring model training method in a limited color space.
In order to solve the above problems, the present invention adopts the following technical solutions.
The invention provides a line draft automatic coloring model training method under a limited color space, which comprises the following steps:
acquiring a corresponding primary color vector according to the line draft type;
carrying out multi-level coding on the main color vector to obtain corresponding color codes and constructing a limited color space;
the method comprises the steps of configuring a coloring model in advance, wherein the coloring model comprises a color reasoning module, a coloring module and a color correcting module, inputting a line draft and a color code into the color reasoning module to obtain a color block image, inputting the line draft and the color block image into the coloring module to obtain a first coloring image, and inputting the first coloring image into the color correcting module to be calibrated to obtain a second coloring image;
inputting the second coloring image and the real color image corresponding to the line draft into a style discriminator to obtain style discrimination loss, inputting the second coloring image into a color discriminator to obtain color discrimination loss, obtaining reconstruction loss by calculating the mean square error of the second coloring image and the real color image corresponding to the second coloring image, and adjusting the coloring model according to the style discrimination loss, the color discrimination loss and the reconstruction loss through a back propagation algorithm to finish the training of the coloring model.
Preferably, the obtaining the corresponding dominant color vector includes: and clustering the colors of the sample images by a k-means algorithm to obtain the main color.
As a preferred scheme, the line draft and the color block image are respectively subjected to down sampling, and then the line draft and the color block image are input into a coloring module for multi-scale coloring to obtain a first coloring image.
Preferably, the loss function of the coloring model includes a reconstruction loss LreJudgment of style loss LstyleAnd color discrimination loss LcolorThen the loss function is formulated as follows:
Lre＝||C2-I||2
Figure BDA0003486918380000022
Lcolor＝1-Sco
wherein I denotes a real color image, DSIRepresenting the output of the true color image I obtained by the style discriminator,
Figure BDA0003486918380000021
representing a second coloring image C2Output obtained by the style discriminator; wherein ScoRepresenting a second coloring image C2And the output is obtained after the color discriminator.
As a preferred scheme, multi-level coding is carried out on the given main color vector through a preset color coding module;
the color coding module comprises 1 size adjustment layer, n convolution layers with convolution kernel size of 3x3 and step size of 1, and a ReLU activation function layer.
The second aspect of the present invention provides an automatic coloring method for line draft in a limited color space, comprising the following steps:
acquiring a line draft to be colored;
inputting the line manuscript to be colored into the coloring model according to any one of claims 1 to 5 to obtain a coloring image.
The third aspect of the present invention provides a training device for an automatic line draft coloring model in a limited color space, the training device comprising:
the color acquisition module is used for acquiring a corresponding primary color vector according to the line draft type;
the color coding module is used for carrying out multi-level coding on the main color vector to obtain corresponding color coding and construct a limited color space;
the model configuration module is used for configuring a coloring model in advance, the coloring model comprises a color reasoning module, a coloring module and a color correction module, and the color reasoning module is used for receiving line draft and carrying out color coding processing to obtain a color block image; the coloring module is used for receiving the line draft and the color block image for processing to obtain a first coloring image; the color correction module is used for receiving the first coloring image for calibration to obtain a second coloring image;
and the model calibration module is used for inputting the second coloring image and any real color image into the style discriminator, inputting the second coloring image into the color discriminator, respectively acquiring style discrimination loss and color discrimination loss, obtaining reconstruction loss by calculating the mean square error of the second coloring image and the corresponding real color image, adjusting the coloring model according to the style discrimination loss, the color discrimination loss and the reconstruction loss through a back propagation algorithm, and finishing the training of the coloring model.
Preferably, the loss function of the coloring model includes a reconstruction loss LreJudgment of style loss LstyleAnd color discrimination loss LcolorThen the loss function is formulated as follows:
Lre＝||C2-I||2
Figure BDA0003486918380000032
Lcolor＝1-Sco
wherein I denotes a real color image, DSIRepresenting true color image I meridian styleThe output obtained by the discriminator is used as the output,
Figure BDA0003486918380000031
representing a second coloring image C2Output obtained by the style discriminator; wherein ScoRepresenting a second coloring image C2And the output is obtained after the color discriminator.
A fourth aspect of the present invention provides an electronic device, including a processor, an input device, an output device, and a memory, where the processor, the input device, the output device, and the memory are connected in sequence, the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the above method.
A fifth aspect of the invention provides a readable storage medium, the storage medium storing a computer program comprising program instructions which, when executed by a processor, cause the processor to perform the method described above.
Compared with the prior art, the invention has the beneficial effects that:
(1) according to the method, the corresponding main color vector is obtained according to the line draft type, the given main color vector is subjected to multi-level coding to obtain the corresponding color code, the limited color space is constructed, the coloring model is configured in advance to color the line draft in the limited color space, different coloring models can be constructed aiming at different painting styles, and then the line draft is colored, so that a coloring image with a specific painting style can be obtained, the personalized requirement is met, and the problem of unreasonable coloring in the existing automatic coloring is solved;
(2) the coloring model of the embodiment of the invention is designed in a multi-stage and multi-scale mode and comprises a color reasoning module, a coloring module and a color correcting module, wherein the coloring module is used for coloring the line draft; the difficulty of finishing coloring at one time is reduced through a staged multi-scale design, and the quality of the colored image is improved;
(3) the line draft and the color code are input into a color reasoning module, the outline of the line draft is roughly colored by selecting proper colors from a limited color space constructed by the color code, an obtained color block image is convenient for more finely coloring in a subsequent stage, the line draft and the color block image are input into a coloring module to obtain a first coloring image, the coloring module adopts a multi-scale structure, the small-scale image is colored firstly, the image is enlarged and colored step by step, a finer coloring result can be obtained in a coarse-to-fine mode, the first coloring image is input into a color correcting module to be calibrated, the coloring result is optimized, and a second coloring image is obtained; in addition, the embodiment of the invention can realize automatic coloring, only a user needs to give a main color, and coloring is automatically finished by utilizing the trained coloring model, so that the coloring model is used for assisting the painter in coloring.
Drawings
The above and other objects, features and advantages of the present application will become more apparent by describing in more detail embodiments of the present application with reference to the attached drawings. The accompanying drawings are included to provide a further understanding of the embodiments of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings, like reference numbers generally represent like parts or steps. In the drawings:
FIG. 1 is a flowchart of a line art automatic coloring model training method in a limited color space according to an embodiment of the present invention;
FIG. 2 is a block diagram of a color inference module in accordance with an embodiment of the present invention;
fig. 3 is a schematic diagram of a process of coloring a line draft according to an embodiment of the present invention;
FIG. 4 illustrates a block diagram of an electronic device in accordance with an embodiment of the present application;
FIG. 5 is a block diagram of a coloring module according to an embodiment of the invention;
fig. 6 is a block diagram of an apparatus for training a line art automatic coloring model in a limited color space according to an embodiment of the present invention.
Detailed Description
Hereinafter, example embodiments according to the present application will be described in detail with reference to the accompanying drawings. It should be understood that the described embodiments are only some embodiments of the present application and not all embodiments of the present application, and that the present application is not limited by the example embodiments described herein.
Exemplary method
As shown in fig. 1, this embodiment provides a line draft automatic coloring model training method under a limited color space, where the method includes the following steps:
s110: and acquiring a corresponding primary color vector according to the line draft type.
Specifically, in this example, the line art refers to an image that needs to be colored. The line drawing type mainly refers to the type of line drawing to be colored, taking Tibetan style line drawing as an example, the Tibetan style color drawing uses local mineral and plant pigments and has a relatively fixed plurality of main colors, such as red, blue, yellow and green. Accordingly, the RGB pixel values of the main color form a three-channel color matrix, and a corresponding main color vector is obtained. And during line draft extraction, extracting a real color image I as a line draft by using the existing edge extraction algorithm or photoshop, and constructing a training set.
It should be understood that the obtaining of the corresponding primary color vector may be manually given, or may be a method of obtaining the corresponding color matrix by statistically selecting the primary color through a color histogram according to a large number of samples, or clustering the colors of the sample images by using a k-means algorithm to obtain the primary color.
S120: and carrying out multi-level coding on the main color vector to obtain corresponding color codes, and constructing a limited color space.
Specifically, taking the Tibetan color drawing manuscript as an example, the primary colors can be red, blue, yellow and green, and the primary color vector passes through a preset color coding module ECThe color code C is obtained by encoding the color code for multiple timesE. The step is mainly to encode the color vectors layer by layer to obtain color information under a hidden space, and construct a limited color space for guiding subsequent coloring. For example, in the case of the Tibetan sketching, the main colors in the limited color space include red, blue, yellow and green, and the subsequent coloring process will only select a suitable color from these colors.
As a possible way of implementing the method,color coding module ECComprising 1 size adjustment layer, n convolution layers with convolution kernel size of 3x3 and step size of 1 and ReLU activation function layer, the input color vector size in this example is 16x3, the adjusted size is 4x4x3, n is 4, and color coding CEDimensions 4x4x 512; table 1 shows the corresponding output sizes of the layers of the color coding module.
TABLE 1
Size adjusting layer 4x4x3
Convolutional layer with convolution kernel size of 3x3 with step size of 1, ReLU 4x4x64
Convolutional layer with convolution kernel size of 3x3 with step size of 1, ReLU 4x4x128
Convolutional layer with convolution kernel size of 3x3 and step length of l, ReLU 4x4x256
Convolutional layer with convolution kernel size of 3x3 with step size of 1, ReLU 4x4x512
S130: the method comprises the steps of configuring a coloring model in advance, wherein the coloring model comprises a color reasoning module, a coloring module and a color correction module, inputting a line draft and a color code into the color reasoning module to obtain a color block image, inputting the line draft and the color block image into the coloring module to obtain a first coloring image, inputting the first coloring image into the color correction module to be calibrated, and obtaining a second coloring image.
Specifically, the color inference module, the coloring module, and the color correction module in this example are all configured in advance, and for example, a coding-decoding structure network with jump connection like U-Net may be selected. The color block image is an image obtained by roughly coloring the line contour of the line manuscript but without specific details. As shown, fig. 3a is a line image, fig. 3b is a color patch image, and fig. 3c is a color image finally colored. The coloring model of the example is designed by stages and multiple scales, so that the difficulty of coloring can be reduced at one time, and the quality of the colored image is improved.
S140: inputting the second coloring image and the real color image corresponding to the line draft into a style discriminator, inputting the second coloring image into a color discriminator, respectively acquiring style discrimination loss and color discrimination loss, obtaining reconstruction loss by calculating the mean square error of the second coloring image and the real color image corresponding to the second coloring image, adjusting the coloring model according to the style discrimination loss, the color discrimination loss and the reconstruction loss through a back propagation algorithm, and finishing the training of the coloring model.
Specifically, the real color image in this example refers to a reference image, i.e., an image artificially colored on a line manuscript. The reference image may be any style image, such as a Tibetan color drawing image, that is, the input style discriminator may be any image in the same style as any line of the reference image data set, such as any one of the real Tibetan color drawing image data sets. Inputting the second coloring image and the real color image I into a style discriminator DsInputting the second color image into a color discriminator D by constraining the color image generated by the style discrimination loss to be consistent with the style of the image in the data setcAnd the color image generated by restricting the color discrimination loss is uniformly colored and full in color.
In one example, style discriminator DsSpecifically, the method comprises w discrimination convolution modules consisting of convolution layer with convolution kernel size of 3x3 and step length of 2 and LeakyReLU activation function layer, and one convolution kernel with convolution kernel size of 3x3 and step length of 1And a Sigmoid activation function layer, w is 4 in this example.
In one example, color discriminator DcSpecifically, the system comprises v discrimination convolution modules consisting of convolution layers with convolution kernel size of 3x3 and step size of 2 and a LeakyReLU activation function layer, and an output module consisting of convolution layers with convolution kernel size of 3x3 and step size of 1 and a Sigmoid activation function layer, wherein v is 4 in the example. The color discriminator is pre-trained in advance, and is trained by inputting a real color image I and an image P with poor color (such as uneven coloring, overflow and unsaturated color), and the loss function is as follows:
LDC＝-(log(DCI)+log(1-DCP))
wherein DCIRepresenting the output, DC, of a true colour image I obtained by a colour discriminatorPRepresenting the output of the poor color image P obtained by the color discriminator.
The loss function to obtain the coloring model includes the reconstruction loss LreJudgment of style loss LstyleAnd color discrimination loss LcolorThen the loss function is formulated as follows:
Lre＝||C2-I||2
Figure BDA0003486918380000062
Lcolor＝1-Sco
wherein, DSIRepresenting the output of the true color image I obtained by the style discriminator,
Figure BDA0003486918380000061
representing a second coloring image C2Output obtained by the style discriminator; s. thecoRepresenting a second coloring image C2And the output is obtained after the color discriminator.
According to the method, the corresponding main color vector is obtained according to the line draft type, the given main color vector is subjected to multi-level coding to obtain the corresponding color code, the limited color space is constructed, the coloring model is configured in advance to color the line draft in the limited color space, different coloring models can be constructed according to different painting styles, then the line draft is colored, the coloring image with the specific painting style can be obtained, and the personalized requirement is met.
In one example, in step S130, inputting the line script and the color code into the color inference module, and obtaining the color block image specifically includes: for the input line draft S, the color reasoning module U is usedCObtaining a color block image ScdThe following formula:
Scd＝UC(S，CE)
in particular, the color reasoning module UCAs shown in fig. 2, the U-Net network with a residual block includes: m first convolution modules consisting of convolution layer with convolution kernel size of 3x3 step size of 2, example normalization layer and ReLU activation function layer, k second convolution modules consisting of convolution layer with convolution kernel size of 3x3 step size of 1, example normalization layer and ReLU activation function layer, 1 convolution kernel size of 1x1 step size of 1, third convolution module consisting of sigmoid activation function layer, k bilinear upsampling layers, 1 multiplication layer, m-1 residual blocks, m-1 deconvolution layer consisting of convolution layer size of 3x3 step size of 2, deconvolution layer of example normalization layer and ReLU activation function layer, and 1 deconvolution layer consisting of convolution kernel size of 3x3 step size of 2 and 1 tanh activation function layer, each residual block comprising convolution layer size of 3x3 step size of 1, connected in series in sequence, A ReLU activation function layer, an instance standardization layer and a shortcut connection; in particular, the output characteristics of the 7 th and third convolution modules can be regarded as the weight and the color code CEThe multiplication of the corresponding positions is equivalent to selecting a color suitable for the line draft in a given limited color space for subsequent coloring, the selected color characteristic is marked as SC, a 10, 12 and 14 characteristic diagram is obtained through a second convolution module, 10 is embedded into 16, 12 is embedded into 18, 14 is embedded into 20, the selected color information is embedded into the line draft through the three embedding operations, and the network efficiency can be further improved through the use of a residual block. Book (I)In the example, m is 6, k is 3, and the size of the input line script S is 256x256, where table 2 is the output size corresponding to each layer of the color inference module, and the sequence number in the table is the sequence in which the corresponding network structures are sequentially connected.
TABLE 2
Figure BDA0003486918380000071
Figure BDA0003486918380000081
In one example, in step S130, the line script S, the selected color feature SC and the patch image S in the color inference module are combinedcdInput coloring module GcObtaining a preliminary coloring image C1The formula is as follows:
C1＝GC(S，SC，Scd)
as a preferred scheme, before the line draft and the color block image are input into the coloring module, the line draft and the color block image are respectively subjected to down sampling. It should be noted that to construct a multi-scale coloring module, coloring is required to start from a small resolution, and then gradually enlarge, and gradually color from easy to difficult. Therefore, the line draft is firstly down-sampled to 4 times and 2 times of the original resolution, such as 256x256, and then reduced to 128x128 and 64x64, and then the 64x64 is colored first, then 128x128 is colored, and finally 256x256 is colored; wherein the coloring result of each size is used in the coloring of the next size.
Specifically, the coloring module comprises a plurality of groups of U-Net networks. Writing lines S and color block images ScdRespectively carrying out 2 times of downsampling and 4 times of downsampling to obtain a line draft S2x，S4xColor block image Scd2x，Scd4xDraft the line S4xAnd patch image Scd4xInput the coloring module G togethercU-Net1, and obtaining the output SC of 8 in the color inference module through a bilinear upsampling layer, a convolution layer with convolution kernel size of 3x3 and step length of 1, and a ReLU activation functionThe color feature SC1 with the size of 8x8x128 is embedded into a 2 nd deconvolution module, and the result obtained by U-Net1 is up-sampled by 2 times and then is summed into a line script S2xAnd color patch image Scd2xInput coloring module G togethercThe U-Net2 and the color feature SC1 are simultaneously embedded into the 2 nd deconvolution module, and the obtained result is up-sampled by 2 times and is summed with the line draft S and the color block image ScdInput coloring module G togethercIn U-Net3, color feature SC1 passes through a convolution layer with convolution kernel size of 1x1 and step size of 1 and a ReLU activation function to obtain color feature SC2 with size of 8x8x512, and the color feature SC2 is embedded into a 2 nd deconvolution module to obtain a preliminary coloring image C1. It should be noted that there is a separate network for each scale, the U-Net network itself is a common structure in the art, and the coloring module may also use other structures as long as it can complete encoding-decoding and has a larger receptive field and has a jump connection, for example, an encoder-decoder structure with a hole convolution, and is not intended to limit the scope of the present invention.
The U-Net1 specifically comprises: m is1A convolution module consisting of convolution layer with convolution kernel size of 3 × 3 and step size of 2, example normalization layer and ReLU activation function layer, m11 deconvolution module consisting of deconvolution layers of convolution kernel size 3x3 step size 2, an example normalization layer and a ReLU activation function layer, and 1 deconvolution layer of convolution kernel size 3x3 step size 2 and 1 tanh activation function layer, m in this example1Is 5; table 3 shows the corresponding output sizes of the respective layers of U-Net 1.
TABLE 3
Convolutional layer, instance normalization, ReLU 32x32x32
Convolutional layer, instance normalization, ReLU 16x16x64
Convolutional layer, instance normalization, ReLU 8x8x128
Convolutional layer, instance normalization, ReLU 4x4x128
Convolutional layer, instance normalization, ReLU 2x2x128
Deconvolution, instance normalization, ReLU 4x4x128
Deconvolution, instance normalization, ReLU 8x8x128
Deconvolution, instance normalization, ReLU 16x16x64
Deconvolution, instance normalization, ReLU 32x32x32
Deconvolution layer, tanh 64x64x3
The U-Net2 specifically comprises: m is2A convolution module consisting of convolution layer with convolution kernel size of 3x3 step size of 2, example standard layer and ReLU activation function layer, m21 deconvolution module consisting of deconvolution layers with convolution kernel size 3x3 step size 2, example normalization layer and ReLU activation function layer, and 1 deconvolution layer with convolution kernel size 3x3 step size 2 and 1Layer of tan h activation function, m in this example2Is 6; where table 4 is the corresponding output size of each layer of U-Net 2.
TABLE 4
Figure BDA0003486918380000091
Figure BDA0003486918380000101
The U-Net3 specifically comprises: m is3A convolution module consisting of convolution layer with convolution kernel size of 3x3 and step size of 2, example normalization layer and ReLU activation function layer, m31 deconvolution module consisting of deconvolution layers with convolution kernel size 3x3 step size 2, example normalization layer and ReLU activation function layer, and 1 deconvolution layer with convolution kernel size 3x3 step size 2 and 1 tanh activation function layer, m in this example3Is 7; table 5 shows the corresponding output sizes of the respective layers.
TABLE 5
Convolutional layer, instance normalization, ReLU 128x128x64
Convolutional layer, instance normalization, ReLU 64x64x128
Convolutional layer, example normalization, ReLU 32x32x256
Convolutional layer, instance normalization, ReLU 16x16x512
Convolutional layer, instance normalization, ReLU 8x8x512
Convolutional layer, example normalization, ReLU 4x4x512
Convolutional layer, instance normalization, ReLU 2x2x512
Deconvolution, instance normalization, ReLU 4x4x512
Deconvolution, instance normalization, ReLU 8x8x512
Deconvolution, instance normalization, ReLU 16x16x512
Deconvolution, instance normalization, ReLU 32x32x256
Deconvolution, instance normalization, ReLU 64x64x128
Deconvolution, example normalization, ReLU 128x128x64
Deconvolution layer, tanh 256x256x3
In one example, in step S130, inputting the first color image into a color correction module for calibration, and obtaining the second color image specifically includes: the generated preliminary coloring image C1And inputting the color correction module P to obtain a second coloring image. The color correction module P specifically includes a convolution module consisting of h convolution layers with convolution kernel size of 3x3 and step size of 1 and a ReLU activation function layer, 1 convolution layer with convolution kernel size of 3x3 and step size of 1 and 1 tanh activation function layer, where h is 6 in this example.
In one example, there is also provided an automatic coloring method for line art in a limited color space, the method including the steps of:
acquiring a line draft to be colored; the line art refers to an image that needs to be colored. The line draft type mainly refers to the type of line draft needing coloring. And inputting the line draft to be colored into the coloring model to obtain a colored image. This example can realize automatic coloring, only needs the user to give the primary color, utilizes the model of coloring of training completion, then accomplishes automatically and colors for supplementary painter colors.
Exemplary devices
As shown in fig. 6, an apparatus for training a line art automatic coloring model under a limited color space includes:
a color obtaining module 20, configured to obtain a corresponding primary color vector according to the line draft type;
a color coding module 30, configured to perform multi-level coding on the given dominant color vector to obtain a corresponding color code, and construct a limited color space;
the model configuration module 40 is used for configuring a coloring model in advance, the coloring model comprises a color reasoning module, a coloring module and a color correction module, and the color reasoning module is used for receiving line draft and carrying out color coding processing to obtain a color block image; the coloring module is used for receiving the line draft and the color block image for processing to obtain a first coloring image; the color correction module is used for receiving the first coloring image for calibration to obtain a second coloring image;
and the model calibration module 50 is configured to input the real color image corresponding to the line draft into a style discriminator, input the second coloring image into a color discriminator, obtain a style discrimination loss and a color discrimination loss respectively, obtain a reconstruction loss by calculating a mean square error of the second coloring image and the corresponding real color image, adjust the coloring model according to the style discrimination loss, the color discrimination loss and the reconstruction loss by using a back propagation algorithm, and complete the coloring model training.
Preferably, the loss function of the coloring model includes a reconstruction loss LreJudgment of style loss LstyleAnd color discrimination loss LcolorThen the loss function is formulated as follows:
Lre＝||C2-I||2
Figure BDA0003486918380000112
Lcolor＝1-Sco
wherein DSIRepresenting the output of the true color image I obtained by the style discriminator,
Figure BDA0003486918380000111
representing a second coloring image C2Output obtained by the style discriminator; scoRepresenting a second coloring image C2And the output is obtained after the color discriminator.
Exemplary electronic device
Next, an electronic apparatus according to an embodiment of the present application is described with reference to fig. 4. The electronic device may be the mobile device itself, or a stand-alone device separate therefrom, which may communicate with the mobile device to receive the collected input signals therefrom and to transmit the selected goal decision behavior thereto.
FIG. 4 illustrates a block diagram of an electronic device in accordance with an embodiment of the present application.
As shown in fig. 4, the electronic device 10 includes one or more processors 11 and memory 12.
The processor 11 may be a Central Processing Unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.
Memory 12 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, Random Access Memory (RAM), cache memory (cache), and/or the like. The non-volatile memory may include, for example, Read Only Memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium and executed by the processor 11 to implement the decision-making behavior decision-making methods of the various embodiments of the present application described above and/or other desired functionality.
In one example, the electronic device 10 may further include: an input device 13 and an output device 14, which are interconnected by a bus system and/or other form of connection mechanism (not shown). For example, the input device 13 may include various devices such as an on-board diagnostic system (OBD), a Universal Diagnostic Service (UDS), an Inertial Measurement Unit (IMU), a camera, a lidar, a millimeter-wave radar, an ultrasonic radar, an on-board communication (V2X), and the like. The input device 13 may also include, for example, a keyboard, a mouse, and the like. The output devices 14 may include, for example, a display, speakers, a printer, and a communication network and its connected remote output devices, among others.
Of course, for simplicity, only some of the components of the electronic device 10 relevant to the present application are shown in fig. 4, omitting components such as buses, input/output interfaces, and the like. In addition, the electronic device 10 may include any other suitable components depending on the particular application.
Exemplary computer program product and computer-readable storage Medium
In addition to the above-described methods and apparatus, embodiments of the present application may also be a computer program product comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in a decision-making behavior decision-making method according to various embodiments of the present application described in the "exemplary methods" section of this specification above.
The computer program product may be written with program code for performing the operations of embodiments of the present application in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.
Furthermore, embodiments of the present application may also be a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform steps in a decision-making behavior decision method according to various embodiments of the present application, described in the "exemplary methods" section above of this specification.
The computer-readable storage medium may take any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The foregoing describes the general principles of the present application in conjunction with specific embodiments, however, it is noted that the advantages, effects, etc. mentioned in the present application are merely examples and are not limiting, and they should not be considered essential to the various embodiments of the present application. Furthermore, the foregoing disclosure of specific details is for the purpose of illustration and description and is not intended to be limiting, since the foregoing disclosure is not intended to be exhaustive or to limit the disclosure to the precise details disclosed.
The block diagrams of devices, apparatuses, devices, systems referred to in this application are only used as illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. These devices, apparatuses, devices, systems may be connected, arranged, configured in any manner, as will be appreciated by those skilled in the art. Words such as "including," "comprising," "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably therewith. The words "or" and "as used herein mean, and are used interchangeably with, the word" and/or, "unless the context clearly dictates otherwise. The word "such as" is used herein to mean, and is used interchangeably with, the phrase "such as but not limited to".
It should also be noted that in the devices, apparatuses, and methods of the present application, the components or steps may be decomposed and/or recombined. These decompositions and/or recombinations should be considered as equivalents of the present application.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing description has been presented for purposes of illustration and description. Furthermore, the description is not intended to limit embodiments of the application to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.

Claims (10)

1. A line draft automatic coloring model training method under a limited color space is characterized by comprising the following steps:
acquiring a corresponding primary color vector according to the line draft type;
carrying out multi-level coding on the main color vector to obtain corresponding color codes and constructing a limited color space;
the method comprises the steps of configuring a coloring model in advance, wherein the coloring model comprises a color reasoning module, a coloring module and a color correcting module, inputting a line draft and a color code into the color reasoning module to obtain a color block image, inputting the line draft and the color block image into the coloring module to obtain a first coloring image, and inputting the first coloring image into the color correcting module to be calibrated to obtain a second coloring image;
inputting the second coloring image and the real color image corresponding to the line draft into a style discriminator to obtain style discrimination loss, inputting the second coloring image into a color discriminator to obtain color discrimination loss, obtaining reconstruction loss by calculating the mean square error of the second coloring image and the real color image corresponding to the second coloring image, and adjusting the coloring model according to the style discrimination loss, the color discrimination loss and the reconstruction loss through a back propagation algorithm to finish the training of the coloring model.
2. The method for training line art automatic coloring model in limited color space according to claim 1, wherein said obtaining corresponding primary color vector comprises: and clustering the colors of the sample images by a k-means algorithm to obtain the main color.
3. The method for training the line draft automatic coloring model under the limited color space of claim 2, wherein the line draft and the color block image are respectively down-sampled and then input into a coloring module for multi-scale coloring to obtain a first colored image.
4. The method of claim 1, wherein the loss function of the coloring model comprises a reconstruction loss LreJudgment of style loss LstyleAnd color discrimination loss LcolorThen the loss function is formulated as follows:
Lre＝||C2-I||2
Figure FDA0003486918370000011
Lcolor＝1-Sco
wherein I denotes a real color image, DSIRepresenting the output of the true color image I obtained by the style discriminator,
Figure FDA0003486918370000012
representing a second coloring image C2Output obtained by the style discriminator; wherein ScoRepresenting a second coloring image C2And the output is obtained after the color discriminator.
5. The method for training the line draft automatic coloring model under the limited color space according to claim 1, wherein the given primary color vector is subjected to multi-level coding by a preset color coding module;
the color coding module comprises 1 size adjustment layer, n convolution layers with convolution kernel size of 3x3 and step size of 1, and a ReLU activation function layer.
6. An automatic coloring method for line draft in limited color space is characterized by comprising the following steps:
acquiring a line draft to be colored;
inputting the line manuscript to be colored into the coloring model according to any one of claims 1 to 5 to obtain a coloring image.
7. The utility model provides a line manuscript automatic model training device that colors under limited color space which characterized in that, the device includes:
the color acquisition module is used for acquiring a corresponding primary color vector according to the line draft type;
the color coding module is used for carrying out multi-level coding on the main color vector to obtain corresponding color coding and construct a limited color space;
the model configuration module is used for configuring a coloring model in advance, the coloring model comprises a color reasoning module, a coloring module and a color correction module, and the color reasoning module is used for receiving line draft and carrying out color coding processing to obtain a color block image; the coloring module is used for receiving the line draft and the color block image for processing to obtain a first coloring image; the color correction module is used for receiving the first coloring image for calibration to obtain a second coloring image;
and the model calibration module is used for inputting the second coloring image and any real color image into the style discriminator, inputting the second coloring image into the color discriminator, respectively acquiring style discrimination loss and color discrimination loss, obtaining reconstruction loss by calculating the mean square error of the second coloring image and the corresponding real color image, adjusting the coloring model according to the style discrimination loss, the color discrimination loss and the reconstruction loss through a back propagation algorithm, and finishing the training of the coloring model.
8. The device for training the line draft automatic coloring model in the limited color space according to claim 7, wherein the loss function of the coloring model comprises a reconstruction loss LreJudgment of style loss LstyleAnd color discrimination loss LcolorThen the loss function is formulated as follows:
Lre＝||C2-I||2
Figure FDA0003486918370000021
Lcolor＝1-Sco
wherein I denotes a real color image, DSIRepresenting the output of the true color image I obtained by the style discriminator,
Figure FDA0003486918370000022
representing a second coloring image C2Output obtained by the style discriminator; wherein ScoRepresenting a second coloring image C2And the output is obtained after the color discriminator.
9. An electronic device comprising a processor, an input device, an output device, and a memory, the processor, the input device, the output device, and the memory being connected in series, the memory being configured to store a computer program comprising program instructions, the processor being configured to invoke the program instructions to perform the method of any of claims 1-6.
10. A readable storage medium, characterized in that the storage medium stores a computer program comprising program instructions which, when executed by a processor, cause the processor to carry out the method according to any one of claims 1-6.
CN202210084599.6A 2022-01-25 2022-01-25 Line manuscript automatic coloring model training method and device under limited color space, electronic equipment and storage medium Active CN114494523B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210084599.6A CN114494523B (en) 2022-01-25 2022-01-25 Line manuscript automatic coloring model training method and device under limited color space, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210084599.6A CN114494523B (en) 2022-01-25 2022-01-25 Line manuscript automatic coloring model training method and device under limited color space, electronic equipment and storage medium

Publications (2)

Publication Number Publication Date
CN114494523A true CN114494523A (en) 2022-05-13
CN114494523B CN114494523B (en) 2024-07-02

Family

ID=81475447

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210084599.6A Active CN114494523B (en) 2022-01-25 2022-01-25 Line manuscript automatic coloring model training method and device under limited color space, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN114494523B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116416342A (en) * 2023-06-12 2023-07-11 腾讯科技（深圳）有限公司 Image processing method, apparatus, computer device, and computer-readable storage medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108615252A (en) * 2018-05-03 2018-10-02 苏州大学 The training method and device of color model on line original text based on reference picture
CN110223359A (en) * 2019-05-27 2019-09-10 浙江大学 It is a kind of that color model and its construction method and application on the dual-stage polygamy colo(u)r streak original text of network are fought based on generation
CN110378985A (en) * 2019-07-19 2019-10-25 中国传媒大学 A kind of animation drawing auxiliary creative method based on GAN
CN111553837A (en) * 2020-04-28 2020-08-18 武汉理工大学 Artistic text image generation method based on neural style migration
WO2021114184A1 (en) * 2019-12-12 2021-06-17 华为技术有限公司 Neural network model training method and image processing method, and apparatuses therefor
US20210286975A1 (en) * 2020-08-20 2021-09-16 Beijing Baidu Netcom Science And Technology Co., Ltd. Image processing method, electronic device, and storage medium

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108615252A (en) * 2018-05-03 2018-10-02 苏州大学 The training method and device of color model on line original text based on reference picture
CN110223359A (en) * 2019-05-27 2019-09-10 浙江大学 It is a kind of that color model and its construction method and application on the dual-stage polygamy colo(u)r streak original text of network are fought based on generation
CN110378985A (en) * 2019-07-19 2019-10-25 中国传媒大学 A kind of animation drawing auxiliary creative method based on GAN
WO2021114184A1 (en) * 2019-12-12 2021-06-17 华为技术有限公司 Neural network model training method and image processing method, and apparatuses therefor
CN111553837A (en) * 2020-04-28 2020-08-18 武汉理工大学 Artistic text image generation method based on neural style migration
US20210286975A1 (en) * 2020-08-20 2021-09-16 Beijing Baidu Netcom Science And Technology Co., Ltd. Image processing method, electronic device, and storage medium

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
叶雨琪;胡文瑾;: "基于多特征的彩色唐卡修复图像无参考质量评价方法", 激光与光电子学进展, no. 08, 31 December 2020 (2020-12-31) *
朱松;陈昭炯;叶东毅;: "场景线稿动漫效果的自动上色算法", 模式识别与人工智能, no. 08, 15 August 2020 (2020-08-15) *
李炬;: "基于生成对抗网络的人脸灰度图上色", 电脑知识与技术, no. 11, 15 April 2018 (2018-04-15) *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116416342A (en) * 2023-06-12 2023-07-11 腾讯科技（深圳）有限公司 Image processing method, apparatus, computer device, and computer-readable storage medium
CN116416342B (en) * 2023-06-12 2023-09-19 腾讯科技（深圳）有限公司 Image processing method, apparatus, computer device, and computer-readable storage medium

Also Published As

Publication number Publication date
CN114494523B (en) 2024-07-02

Similar Documents

Publication Publication Date Title
Zhao et al. Pixelated semantic colorization
US10922860B2 (en) Line drawing generation
CN113591546B (en) Semantic enhancement type scene text recognition method and device
CN112381897B (en) Low-illumination image enhancement method based on self-coding network structure
CN110678899B (en) Imaging system and image processing method
CN109948714A (en) Chinese scene text row recognition methods based on residual error convolution sum recurrent neural network
CN110163884B (en) Single image segmentation method based on full-connection deep learning neural network
WO2023000895A1 (en) Image style conversion method and apparatus, electronic device and storage medium
CN110738102A (en) face recognition method and system
CN109961102B (en) Image processing method, image processing device, electronic equipment and storage medium
CN108921932B (en) Method for generating multiple reasonable colorings of black and white figure pictures based on convolutional neural network
CN110717868A (en) Video high dynamic range inverse tone mapping model construction and mapping method and device
CN113052775B (en) Image shadow removing method and device
WO2020238120A1 (en) System and method for single-modal or multi-modal style transfer and system for random stylization using the same
Liu et al. Very lightweight photo retouching network with conditional sequential modulation
US11138693B2 (en) Attention-driven image manipulation
CN115731597A (en) Automatic segmentation and restoration management platform and method for mask image of face mask
Cotogni et al. TreEnhance: A tree search method for low-light image enhancement
CN114494523A (en) Line draft automatic coloring model training method and device under limited color space, electronic equipment and storage medium
CN113436081A (en) Data processing method, image enhancement method and model training method thereof
Bugeau et al. Influence of color spaces for deep learning image colorization
CN112270404A (en) Detection structure and method for bulge defect of fastener product based on ResNet64 network
CN115018729B (en) Content-oriented white box image enhancement method
CN116524352A (en) Remote sensing image water body extraction method and device
CN111161266A (en) Multi-style font generation method of variational self-coding machine based on vector quantization

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant