CN114494523B - Line manuscript automatic coloring model training method and device under limited color space, electronic equipment and storage medium - Google Patents

Line manuscript automatic coloring model training method and device under limited color space, electronic equipment and storage medium Download PDF

Info

Publication number
CN114494523B
CN114494523B CN202210084599.6A CN202210084599A CN114494523B CN 114494523 B CN114494523 B CN 114494523B CN 202210084599 A CN202210084599 A CN 202210084599A CN 114494523 B CN114494523 B CN 114494523B
Authority
CN
China
Prior art keywords
color
coloring
image
module
style
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202210084599.6A
Other languages
Chinese (zh)
Other versions
CN114494523A (en
Inventor
陈缘
赵洋
曹力
李琳
谢文军
刘晓平
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hefei University of Technology
Original Assignee
Hefei University of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hefei University of Technology filed Critical Hefei University of Technology
Priority to CN202210084599.6A priority Critical patent/CN114494523B/en
Publication of CN114494523A publication Critical patent/CN114494523A/en
Application granted granted Critical
Publication of CN114494523B publication Critical patent/CN114494523B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/40—Filling a planar surface by adding surface attributes, e.g. colour or texture
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/23—Clustering techniques
    • G06F18/232—Non-hierarchical techniques
    • G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
    • G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a line manuscript automatic coloring model training method and device under a limited color space, electronic equipment and a storage medium, and belongs to the technical field of image processing. According to the method, a corresponding dominant color matrix is obtained as a dominant color vector according to the type of a manuscript, the given dominant color vector is subjected to multi-level coding to obtain a corresponding color code, a limited color space is built, a coloring model is pre-configured to color the manuscript in the limited color space, different coloring models are built aiming at different painting styles, the manuscript is further colored, a coloring image with a specific painting style can be obtained, and personalized coloring requirements are met.

Description

Line manuscript automatic coloring model training method and device under limited color space, electronic equipment and storage medium
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to a line manuscript automatic coloring model training method and device under a limited color space, electronic equipment and a storage medium.
Background
Traditional draft coloring generally refers to a process of manually painting reasonable and aesthetic colors on draft by a painter, such as coloring a draft of a picture by a painter, coloring a split-mirror draft in an animation production process, and the like. However, these purely manual processes are not only demanding for the painter, but are also time-and labor-consuming processes. Therefore, the automatic coloring of the line manuscript has important application prospect.
In recent years, the fields of computer vision, image processing and the like develop rapidly, and particularly, a deep learning method can intelligently solve some problems, but related coloring applications mainly include gray level image coloring and line manuscript coloring with reference images or manual interaction. For example, a method of coloring a document image is disclosed; the method comprises the following steps: acquiring a first manuscript image and a first reference image, wherein the first reference image comprises image information which is referred when the first manuscript image is colored; and coloring the first document image based on the first reference image by using a model based on machine learning training.
Although the methods can obtain good coloring effect, manual participation is still needed, and the coloring task of the manuscript cannot be automatically and efficiently completed. And the computer is enabled to fully and automatically color the line manuscript, which is a difficult task, and unreasonable coloring results which do not accord with the aesthetic sense of people are easy to appear. For example, a red flag line manuscript is colored, and a deep network lacks sufficient semantic analysis capability and may be colored in various colors, which does not necessarily meet the specified requirements. In practice, there are many situations where a limited color space is used, such as drawing of a specified style, artistic creation, a specific cartoon animation, etc., often with a fixed color style prior. Therefore, the above method is difficult to learn the painting style thereof, resulting in a large difference in coloring effect from the artificial coloring result.
Disclosure of Invention
Aiming at the problem that the coloring effect of an image with a specified painting style is poor in the prior art, the invention provides a training method of an automatic coloring model of a line manuscript under a limited color space.
In order to solve the problems, the invention adopts the following technical scheme.
The first aspect of the invention provides a line manuscript automatic coloring model training method under a limited color space, which comprises the following steps:
Acquiring a corresponding main color vector according to the line manuscript type;
Performing multi-level coding on the main color vector to obtain corresponding color codes, and constructing a limited color space;
a coloring model is pre-configured, the coloring model comprises a color reasoning module, a coloring module and a color correction module, a line manuscript and a color code are input into the color reasoning module to obtain a color block image, the line manuscript and the color block image are input into the coloring module to obtain a first coloring image, and the first coloring image is input into the color correction module to be calibrated to obtain a second coloring image;
Inputting the second colored image and the real color image corresponding to the line manuscript into a style discriminator to obtain style discrimination loss, inputting the second colored image into a color discriminator to obtain color discrimination loss, obtaining reconstruction loss by calculating the mean square error of the second colored image and the real color image corresponding to the second colored image, and adjusting the coloring model through a counter propagation algorithm according to the style discrimination loss, the color discrimination loss and the reconstruction loss to complete training of the coloring model.
Preferably, the obtaining the corresponding dominant color vector includes: and clustering the sample image colors through a k-means algorithm to obtain main colors.
As a preferred scheme, downsampling is carried out on the line manuscript and the color lump image respectively, and then the line manuscript and the color lump image are input into a coloring module for multi-scale coloring, so as to obtain a first coloring image.
Preferably, the loss function of the coloring model includes a reconstruction loss L re, a style discrimination loss L style and a color discrimination loss L color, and the loss function is as follows:
Lre＝||C2-I||2
Lcolor＝1-Sco
wherein I represents a true color image, DS I represents an output of the true color image I obtained by a style discriminator, Representing the output of the second colored image C 2 obtained by the style discriminator; where S co represents the output of the second color image C 2 obtained by passing through the color discriminator.
As a preferred scheme, the given main color vector is subjected to multi-level coding through a preset color coding module;
The color coding module comprises 1 size-adjusting layer, n convolution layers with a convolution kernel size of 3x3 and a step size of 1 and a ReLU activation function layer.
The second aspect of the present invention provides a method for automatically coloring a line manuscript in a limited color space, the method comprising the steps of:
acquiring a line manuscript to be colored;
Inputting the draft to be colored into a coloring model according to any one of claims 1-5 to obtain a colored image.
A third aspect of the present invention provides an apparatus for training an automatic line manuscript coloring model in a limited color space, the apparatus comprising:
the color acquisition module is used for acquiring a corresponding main color vector according to the line manuscript type;
The color coding module is used for carrying out multi-level coding on the main color vector to obtain corresponding color codes and constructing a limited color space;
The model configuration module is used for pre-configuring a coloring model, wherein the coloring model comprises a color reasoning module, a coloring module and a color correction module, and the color reasoning module is used for receiving a line manuscript and performing color coding processing to obtain a color block image; the coloring module is used for receiving the line manuscript and the color block image for processing to obtain a first coloring image; the color correction module is used for receiving the first coloring image for calibration and obtaining a second coloring image;
The model calibration module is used for inputting the second coloring image and any real color image into a style discriminator, inputting the second coloring image into a color discriminator, respectively obtaining style discrimination loss and color discrimination loss, obtaining reconstruction loss by calculating the mean square error of the second coloring image and the corresponding real color image, and adjusting the coloring model through a counter propagation algorithm according to the style discrimination loss, the color discrimination loss and the reconstruction loss to finish the training of the coloring model.
Preferably, the loss function of the coloring model includes a reconstruction loss L re, a style discrimination loss L style and a color discrimination loss L color, and the loss function is as follows:
Lre＝||C2-I||2
Lcolor＝1-Sco
wherein I represents a true color image, DS I represents an output of the true color image I obtained by a style discriminator, Representing the output of the second colored image C 2 obtained by the style discriminator; where S co represents the output of the second color image C 2 obtained by passing through the color discriminator.
A fourth aspect of the present invention provides an electronic device comprising a processor, an input device, an output device and a memory, the processor, the input device, the output device and the memory being connected in sequence, the memory being for storing a computer program comprising program instructions, the processor being configured to invoke the program instructions to perform the method described above.
A fifth aspect of the present invention provides a readable storage medium storing a computer program comprising program instructions which, when executed by a processor, cause the processor to perform the method described above.
Compared with the prior art, the invention has the beneficial effects that:
(1) According to the method, a corresponding main color vector is obtained according to the type of the manuscript, the given main color vector is subjected to multi-level coding to obtain a corresponding color code, a limited color space is built, a coloring model is pre-configured to color the manuscript in the limited color space, different coloring models can be built aiming at different painting styles, the manuscript is further colored, a colored image with a specific painting style can be obtained, personalized requirements are met, and the unreasonable coloring problem existing in the current automatic coloring is solved;
(2) The coloring model comprises a color reasoning module, a coloring module and a color correction module for coloring the manuscript by a design of multiple scales in stages; the difficulty of finishing coloring at one time is reduced through the design of multiple scales in stages, and the quality of the colored image is improved;
(3) The invention is exemplified by inputting a line manuscript and a color code into a color reasoning module, roughly coloring the outline of the line manuscript by selecting proper colors in a limited color space constructed by the color code, enabling the obtained color block image to be convenient for finer coloring in the subsequent stage, inputting the line manuscript and the color block image into a coloring module to obtain a first coloring image, enabling the coloring module to adopt a multi-scale structure, coloring a small-scale image, gradually amplifying the image to color, obtaining a finer coloring result in a coarse-to-fine mode, inputting the first coloring image into a color correction module to calibrate, optimizing the coloring result, and obtaining a second coloring image; in addition, the example of the invention can realize automatic coloring, and the coloring is automatically finished only by giving the main color by a user and utilizing the trained coloring model, so as to assist a painter in coloring.
Drawings
The above and other objects, features and advantages of the present application will become more apparent by describing embodiments of the present application in more detail with reference to the attached drawings. The accompanying drawings are included to provide a further understanding of embodiments of the application and are incorporated in and constitute a part of this specification, illustrate the application and together with the embodiments of the application, and not constitute a limitation to the application. In the drawings, like reference numerals generally refer to like parts or steps. In the accompanying drawings:
FIG. 1 is a flow chart of a method for training an automatic line manuscript coloring model in a limited color space according to an embodiment of the invention;
FIG. 2 is a block diagram of a color reasoning module according to an embodiment of the invention;
FIG. 3 is a schematic diagram of a document coloring process according to an embodiment of the present invention;
FIG. 4 illustrates a block diagram of an electronic device in accordance with an embodiment of the application;
FIG. 5 is a block diagram of a coloring module according to an embodiment of the invention;
Fig. 6 is a block diagram of an automatic line manuscript coloring model training device under a limited color space according to an embodiment of the present invention.
Detailed Description
Hereinafter, exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. It should be apparent that the described embodiments are only some embodiments of the present application and not all embodiments of the present application, and it should be understood that the present application is not limited by the example embodiments described herein.
Exemplary method
As shown in fig. 1, the present embodiment provides a method for training an automatic line manuscript coloring model in a limited color space, which includes the following steps:
s110: and obtaining a corresponding main color vector according to the line manuscript type.
Specifically, in this example, a line manuscript refers to an image that needs to be colored. The line manuscript type mainly refers to the type of line manuscript to be colored, and takes the hidden colored drawing line manuscript as an example, the hidden colored drawing is colored by using local mineral plant pigment, and has a plurality of main colors which are relatively fixed, such as red, blue, yellow and green. The RGB pixel values of the main color can be given to form a three-channel color matrix, and the corresponding main color vector is obtained. When the line manuscript is extracted, the actual color image I is extracted by using the existing edge extraction algorithm or photoshop to serve as the line manuscript, and a training set is constructed.
It should be understood that the acquisition of the corresponding dominant color vector may be manually given, or may be acquired by selecting the dominant color through color histogram statistics according to a large number of templates, or clustering the colors of the images of the templates by using a k-means algorithm to obtain the dominant color.
S120: and carrying out multi-level coding on the main color vector to obtain corresponding color codes, and constructing a limited color space.
Specifically, taking a hidden colored drawing draft as an example, the main colors can be red, blue, yellow and green, and the main color vector is encoded for multiple times by a preset color encoding module E C to obtain color code C E. The step is mainly to obtain color information in a hidden space by carrying out layer-by-layer coding on the color vectors, and construct a limited color space for guiding the subsequent coloring. For example, taking a hidden colored draft as an example, the primary colors in the limited color space include red, blue, yellow, and green, and the subsequent coloring process will select only the appropriate colors for use.
As a possible implementation, the color coding module E C includes 1 resizing layer, n convolution layers with a convolution kernel size of 3x3 steps of 1 and a ReLU activation function layer, the color vector size input in this example is 16x3, the adjusted size is 4x4x3, n is 4, and the color code C E size is 4x4x512; wherein table 1 is the output size corresponding to each layer of the color coding module.
TABLE 1
Size adjusting layer 4x4x3
Convolution layer with convolution kernel size of 3x3 step length of 1 and ReLU 4x4x64
Convolution layer with convolution kernel size of 3x3 step length of 1 and ReLU 4x4x128
Convolution layer with convolution kernel size of 3x3 step length of l and ReLU 4x4x256
Convolution layer with convolution kernel size of 3x3 step length of 1 and ReLU 4x4x512
S130: the method comprises the steps of configuring a coloring model in advance, wherein the coloring model comprises a color reasoning module, a coloring module and a color correction module, inputting a line manuscript and a color code into the color reasoning module to obtain a color block image, inputting the line manuscript and the color block image into the coloring module to obtain a first coloring image, and inputting the first coloring image into the color correction module to calibrate to obtain a second coloring image.
Specifically, the color reasoning module, the coloring module and the color correction module in this example are all preconfigured, and for example, a coding-decoding structure network with jump connection like U-Net can be selected. A patch image is an image in which the line outline of a document is roughly colored, but does not have specific details. As shown, fig. 3a is a document image, fig. 3b is a patch image, and fig. 3c is a final colored color image. The coloring model of the example is designed in a staged multi-scale mode, so that the difficulty in completing coloring at one time can be reduced, and the quality of an image after coloring is improved.
S140: and inputting the second coloring image and the real color image corresponding to the line manuscript into a style discriminator, inputting the second coloring image into a color discriminator, respectively obtaining style discrimination loss and color discrimination loss, obtaining reconstruction loss by calculating the mean square error of the second coloring image and the real color image corresponding to the second coloring image, and adjusting the coloring model through a counter propagation algorithm according to the style discrimination loss, the color discrimination loss and the reconstruction loss to finish training of the coloring model.
Specifically, the true color image in this example refers to a reference image, i.e., an image that is manually colored to a document. The reference image may be any style image, such as a Tibetan colored drawing image, that is, the input style identifier may be any image with the same style as any line draft in the reference image dataset, such as any image in the real Tibetan colored drawing image dataset. The second coloring image and the real color image I are input into a style discriminator D s, the generated color image is restrained to be consistent with the image style in the data set through style discrimination loss, the second coloring image is input into a color discriminator D c, and the generated color image is restrained to be uniformly colored and full in color through color discrimination loss.
In one example, the style discriminator D s specifically includes w discriminating convolution modules composed of a convolution layer with a 3x3 step size of 2 and a LeakyReLU activation function layer, and an output module composed of a convolution layer with a 3x3 step size of 1 and a Sigmoid activation function layer, where w is 4.
In one example, the color discriminator D c specifically includes v discriminating convolution modules composed of a convolution layer with a 3x3 step size of 2 and a LeakyReLU activation function layer, and an output module composed of a convolution layer with a 3x3 step size of 1 and a Sigmoid activation function layer, where v is 4. The color discriminator is pre-trained in advance, training is carried out by inputting a real color image I and an image P with poorer colors (such as uneven color, overflowed color and not full color), and the loss function is as follows:
LDC＝-(log(DCI)+log(1-DCP))
Where DC I represents the output of the true color image I via the color discriminator, and DC P represents the output of the color poor image P via the color discriminator.
The loss function of the obtained coloring model includes reconstruction loss L re, style discrimination loss L style and color discrimination loss L color, and the loss function formula is as follows:
Lre＝||C2-I||2
Lcolor＝1-Sco
Wherein DS I represents the output of the real color image I obtained by the style discriminator, Representing the output of the second colored image C 2 obtained by the style discriminator; s co represents the output of the second color image C 2 obtained by passing through the color discriminator.
According to the method, corresponding main color vectors are obtained according to the types of the manuscript, multi-level coding is carried out on the given main color vectors to obtain corresponding color codes, a limited color space is built, a coloring model is pre-configured to color the manuscript in the limited color space, different coloring models can be built according to different painting styles, the manuscript is further colored, a colored image with a specific painting style can be obtained, and personalized requirements are met.
In one example, in step S130, inputting the line script and the color code into the color reasoning module, the obtaining the color patch image specifically includes: for the input line manuscript S, a color block image S cd is obtained through a color reasoning module U C, and the following formula is given:
Scd＝UC(S,CE)
specifically, as shown in fig. 2, the color reasoning module U C includes a U-Net network with a residual block, where the U-Net network with a residual block in this example specifically includes: m first convolution modules consisting of convolution layers with the size of 3x3 step sizes, an example standardization layer and a ReLU activation function layer, k second convolution modules consisting of convolution layers with the size of 3x3 step sizes, an example standardization layer and a ReLU activation function layer, 1 convolution layer with the size of 1x1 step sizes, a third convolution module consisting of a sigmoid activation function layer, k bilinear upsampling layers, 1 multiplication layer, m-1 residual blocks, m-1 deconvolution modules consisting of deconvolution layers with the size of 3x3 step sizes, an example standardization layer and a ReLU activation function layer, and 1 deconvolution layer with the size of 3x3 step sizes and 1 tanh activation function layer, wherein each residual block comprises the convolution layers with the size of 3x3 step sizes, the ReLU activation function layer and the example standardization layer which are connected in series in sequence; particularly, the output characteristics of the 7 th third convolution module can be regarded as weights, the weights are multiplied by the positions corresponding to the color codes C E, the colors suitable for the line manuscript are selected in a given limited color space for subsequent coloring, the selected color characteristics are marked as SC, the 10, 12 and 14 characteristic diagrams are obtained through the second convolution module, the 10 embedding 16, the 12 embedding 18 and the 14 embedding 20 are obtained through the second convolution module, the selected color information is embedded into the line manuscript through the three embedding operations, and the network efficiency can be further improved through the use of residual blocks. In this example, m is 6,k and is 3, and the size of the input line manuscript S is 256x256, where table 2 is the output size corresponding to each layer of the color reasoning module, and the sequence number in the table is the sequence in which the corresponding network structures are sequentially connected.
TABLE 2
In one example, in step S130, the line manuscript S, the selected color feature SC and the color block image S cd in the color reasoning module are input into the coloring module G c to obtain a preliminary coloring image C 1, where the formula is as follows:
C1＝GC(S,SC,Scd)
preferably, the line manuscript and the color block image are respectively downsampled before being input into the coloring module. It should be noted that, to construct a multi-scale coloring module, it is necessary to start coloring from a small resolution, then gradually enlarge, and gradually color from easy to difficult. The line manuscript is downsampled to 4 times and 2 times of the original resolution, such as 256x256, 128x128 and 64x64, then 64x64 is colored, 128x128 is colored, and 256x256 is colored; wherein the coloring result of each size is used in the coloring of the next size.
Specifically, the coloring module comprises a plurality of groups of U-Net networks. Line manuscript S and color block image S cd are respectively subjected to 2 times downsampling and 4 times downsampling to obtain line manuscript S 2x,S4x and color block image S cd2x,Scd4x, line manuscript S 4x and color block image S cd4x are input into U-Net1 in a coloring module G c together, And the output SC of the 8 in the color reasoning module is processed by a bilinear upsampling layer, a convolution layer with the size of 3x3 step length of 1 and a ReLU activation function to obtain the color characteristic SC1 with the size of 8x8x128, the color characteristic SC1 is embedded into a2 nd deconvolution module, the result obtained by the U-Net1 is upsampled by 2 times and then is input into the U-Net2 in the coloring module G c together with the line manuscript S 2x and the color lump image S cd2x, The color feature SC1 is embedded into the 2 nd deconvolution module at the same time, the obtained result is up-sampled by 2 times and then is input into the U-Net3 in the coloring module G c together with the line manuscript S and the color block image S cd, the color feature SC1 is subjected to a convolution layer with the step length of 1x1 and a ReLU activation function to obtain the color feature SC2 with the size of 8x8x512, And embedding the color image into a2 nd deconvolution module to obtain a primary coloring image C 1. It should be noted that, each scale has an independent network, and the U-Net network is a structure commonly used in the art, and the coloring module may also use other structures, so long as encoding-decoding can be completed and there is a large receptive field, and the structure of the encoder-decoder with hole convolution, for example, is provided with jump connection, which is not used to limit the protection scope of the present invention.
U-Net1 specifically comprises: m 1 convolution modules consisting of a convolution layer with a convolution kernel of 3x3 step size of 2, an instance normalization layer and a ReLU activation function layer, m 1 -1 deconvolution modules consisting of a deconvolution layer with a convolution kernel of 3x3 step size of 2, an instance normalization layer and a ReLU activation function layer, and 1 deconvolution layer with a convolution kernel of 3x3 step size of 2 and 1 tanh activation function layer, m 1 in this example being 5; wherein Table 3 shows the output sizes of the U-Net1 layers.
TABLE 3 Table 3
Convolutional layer, instance normalization, reLU 32x32x32
Convolutional layer, instance normalization, reLU 16x16x64
Convolutional layer, instance normalization, reLU 8x8x128
Convolutional layer, instance normalization, reLU 4x4x128
Convolutional layer, instance normalization, reLU 2x2x128
Deconvolution layer, instance normalization, reLU 4x4x128
Deconvolution layer, instance normalization, reLU 8x8x128
Deconvolution layer, instance normalization, reLU 16x16x64
Deconvolution layer, instance normalization, reLU 32x32x32
Deconvolution layer, tanh 64x64x3
U-Net2 specifically comprises: m 2 convolution modules consisting of a convolution layer with a 3x3 step size of a convolution kernel of 2, an example standard layer and a ReLU activation function layer, m 2 -1 deconvolution modules consisting of a deconvolution layer with a 3x3 step size of a convolution kernel of 2, an example standardization layer and a ReLU activation function layer, and 1 deconvolution layer with a 3x3 step size of a convolution kernel of 2 and 1 tanh activation function layer, m 2 in this example being 6; wherein Table 4 shows the output sizes of the U-Net2 layers.
TABLE 4 Table 4
U-Net3 specifically comprises: m 3 convolution modules consisting of a convolution layer with a 3x3 step size of a convolution kernel of 2, an instance normalization layer and a ReLU activation function layer, m 3 -1 deconvolution modules consisting of a deconvolution layer with a 3x3 step size of a convolution kernel of 2, an instance normalization layer and a ReLU activation function layer, and 1 deconvolution layer with a 3x3 step size of a convolution kernel of 2 and 1 tanh activation function layer, m 3 in this example being 7; wherein table 5 is the corresponding output size of each layer.
TABLE 5
Convolutional layer, instance normalization, reLU 128x128x64
Convolutional layer, instance normalization, reLU 64x64x128
Convolutional layer, instance normalization, reLU 32x32x256
Convolutional layer, instance normalization, reLU 16x16x512
Convolutional layer, instance normalization, reLU 8x8x512
Convolutional layer, instance normalization, reLU 4x4x512
Convolutional layer, instance normalization, reLU 2x2x512
Deconvolution layer, instance normalization, reLU 4x4x512
Deconvolution layer, instance normalization, reLU 8x8x512
Deconvolution layer, instance normalization, reLU 16x16x512
Deconvolution layer, instance normalization, reLU 32x32x256
Deconvolution layer, instance normalization, reLU 64x64x128
Deconvolution layer, instance normalization, reLU 128x128x64
Deconvolution layer, tanh 256x256x3
In one example, in step S130, the step of inputting the first colored image into the color correction module for calibration, and obtaining the second colored image includes: and inputting the generated primary coloring image C 1 into a color correction module P to obtain a second coloring image. The color correction module P specifically includes a convolution module formed by h convolution layers with a convolution kernel size of 3x3 and a step size of 1 and a ReLU activation function layer, 1 convolution layer with a convolution kernel size of 3x3 and a step size of 1 and 1 tanh activation function layer, where h is 6 in this example.
There is also provided in one example a method of automatic document coloring in a limited color space, the method comprising the steps of:
Acquiring a line manuscript to be colored; the line manuscript refers to an image that needs to be colored. The line manuscript type mainly refers to the type of the line manuscript needing to be colored. And inputting the line manuscript to be colored into a coloring model to obtain a colored color image. The present example may enable automatic coloring, which is automatically accomplished using a trained coloring model for assisting the painter in coloring, by only requiring the user to give the primary color.
Exemplary apparatus
As shown in fig. 6, an apparatus for training a line manuscript automatic coloring model in a limited color space, the apparatus comprising:
a color acquisition module 20, configured to acquire a corresponding dominant color vector according to a line manuscript type;
A color coding module 30, configured to perform multi-level coding on the given main color vector to obtain a corresponding color code, and construct a limited color space;
A model configuration module 40, configured to pre-configure a coloring model, where the coloring model includes a color reasoning module, a coloring module, and a color correction module, and the color reasoning module is configured to receive a line draft and perform color coding processing to obtain a color block image; the coloring module is used for receiving the line manuscript and the color block image for processing to obtain a first coloring image; the color correction module is used for receiving the first coloring image for calibration and obtaining a second coloring image;
the model calibration module 50 is configured to input the true color image corresponding to the line manuscript into a style discriminator, input the second color image into a color discriminator, respectively obtain a style discrimination loss and a color discrimination loss, calculate a mean square error of the second color image and the true color image corresponding to the second color image to obtain a reconstruction loss, and adjust the color model according to the style discrimination loss, the color discrimination loss and the reconstruction loss by a back propagation algorithm, so as to complete training of the color model.
Preferably, the loss function of the coloring model includes a reconstruction loss L re, a style discrimination loss L style and a color discrimination loss L color, and the loss function is as follows:
Lre＝||C2-I||2
Lcolor＝1-Sco
where DS I represents the output of the real color image I via the style discriminator, Representing the output of the second colored image C 2 obtained by the style discriminator; s co represents the output of the second color image C 2 obtained by passing through the color discriminator.
Exemplary electronic device
Next, an electronic device according to an embodiment of the present application is described with reference to fig. 4. The electronic device may be the mobile device itself, or a stand-alone device independent thereof, which may communicate with the mobile device to receive the acquired input signals from them and to send the selected target decision-making actions thereto.
Fig. 4 illustrates a block diagram of an electronic device according to an embodiment of the application.
As shown in fig. 4, the electronic device 10 includes one or more processors 11 and a memory 12.
The processor 11 may be a Central Processing Unit (CPU) or other form of processing unit having data processing and/or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.
Memory 12 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, random Access Memory (RAM) and/or cache memory (cache), and the like. The non-volatile memory may include, for example, read Only Memory (ROM), hard disk, flash memory, and the like. One or more computer program instructions may be stored on the computer readable storage medium that can be executed by the processor 11 to implement the decision making methods and/or other desired functions of the various embodiments of the present application described above.
In one example, the electronic device 10 may further include: an input device 13 and an output device 14, which are interconnected by a bus system and/or other forms of connection mechanisms (not shown). For example, the input device 13 may include various devices such as an on-board diagnostic system (OBD), a Universal Diagnostic Service (UDS), an Inertial Measurement Unit (IMU), a camera, a laser radar, a millimeter wave radar, an ultrasonic radar, and vehicle-mounted communication (V2X). The input device 13 may also comprise, for example, a keyboard, a mouse, etc. The output means 14 may include, for example, a display, speakers, a printer, and a communication network and remote output devices connected thereto, etc.
Of course, only some of the components of the electronic device 10 that are relevant to the present application are shown in fig. 4 for simplicity, components such as buses, input/output interfaces, etc. are omitted. In addition, the electronic device 10 may include any other suitable components depending on the particular application.
Exemplary computer program product and computer readable storage Medium
In addition to the methods and apparatus described above, embodiments of the application may also be a computer program product comprising computer program instructions which, when executed by a processor, cause the processor to perform the steps in the decision making method according to the various embodiments of the application described in the "exemplary methods" section of this specification.
The computer program product may write program code for performing operations of embodiments of the present application in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server.
Furthermore, embodiments of the present application may also be a computer-readable storage medium, having stored thereon computer program instructions, which when executed by a processor, cause the processor to perform the steps in the decision-making method according to the various embodiments of the present application described in the "exemplary methods" section of the present specification.
The computer readable storage medium may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The basic principles of the present application have been described above in connection with specific embodiments, but it should be noted that the advantages, benefits, effects, etc. mentioned in the present application are merely examples and not intended to be limiting, and these advantages, benefits, effects, etc. are not to be construed as necessarily possessed by the various embodiments of the application. Furthermore, the specific details disclosed herein are for purposes of illustration and understanding only, and are not intended to be limiting, as the application is not necessarily limited to practice with the above described specific details.
The block diagrams of the devices, apparatuses, devices, systems referred to in the present application are only illustrative examples and are not intended to require or imply that the connections, arrangements, configurations must be made in the manner shown in the block diagrams. As will be appreciated by one of skill in the art, the devices, apparatuses, devices, systems may be connected, arranged, configured in any manner. Words such as "including," "comprising," "having," and the like are words of openness and mean "including but not limited to," and are used interchangeably therewith. The terms "or" and "as used herein refer to and are used interchangeably with the term" and/or "unless the context clearly indicates otherwise. The term "such as" as used herein refers to, and is used interchangeably with, the phrase "such as, but not limited to.
It is also noted that in the apparatus, devices and methods of the present application, the components or steps may be disassembled and/or assembled. Such decomposition and/or recombination should be considered as equivalent aspects of the present application.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit embodiments of the application to the form disclosed herein. Although a number of example aspects and embodiments have been discussed above, a person of ordinary skill in the art will recognize certain variations, modifications, alterations, additions, and subcombinations thereof.

Claims (10)

1. A training method for an automatic line manuscript coloring model under a limited color space is characterized by comprising the following steps:
Acquiring a corresponding main color vector according to the line manuscript type;
Performing multi-level coding on the main color vector to obtain corresponding color codes, and constructing a limited color space;
a coloring model is pre-configured, the coloring model comprises a color reasoning module, a coloring module and a color correction module, a line manuscript and a color code are input into the color reasoning module to obtain a color block image, the line manuscript and the color block image are input into the coloring module to obtain a first coloring image, and the first coloring image is input into the color correction module to be calibrated to obtain a second coloring image;
Inputting the second colored image and the real color image corresponding to the line manuscript into a style discriminator to obtain style discrimination loss, inputting the second colored image into a color discriminator to obtain color discrimination loss, obtaining reconstruction loss by calculating the mean square error of the second colored image and the real color image corresponding to the second colored image, and adjusting the coloring model through a counter propagation algorithm according to the style discrimination loss, the color discrimination loss and the reconstruction loss to complete training of the coloring model.
2. The method for training a line manuscript automatic coloring model in a limited color space according to claim 1, wherein said obtaining a corresponding dominant color vector comprises: and clustering the sample image colors through a k-means algorithm to obtain main colors.
3. The method for training an automatic line manuscript coloring model in a limited color space according to claim 2, wherein the line manuscript and the color lump image are respectively downsampled, and then input into a coloring module for multi-scale coloring to obtain a first coloring image.
4. The method for training a line manuscript automatic coloring model in a limited color space according to claim 1, wherein the loss function of the coloring model comprises a reconstruction loss L re, a style discrimination loss L style and a color discrimination loss L color, and the loss function is as follows:
Lre＝||C2-I||2
Lcolor＝1-Sco
wherein I represents a true color image, DS I represents an output of the true color image I obtained by a style discriminator, Representing the output of the second colored image C 2 obtained by the style discriminator; where S co represents the output of the second color image C 2 obtained by passing through the color discriminator.
5. The method for training a line manuscript automatic coloring model in a limited color space according to claim 1, wherein the given main color vector is subjected to multi-level coding through a preset color coding module;
The color coding module comprises 1 size-adjusting layer, n convolution layers with a convolution kernel size of 3x3 and a step size of 1 and a ReLU activation function layer.
6. An automatic coloring method for a line manuscript under a limited color space is characterized by comprising the following steps:
acquiring a line manuscript to be colored;
Inputting the draft to be colored into a coloring model according to any one of claims 1-5 to obtain a colored image.
7. An automatic line manuscript coloring model training device under a limited color space, which is characterized by comprising:
the color acquisition module is used for acquiring a corresponding main color vector according to the line manuscript type;
The color coding module is used for carrying out multi-level coding on the main color vector to obtain corresponding color codes and constructing a limited color space;
The model configuration module is used for pre-configuring a coloring model, wherein the coloring model comprises a color reasoning module, a coloring module and a color correction module, and the color reasoning module is used for receiving a line manuscript and performing color coding processing to obtain a color block image; the coloring module is used for receiving the line manuscript and the color block image for processing to obtain a first coloring image; the color correction module is used for receiving the first coloring image for calibration and obtaining a second coloring image;
The model calibration module is used for inputting the second coloring image and any real color image into a style discriminator, inputting the second coloring image into a color discriminator, respectively obtaining style discrimination loss and color discrimination loss, obtaining reconstruction loss by calculating the mean square error of the second coloring image and the corresponding real color image, and adjusting the coloring model through a counter propagation algorithm according to the style discrimination loss, the color discrimination loss and the reconstruction loss to finish the training of the coloring model.
8. The device for training a line manuscript automatic coloring model in a limited color space according to claim 7, wherein the loss function of the coloring model comprises a reconstruction loss L re, a style discrimination loss L style and a color discrimination loss L color, and the loss function is as follows:
Lre＝||C2-I||2
Lcolor＝1-Sco
wherein I represents a true color image, DS I represents an output of the true color image I obtained by a style discriminator, Representing the output of the second colored image C 2 obtained by the style discriminator; where S co represents the output of the second color image C 2 obtained by passing through the color discriminator.
9. An electronic device comprising a processor, an input device, an output device and a memory, the processor, the input device, the output device and the memory being connected in sequence, the memory being for storing a computer program comprising program instructions, the processor being configured to invoke the program instructions to perform the method of any of claims 1-6.
10. A readable storage medium, characterized in that the storage medium stores a computer program comprising program instructions which, when executed by a processor, cause the processor to perform the method of any of claims 1-6.
CN202210084599.6A 2022-01-25 2022-01-25 Line manuscript automatic coloring model training method and device under limited color space, electronic equipment and storage medium Active CN114494523B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210084599.6A CN114494523B (en) 2022-01-25 2022-01-25 Line manuscript automatic coloring model training method and device under limited color space, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210084599.6A CN114494523B (en) 2022-01-25 2022-01-25 Line manuscript automatic coloring model training method and device under limited color space, electronic equipment and storage medium

Publications (2)

Publication Number Publication Date
CN114494523A CN114494523A (en) 2022-05-13
CN114494523B true CN114494523B (en) 2024-07-02

Family

ID=81475447

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210084599.6A Active CN114494523B (en) 2022-01-25 2022-01-25 Line manuscript automatic coloring model training method and device under limited color space, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN114494523B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116416342B (en) * 2023-06-12 2023-09-19 腾讯科技（深圳）有限公司 Image processing method, apparatus, computer device, and computer-readable storage medium

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108615252A (en) * 2018-05-03 2018-10-02 苏州大学 The training method and device of color model on line original text based on reference picture
CN110223359A (en) * 2019-05-27 2019-09-10 浙江大学 It is a kind of that color model and its construction method and application on the dual-stage polygamy colo(u)r streak original text of network are fought based on generation

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110378985B (en) * 2019-07-19 2023-04-28 中国传媒大学 Animation drawing auxiliary creation method based on GAN
CN114730456A (en) * 2019-12-12 2022-07-08 华为技术有限公司 Training method of neural network model, image processing method and device thereof
CN111553837B (en) * 2020-04-28 2022-04-29 武汉理工大学 Artistic text image generation method based on neural style migration
CN112102154B (en) * 2020-08-20 2024-04-26 北京百度网讯科技有限公司 Image processing method, device, electronic equipment and storage medium

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108615252A (en) * 2018-05-03 2018-10-02 苏州大学 The training method and device of color model on line original text based on reference picture
CN110223359A (en) * 2019-05-27 2019-09-10 浙江大学 It is a kind of that color model and its construction method and application on the dual-stage polygamy colo(u)r streak original text of network are fought based on generation

Also Published As

Publication number Publication date
CN114494523A (en) 2022-05-13

Similar Documents

Publication Publication Date Title
US10922860B2 (en) Line drawing generation
CN110097609B (en) Sample domain-based refined embroidery texture migration method
JP6994588B2 (en) Face feature extraction model training method, face feature extraction method, equipment, equipment and storage medium
US11501563B2 (en) Image processing method and system
CN110070517B (en) Blurred image synthesis method based on degradation imaging mechanism and generation countermeasure mechanism
US20230245351A1 (en) Image style conversion method and apparatus, electronic device, and storage medium
Mouzon et al. Joint CNN and variational model for fully-automatic image colorization
CN111080531A (en) Super-resolution reconstruction method, system and device for underwater fish image
CN110097615B (en) Stylized and de-stylized artistic word editing method and system
CN114494523B (en) Line manuscript automatic coloring model training method and device under limited color space, electronic equipment and storage medium
WO2020238120A1 (en) System and method for single-modal or multi-modal style transfer and system for random stylization using the same
CN109801218A (en) Multi-spectral remote sensing image Pan-sharpening method based on multi-layer-coupled convolutional neural networks
CN115588055A (en) Color standardization method and system for digital pathological image
CN115731597A (en) Automatic segmentation and restoration management platform and method for mask image of face mask
EP4220555A1 (en) Training method and apparatus for image segmentation model, image segmentation method and apparatus, and device
CN114037893A (en) High-resolution remote sensing image building extraction method based on convolutional neural network
Chen et al. Learning a deep color difference metric for photographic images
CN111754589B (en) Color matching method, device, computer equipment and storage medium
CN113724282A (en) Image processing method and related product
CN110782408A (en) Intelligent beautifying method and system based on convolutional neural network
CN115457015A (en) Image no-reference quality evaluation method and device based on visual interactive perception double-flow network
CN110866552A (en) Hyperspectral image classification method based on full convolution space propagation network
CN115688234A (en) Building layout generation method, device and medium based on conditional convolution
CN110489584B (en) Image classification method and system based on dense connection MobileNet model
CN112288748B (en) Semantic segmentation network training and image semantic segmentation method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant