CN109849576B - Method for assisting drawing by referring to gray level diagram - Google Patents
Method for assisting drawing by referring to gray level diagram Download PDFInfo
- Publication number
- CN109849576B CN109849576B CN201910149284.3A CN201910149284A CN109849576B CN 109849576 B CN109849576 B CN 109849576B CN 201910149284 A CN201910149284 A CN 201910149284A CN 109849576 B CN109849576 B CN 109849576B
- Authority
- CN
- China
- Prior art keywords
- image
- semantic segmentation
- gray level
- gray
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Image Analysis (AREA)
Abstract
The invention discloses a method for assisting drawing by referring to a gray level diagram, which comprises the following training stages: acquiring a training image, converting the training image into a gray-scale image, performing semantic segmentation and annotation on the training image to obtain a semantic segmentation image, training a deep learning model capable of converting the image into the image by taking the semantic segmentation image as input and taking a corresponding gray-scale image as output, determining model parameters, and obtaining a gray-scale image generation model; an application stage: drawing a semantic segmentation graph, inputting the drawn semantic segmentation graph into a gray level graph generation model, generating a plurality of gray level graphs through calculation, and performing texture drawing on the semantic segmentation graph according to the gray level graph which can be referred to selected from the gray level graphs so as to perfect the painting works. The drawing auxiliary method converts the semantic segmentation graph into the generated gray level graph, provides light and shade and texture reference on content details on the drawing, supports an author to find inspiration from the gray level graph, and reduces time of the author on conception of the details of the work.
Description
Technical Field
The invention belongs to the field of drawing assistance, and particularly relates to a method for assisting drawing by referring to a gray-scale image.
Background
With the development of the computer industry, computers undergo the transition of execution, simulation and assistant in the creative design process. In the field of drawing creation, computers are gradually changing from a tool that provides multiple functions to a drawing-assisted character. Technological advances have made it possible for computers to simulate painting creation by learning the work of a painter, capturing the style of the work, and simulating the work. The assistant role played by the computer can realize the collaborative creation of the user and the computer, provide various supports for the user in the creation process, and enable non-professional persons to perform high-level drawing creation. In the field of drawing assistance, computer assistance still has a great expandable space.
The application of the existing artificial intelligence technology generation technology in drawing can be simply divided into two aspects of work generation and auxiliary information generation. Production of a work directly produces the final result, which is usually fixed and invariant. The auxiliary information generation provides auxiliary information during the drawing process to help the painter quickly obtain the final result.
The work generation application is based on the primary draft drawn by the author, the author designs simple work primary drafts, such as drafts, line drafts, layout drawings and the like, the artificial intelligence technology directly completes the next generation work, and the simple work primary drafts are directly generated into final works. Although this method is very convenient, it gives a considerable free space for the model, and the variety of results that the model can select to generate is too many, resulting in very poor image effect of the generated image, and at the same time, depending on the content and drawing level of the manuscript excessively, only a few parts of the manuscript content generated according to the model can obtain good results. Although the current deep learning model can learn the style of the painting and can obtain the final work in a style migration mode with good effect, the work is often abstract, and the experience of the whole painting is not influenced by slight difference. Whereas on non-abstract visual images humans are sensitive to small distortions and confusion, the work produced by artificial intelligence techniques often has errors that are in violation of common sense. Limited to the current generation technology, the direct use of the results generated by the artificial intelligence technology as a work in the non-abstract image field still has many difficulties to be solved.
The application of auxiliary information generation provides the generated result to the author for reference in the drawing process, and the reference mode is more various, for example, a predicted line of the next stroke is provided when drawing a manuscript, and the generation effect is provided for reference when drawing a layout. These approaches provide a single reference effect, usually based on line and global effects, and mostly provide no guidance on details. The painter needs to enrich the drawing details in the usual drawing process, needs to acquire the inspiration of the details from daily life and memory, and needs guidance and reference in the aspect. Related application support of artificial intelligence technology is also lacking at present to provide references on content details.
Disclosure of Invention
The invention aims to provide a method for assisting drawing by referring to a gray level image, which converts a semantic segmentation image into a generated gray level image, provides light and shade and texture reference on content details on the drawing, supports an author to find inspiration from the gray level image, reduces time of a painter on conception of the details of a work, and further can efficiently and conveniently assist drawing.
In order to achieve the purpose, the invention provides the following technical scheme:
a method for assisting drawing by referring to a gray scale map comprises the following steps:
a training stage: acquiring a training image, converting the training image into a gray scale image, determining semantic type and color-semantic correspondence, performing semantic segmentation on the training image according to the color-semantic correspondence to obtain a semantic segmentation image, performing repeated iterative training on a deep learning model capable of realizing image conversion into an image by taking the semantic segmentation image as input and the corresponding gray scale image as output to obtain a conversion model from the semantic segmentation image to the gray scale image;
an application stage: according to the color-semantic correspondence and semantic types, drawing a semantic segmentation graph according to the drawing intention, inputting the drawn semantic segmentation graph into a conversion model from the semantic segmentation graph to a gray level graph, generating a plurality of gray level graphs through calculation, and performing texture drawing on the semantic segmentation graph according to a reference gray level graph selected from the gray level graphs, thereby enriching the content of the drawing.
In the invention, the artificial intelligence technology is used for assisting drawing, and the deep learning model is used for training the semantic segmentation graph to the gray level graph to obtain a model for converting the semantic segmentation graph into the gray level graph. The trained model is used for generating a gray level map meeting the intention of the painter to provide reference, the generation process can be iterated repeatedly, the painter is supported to continuously search inspiration from the gray level map generated by the model, the time of the painter on conception of details of the work is reduced, and the method has the characteristics of high efficiency, convenience and novelty.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a block flow diagram of a method of assisting drawing with reference to a grayscale map of the present invention;
FIG. 2 is an example of a semantic segmentation graph;
fig. 3 is an example of a grayscale map.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the detailed description and specific examples, while indicating the scope of the invention, are intended for purposes of illustration only and are not intended to limit the scope of the invention.
In order to improve the effect and efficiency of drawing, as shown in fig. 1, the invention provides a method for assisting drawing by referring to a gray-scale map, which comprises the following steps:
s101, obtaining a training image, and converting the training image into a gray image through an image graying algorithm.
In this embodiment, the training images may be searched from a web, photographed by a human, or some database. The general training images generally have similar semantic structures, such as sceneries, street views, and high-rise buildings. In order to ensure enough data to train the deep learning model, the obtained training images are not less than 5000.
The gray-scale image is an image with colors and partial details removed on the basis of an original image, but retains information such as textures, light and shade on content details, and the detail richness of the gray-scale image is lower than that of the original image but higher than that of a draft image. The gray-scale image as an image type has three advantages that the gray-scale image can be generated through a deep learning model, rich detail information is provided, the gray-scale image can be obtained by executing an algorithm on a training image, and the characteristics determine that the gray-scale image is suitable for being used as a reference image.
In this embodiment, a formula (1) is adopted to convert a training image (i.e., an RGB color image) into a grayscale image;
Gray(i,j)＝0.299*R(i,j)+0.587*G(i,j)+0.144*B(i,j) (1)
where R (i, j) is an R channel image, G (i, j) is a G channel image, and B (i, j) is a B channel image, the converted grayscale is shown in fig. 3.
S102, determining semantic types and color-semantic correspondence, and performing semantic segmentation on the training image according to the color-semantic correspondence to obtain a semantic segmentation image.
Performing semantic segmentation on the training image according to the color-semantic correspondence comprises the following steps:
segmenting the training image according to the corresponding color of the object in the image, and labeling similar or same objects in the image with the same color to obtain a semantic segmentation image. The process may be performed manually or by using a set algorithm, and the segmented semantic segmentation image may be as shown in fig. 2.
In this embodiment, the objects in the training image are classified into a plurality of classes, each class represents the same or similar object, and the class is represented by a fixed color. Wherein, similar objects refer to objects belonging to the same class, such as: birch, cypress, etc. all belong to the class of trees, and green can be used to represent trees. In this way, the training image can be converted into a semantically segmented image represented by colors only, and regions with the same semantic information are represented by the same colors in the semantically segmented image.
And S103, training a deep learning model capable of realizing image conversion into an image by taking the semantic segmentation image as input and the corresponding gray-scale image as output, and repeating iteration until the model is converged to obtain a conversion model from the semantic segmentation image to the gray-scale image.
In this embodiment, a pix2pixHD model is selected as a deep learning model, and the pix2pixHD model is trained end to obtain a conversion model from a semantic segmentation map to a grayscale map. The Pix2pixHD model is a conditional generation countermeasure network and is mainly applied to the field of image translation. For the task proposed by the invention, the generator G aims at generating the semantic segmentation map into the corresponding gray-scale map, and the discriminator D: 1) distinguishing a real gray scale image from a generated gray scale image; 2) and judging whether the mapping between the gray-scale image and the semantic segmentation image is correct or not. The pix2pixHD model is an image translation model, has strong image conversion capability, and can learn information such as textures, light and shade rules of an image from a training image, namely, a semantic segmentation image can be converted into a gray image, so that detailed information is enriched.
Specifically, the input to the pix2pixHD model is the one-hot vector representation of the label map. The generator of the Pix2pixHD model consists of two sub-generators, G1As a global generator, G2The local enhancement generator is used for enlarging the image size of a generated result image, and both of the local enhancement generator and the local enhancement generator are composed of a set of convolution network, residual error network and transposition convolution network. Considering that the generated gray-scale map is used only as a reference map and does not require excessively high display degree, only the global generator G is used1All input and output sizes are set to be 256 × 512 as a generator, the training process is carried out on a GPU, parameters are updated by adopting an Adam gradient descent method, and the learning rate lr is 0.0002 and β1＝0.5，β20.999. During the first 50 iterations, the learning rate remains constant, linearly decaying as the number of subsequent iterations increases. Within 200 iterations, the model converged to stability.
And S104, drawing the semantic segmentation graph according to the drawing intention according to the color-semantic correspondence and the semantic type.
The common way of drawing is firstly a drawing intention, and the author designs the structure of the whole drawing and then draws the accurate content. It is relatively easy to think of the structure of a painting, but different spatial structures mean that the same semantic block will have different texture details, and the author spends a lot of time trying to and deleting the details of specific texture, shading, etc. of a certain part of the content. The method aims to provide the angle of the gray level image by utilizing the gray level image generation model obtained in the training stage to assist drawing, can provide a light and shade scheme of the whole layout, can provide texture inspiration of content details, and can enable a painter to see the rough effect of the painting through the fuzzy gray level image so as to quickly make corresponding modification, thereby greatly shortening the time required by the writer to draw the painting.
The author expresses the drawing intention by drawing a semantic segmentation graph, and the layout is represented by using the semantic segmentation graph. The semantic segmentation map can represent the overall spatial structure information by the arrangement of the color blocks, the shapes of the color blocks and the represented semantic information.
And S105, inputting the drawn semantic segmentation graph into a conversion model from the semantic segmentation graph to a gray level graph, and generating a plurality of gray level graphs according with the semantic segmentation content through calculation.
Specifically, the multiple generated gray-scale maps meet the layout of the semantic segmentation map in a spatial structure, and the detail content is slightly different due to the randomness of model generation. Generating multiple gray-scale maps may provide more patterns of detail.
And S106, selecting a proper gray level image from the plurality of gray level images as a reference on the detail content, and performing texture drawing on the detail content of the semantic segmentation image to enrich the content of the painting work.
Specifically, a gray-scale image which can be referred to is selected according to the interest, the gray-scale image comprises light, shade and texture information, and the semantic segmentation image is filled and perfected according to the light, shade and texture information presented by the gray-scale image.
The gray level image has the characteristics of texture, shading and the like on details, has richer detail information compared with a semantic segmentation image, is learned from training data by a model, has certain rationality, and represents the texture and shading information which are learned by a computer and are required to be in the place. According to the gray level maps, an author can find inspiration from the gray level maps and enrich the details of each semantic block of the painting.
To better perfect the pictorial representation, the painting assistance method further comprises:
and S107, repeating S104-S106, enriching the drawing content by each iteration, and repeating the iteration until the whole drawing is completed.
Specifically, the painting works with rich details are redrawn or the semantic segmentation images are modified and input into a conversion model from the semantic segmentation images to the gray level images, a plurality of gray level images are generated through calculation, and texture drawing is performed on the perfected painting works according to the gray level images which can be selected from the gray level images, namely S104-S106 are repeated, so that the painting works are further perfected. And the final painting works can be obtained by continuously and repeatedly updating and perfecting the painting works.
According to the drawing auxiliary method, the semantic segmentation image is converted into the generated gray level image, so that light and shade and texture references on content details on the drawing are provided, and the defect of drawing auxiliary in providing detail auxiliary is overcome. The method can repeatedly iterate the generation process, supports the painter to continuously search inspiration from the gray-scale image generated by the model, reduces the time of the painter on conceiving the details of the work, and has the characteristics of high efficiency, convenience and novelty.
The above-mentioned embodiments are intended to illustrate the technical solutions and advantages of the present invention, and it should be understood that the above-mentioned embodiments are only the most preferred embodiments of the present invention, and are not intended to limit the present invention, and any modifications, additions, equivalents, etc. made within the scope of the principles of the present invention should be included in the scope of the present invention.
Claims (7)
1. A method for assisting drawing by referring to a gray scale map comprises the following steps:
a training stage: acquiring a training image, converting the training image into a gray scale image, determining semantic type and color-semantic correspondence, performing semantic segmentation on the training image according to the color-semantic correspondence to obtain a semantic segmentation image, performing repeated iterative training on a deep learning model capable of realizing image conversion into an image by taking the semantic segmentation image as input and the corresponding gray scale image as output to obtain a conversion model from the semantic segmentation image to the gray scale image;
an application stage: according to the color-semantic correspondence and semantic types, drawing a semantic segmentation graph according to the drawing intention, inputting the drawn semantic segmentation graph into a conversion model from the semantic segmentation graph to a gray level graph, generating a plurality of gray level graphs through calculation, and performing texture drawing on the semantic segmentation graph according to a reference gray level graph selected from the gray level graphs, thereby enriching the content of the drawing.
2. The method for assisting drawing with reference to gray scale as claimed in claim 1, wherein the training images are acquired not less than 5000 sheets.
3. The method for assisting drawing with reference to a gray scale map as claimed in claim 1, wherein formula (1) is adopted to convert the training image into the gray scale map;
Gray(i，j)＝0.299*R(i，j)+0.587*G(i，j)+0.144*B(i，j) (1)
where R (i, j) is an R-channel image, G (i, j) is a G-channel image, and B (i, j) is a B-channel image.
4. The method of assisting drawing with reference to a gray scale map as claimed in claim 1, wherein semantically segmenting the training image according to the color-semantic correspondence comprises:
segmenting the training image according to the corresponding color of the object in the image, and labeling similar or same objects in the image with the same color to obtain a semantic segmentation image.
5. The method for assisting in drawing with reference to the gray level map as claimed in claim 1, wherein a pix2pixHD model is selected as a deep learning model, and end-to-end training is performed on the pix2pixHD model to obtain a conversion model from a semantic segmentation map to the gray level map.
6. The method for assisting drawing with reference to gray scale map as claimed in claim 1, wherein the gray scale map to be referred to is selected according to interest, the gray scale map contains information of brightness and texture, and the semantic segmentation map is filled and completed according to the information of brightness and texture presented by the gray scale map.
7. A method for assisting drawing with reference to a gray scale map as set forth in any one of claims 1 to 6, wherein the drawing assisting method further comprises:
and redrawing the pictorial work with abundant details or modifying the semantic segmentation image, inputting the image into a conversion model from the semantic segmentation image to a gray level image, generating a plurality of gray level images through calculation, and performing texture drawing on the perfected pictorial work according to a reference gray level image selected from the gray level images to further perfect the pictorial work.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910149284.3A CN109849576B (en) | 2019-02-28 | 2019-02-28 | Method for assisting drawing by referring to gray level diagram |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910149284.3A CN109849576B (en) | 2019-02-28 | 2019-02-28 | Method for assisting drawing by referring to gray level diagram |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109849576A CN109849576A (en) | 2019-06-07 |
| CN109849576B true CN109849576B (en) | 2020-04-28 |
Family
ID=66899239
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910149284.3A Active CN109849576B (en) | 2019-02-28 | 2019-02-28 | Method for assisting drawing by referring to gray level diagram |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109849576B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110312139A (en) * | 2019-06-18 | 2019-10-08 | 深圳前海达闼云端智能科技有限公司 | The method and apparatus of image transmitting, storage medium |
| CN110322529B (en) * | 2019-07-12 | 2023-04-18 | 电子科技大学 | Deep learning-based auxiliary art drawing method |
| CN111043953B (en) * | 2019-10-17 | 2021-07-27 | 杭州电子科技大学 | Two-dimensional phase unwrapping method based on deep learning semantic segmentation network |
| CN111325212A (en) * | 2020-02-18 | 2020-06-23 | 北京奇艺世纪科技有限公司 | Model training method and device, electronic equipment and computer readable storage medium |
| CN112001839B (en) * | 2020-07-23 | 2022-09-13 | 浙江大学 | Cross-domain image conversion method based on semantic feature transformation, computer device and storage medium |
| CN113111906B (en) * | 2021-02-24 | 2022-07-12 | 浙江大学 | Method for generating confrontation network model based on condition of single pair image training |
| CN116664773B (en) * | 2023-06-02 | 2024-01-16 | 北京元跃科技有限公司 | Method and system for generating 3D model by multiple paintings based on deep learning |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2905724A3 (en) * | 2014-02-07 | 2015-09-16 | Tata Consultancy Services Limited | Object detection system and method |
| CN107945244A (en) * | 2017-12-29 | 2018-04-20 | 哈尔滨拓思科技有限公司 | A kind of simple picture generation method based on human face photo |
| CN107944457A (en) * | 2017-11-23 | 2018-04-20 | 浙江清华长三角研究院 | Drawing object identification and extracting method under a kind of complex scene |
| US10178982B2 (en) * | 2015-07-29 | 2019-01-15 | Perkinelmer Health Sciences, Inc. | System and methods for automated segmentation of individual skeletal bones in 3D anatomical images |
-
2019
- 2019-02-28 CN CN201910149284.3A patent/CN109849576B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2905724A3 (en) * | 2014-02-07 | 2015-09-16 | Tata Consultancy Services Limited | Object detection system and method |
| US10178982B2 (en) * | 2015-07-29 | 2019-01-15 | Perkinelmer Health Sciences, Inc. | System and methods for automated segmentation of individual skeletal bones in 3D anatomical images |
| CN107944457A (en) * | 2017-11-23 | 2018-04-20 | 浙江清华长三角研究院 | Drawing object identification and extracting method under a kind of complex scene |
| CN107945244A (en) * | 2017-12-29 | 2018-04-20 | 哈尔滨拓思科技有限公司 | A kind of simple picture generation method based on human face photo |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109849576A (en) | 2019-06-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109849576B (en) | Method for assisting drawing by referring to gray level diagram | |
| CN108830912B (en) | Interactive gray image coloring method for depth feature-based antagonistic learning | |
| CN102509511B (en) | Rapid map making system based on GIS (Geographic Information System) data | |
| US11386589B2 (en) | Method and device for image generation and colorization | |
| CN110414519A (en) | A kind of recognition methods of picture character and its identification device | |
| CN110222722A (en) | Interactive image stylization processing method, calculates equipment and storage medium at system | |
| KR20200052438A (en) | Deep learning-based webtoons auto-painting programs and applications | |
| CN111475656A (en) | Visual question-answering method and system based on external knowledge aggregation | |
| CN107748744A (en) | A kind of method for building up and device for sketching the contours frame knowledge base | |
| CN106023276A (en) | Pencil drawing making method and pencil drawing making device based on image processing | |
| CN113449355A (en) | Building house type graph automatic generation method based on artificial intelligence | |
| CN112784531A (en) | Chinese font and word stock generation method based on deep learning and part splicing | |
| Yu et al. | Architectural facade recognition and generation through generative adversarial networks | |
| KR20230073751A (en) | System and method for generating images of the same style based on layout | |
| CN106021228B (en) | A kind of method and system carrying out text analyzing using knowledge topographic map | |
| CN115587930A (en) | Image color style migration method, device and medium | |
| CN112507931B (en) | Deep learning-based information chart sequence detection method and system | |
| CN110796150B (en) | Image emotion recognition method based on emotion significant region detection | |
| Liang et al. | [Retracted] Application of Art Activities in Colleges and Universities Based on BP Neural Network Algorithm | |
| CN113704490B (en) | New industrial and scientific education knowledge graph construction method for heterogeneous data | |
| CN117333778B (en) | Knowledge-graph-based zero-sample plant identification method for plant science popularization education | |
| Cui | Application of Line Integral Convolution Techniques in the Generation of Fine Art Painting Effects | |
| Su et al. | Cartoon image colorization based on emotion recognition and superpixel color resolution | |
| Fu et al. | PlanNet: A Generative Model for Component-Based Plan Synthesis | |
| Kang | Integrating artificial intelligence in cartography: Using deep learning for map style transfer and map generalization |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |