WO2022166896A1 - 视频生成方法、装置、设备及可读存储介质 - Google Patents
视频生成方法、装置、设备及可读存储介质 Download PDFInfo
- Publication number
- WO2022166896A1 WO2022166896A1 PCT/CN2022/075037 CN2022075037W WO2022166896A1 WO 2022166896 A1 WO2022166896 A1 WO 2022166896A1 CN 2022075037 W CN2022075037 W CN 2022075037W WO 2022166896 A1 WO2022166896 A1 WO 2022166896A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- sketch
- images
- pixel
- mask
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 115
- 238000003860 storage Methods 0.000 title claims abstract description 30
- 238000004590 computer program Methods 0.000 claims description 25
- 210000004709 eyebrow Anatomy 0.000 claims description 22
- 210000005252 bulbus oculi Anatomy 0.000 claims description 21
- 210000004209 hair Anatomy 0.000 claims description 21
- 238000009877 rendering Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 abstract description 59
- 238000012545 processing Methods 0.000 abstract description 12
- 238000013135 deep learning Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 24
- 230000001815 facial effect Effects 0.000 description 20
- 230000006870 function Effects 0.000 description 9
- 238000004891 communication Methods 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 6
- 239000003086 colorant Substances 0.000 description 5
- 210000001508 eye Anatomy 0.000 description 5
- 230000008859 change Effects 0.000 description 3
- 238000010422 painting Methods 0.000 description 3
- 238000007591 painting process Methods 0.000 description 3
- 238000004088 simulation Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 238000010421 pencil drawing Methods 0.000 description 2
- 239000000047 product Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 239000012467 final product Substances 0.000 description 1
- 238000007499 fusion processing Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 239000012528 membrane Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 210000001747 pupil Anatomy 0.000 description 1
- 238000011946 reduction process Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/20—Drawing from basic elements, e.g. lines or circles
- G06T11/203—Drawing of straight lines or curves
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/001—Texturing; Colouring; Generation of texture or colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T13/00—Animation
- G06T13/80—2D [Two Dimensional] animation, e.g. using sprites
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
- G06V40/171—Local features and components; Facial parts ; Occluding parts, e.g. glasses; Geometrical relationships
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/02—Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
- G11B27/031—Electronic editing of digitised analogue information signals, e.g. audio or video signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/2625—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects for obtaining an image which is composed of images from a temporal image sequence, e.g. for a stroboscopic effect
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/265—Mixing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2210/00—Indexing scheme for image generation or computer graphics
- G06T2210/12—Bounding box
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/272—Means for inserting a foreground image in a background image, i.e. inlay, outlay
- H04N2005/2726—Means for inserting a foreground image in a background image, i.e. inlay, outlay for simulating a person's appearance, e.g. hair style, glasses, clothes
Definitions
- the embodiments of the present disclosure relate to the technical field of image processing, and in particular, to a video generation method, apparatus, device, and readable storage medium.
- Sketching is a kind of image stylization.
- sketching is an art form that uses pencil as a medium to express scenes or characters with lines.
- Sketch can be divided into line structure drawing and fine realistic sketch.
- Creating sketches by hand requires the author to have a certain artistic background.
- the terminal device With the intelligentization of terminal devices, users can obtain sketch images and the like by using the terminal devices.
- the terminal device In the process of acquiring the sketch image, the terminal device performs style conversion processing on the source image selected by the user, thereby obtaining a sketch image.
- a dynamic video which simulates the process of painting by an artist, and draws a sketch image stroke by stroke.
- Embodiments of the present disclosure provide a video generation method, device, device, and readable storage medium, which generate a video simulating a painter's stroke by stroke to create a sketch image based on an image, and the process is simple.
- an embodiment of the present disclosure provides a video generation method, including:
- the target sketch image Based on the target sketch image, multiple sub-images of the target sketch image are generated, the multiple sub-images respectively correspond to the sketch images of the target sketch image under different rendering completion degrees, and the target sketch image is the multiple sketches any sketch image in the image;
- each sub-picture of each sketch image in the plurality of sketch images is set according to the order of the color depth from light to dark and the order of the drawing completion degree from low to high,
- the sketching video is generated.
- the present disclosure provides a video generation device, comprising:
- a first generating unit configured to generate a plurality of sketch images according to the source image, where the plurality of sketch images respectively correspond to sketch images of the source image under different color depths;
- the second generating unit is configured to generate, based on the target sketch image, multiple sub-images of the target sketch image, where the multiple sub-images respectively correspond to the sketch images of the target sketch image under different drawing completion degrees, and the target sketch image the image is any one of the plurality of sketch images;
- the third generating unit is configured to use each sub-image of each sketch image in the plurality of sketch images as a video frame of the sketch drawing video, and set in the order of color depth from light to dark and the order of drawing completion degree from low to high The sequence of the video frames to generate the sketching video.
- an electronic device comprising: at least one processor and a memory;
- the memory stores computer-executable instructions
- the at least one processor executes computer-implemented instructions stored in the memory to cause the at least one processor to perform the video generation method as described in the first aspect and various possible designs of the first aspect above.
- a computer-readable storage medium where computer-executable instructions are stored in the computer-readable storage medium, and when a processor executes the computer-executable instructions, The video generation method described above in the first aspect and various possible designs of the first aspect is implemented.
- an embodiment of the present disclosure provides a computer program product, where the computer program product includes: a computer program, where the computer program is stored in a readable storage medium, and at least one processor of an electronic device stores data from the readable storage medium.
- the medium reads the computer program, the computer program being executed by the at least one processor to cause the electronic device to perform the video generation method as described in the first aspect and various possible designs of the first aspect above.
- an embodiment of the present disclosure provides a computer program that, when executed by a processor, implements the video generation method described in the first aspect and various possible designs of the first aspect.
- the electronic device After the electronic device obtains the source image, it generates sketches at different stages in the process of simulating the painter's drawing of the target sketch based on the source image in order of colors from light to dark.
- Image for each target sketch image, generate multiple sub-images in the order of drawing completion from high to low to simulate the process of drawing the sketch image by the painter, and then draw each sub-image of each sketch image in the multiple sketch images as a sketch
- For the video frames of the video set the sequence of the video frames according to the order of color depth from light to dark and the order of drawing completion from low to high to generate a sketch drawing video.
- various stages of the painter's creation of sketch images are simulated through image processing, the process is simple, and no deep learning process is required, and the efficiency is high.
- FIG. 1 is a schematic diagram of a network architecture of a video generation method provided by an embodiment of the present disclosure
- FIG. 2 is a flowchart of a video generation method provided by an embodiment of the present application.
- FIG. 3 is a schematic diagram of a plurality of sketch images provided by an embodiment of the present disclosure.
- FIG. 4 is a schematic diagram of a plurality of sub-graphs provided by an embodiment of the present disclosure.
- FIG. 5 is a schematic diagram of the process of drawing sub-pictures in the video generation method provided by the present disclosure
- FIG. 6 is a schematic process diagram of a video generation method provided by the present disclosure.
- FIG. 7 is a schematic diagram of a face key point in the video generation method provided by the present disclosure.
- FIG. 8 is a schematic diagram of a second mask of the hair region in the video generation method provided by the present disclosure.
- FIG. 9 is a schematic diagram of a first convex hull area and a second convex hull area in the video generation method provided by the present disclosure.
- FIG. 10 is a structural block diagram of a video generating apparatus according to an embodiment of the present disclosure.
- FIG. 11 is a schematic structural diagram of an electronic device for implementing an embodiment of the present disclosure.
- the present disclosure considers, based on the source image and the target sketch, to generate a video simulating the painter's stroke-by-stroke drawing process, the process is simple, and meets the needs of the majority of ordinary users.
- FIG. 1 is a schematic diagram of a network architecture of a video generation method provided by an embodiment of the present disclosure.
- the network architecture includes a terminal device 1 , a server 2 and a network 3 , and the terminal device 1 and the server 2 establish a network connection through the network 3 .
- Network 3 includes various types of network connections, such as wired, wireless communication links, or fiber optic cables.
- the user uses the terminal device 1 to interact with the server 2 through the network 3 to receive or send messages and the like.
- Various communication client applications are installed on the terminal device 1, such as video playback applications, shopping applications, search applications, instant communication tools, email clients, social platform software, and the like.
- the terminal device 1 may be hardware or software.
- the terminal device 1 is, for example, a mobile phone, a tablet computer, an e-book reader, a laptop computer, a desktop computer, or the like.
- the terminal device 1 is software, it can be installed in the hardware devices listed above. In this case, the terminal device 1 is, for example, multiple software modules or a single software module, which is not limited by the embodiments of the present disclosure.
- the server 2 is a server capable of providing various servers for receiving the source image sent by the terminal device, and generating a video simulating a sketch image created by a painter stroke by stroke based on the source image.
- the server 2 may be hardware or software.
- the server 2 is hardware, the server 2 is a single server or a distributed server cluster composed of multiple servers.
- the server 2 is software, it may be multiple software modules or a single software module, etc., which is not limited in the embodiment of the present disclosure.
- terminal devices 1 , servers 2 and networks 3 in FIG. 1 are only illustrative. In actual implementation, any number of electronic devices 1 , servers 2 and networks 3 are deployed according to actual requirements.
- the server 2 and the network 3 in the above-mentioned FIG. 1 may not exist.
- FIG. 2 is a flowchart of a video generation method provided by an embodiment of the present application.
- the execution subject of this embodiment is an electronic device, and the electronic device is, for example, the terminal device or the server in the above-mentioned FIG. 1 .
- This embodiment includes:
- the electronic device obtains the source image locally, or obtains the source image from the Internet.
- the source image is also called (image_src).
- the source image is a red green blue (RGB) image, a black-and-white photo, etc., which is not limited in the embodiment of the present disclosure.
- the plurality of sketch images respectively correspond to sketch images of the source image at different color depths.
- the process of creating a sketch image by a professional painter is a step-by-step process, and the visual experience of the entire work is from light to dark colors.
- the electronic device generates a plurality of sketch images according to the source image, and the colors of the sketch images are sequentially deepened.
- FIG. 3 please refer to FIG. 3 .
- FIG. 3 is a schematic diagram of a plurality of sketch images provided by an embodiment of the present disclosure.
- the leftmost is the source image, which can be a color RGB image or a black and white photo, etc., 1-4 are different sketch images.
- the electronic device processes the source image by using an image processing algorithm, thereby obtaining sketch images 1-4 based on the source image.
- the sketch image 4 is, for example, the final product, which is equivalent to the finished product created by a professional painter.
- sketch images 1-4 are merely illustrative to illustrate different sketch images, and do not necessarily mean that there are four sketch images. In practical implementation, the number of sketch images with colors ranging from light to dark may be less than or greater than 4.
- the plurality of sub-images respectively correspond to the sketch images of the target sketch image under different drawing completion degrees
- the target sketch image is any one of the plurality of sketch images.
- the electronic device when a professional painter creates the target sketch image, it is impossible to complete the sketch image in one stroke, but is completed step by step and multiple times.
- the electronic device generates sketch images with different drawing completion degrees for the target sketch image, which are hereinafter referred to as sub-images.
- sub-images For the adjacent first sub-image and the second sub-image, the second sub-image contains more stroke areas than the first sub-image contains. For example, if the degree of drawing completion indicates that the mouth is drawn first and then the eyes are drawn, the first sub-picture includes the drawn mouth, and the second sub-picture includes the drawn eyes in addition to the drawn mouth.
- the first sub-image contains the drawn outline of the mouth
- the second sub-image not only contains the drawn outline of the mouth, but also includes the filling of the mouth, etc.
- FIG. 4 is a schematic diagram of multiple sub-graphs provided by an embodiment of the present disclosure.
- the length of the video is 15 seconds, 30 frames per second, and 450 frames of images in total.
- 30 frames of the 450 frames of images are used for drawing the outline of the human face, which is equivalent to having 30 sub-images, and the images of the 30 frames are equivalent to simulating the outline of the human face drawn by a professional painter in 30 strokes.
- the stroke area is the outline of the face.
- the outline of the face in the first frame image is the least, and the outline of the face in the second frame image gradually increases. In the 30th frame image, the whole face Outlines are drawn.
- each sub-picture of each sketch image in the plurality of sketch images as the video frame of the sketch drawing video, and set the video frame in the order of color depth from light to dark and the order of the degree of completion of drawing from low to high. sequence, generating the sketching video.
- all the sub-images are used as video frames. After that, set the order of each sub-image of the target sketch image in the order of the drawing completion degree from low to high, so as to obtain the sub-video that simulates the creation of the target sketch image, and then set the sub-video in the order of color from light to dark. These sub-videos are sequenced and synthesized, resulting in a sketching video.
- the electronic device When the electronic device is a server, the electronic device sends the sketching video to a mobile terminal such as a mobile phone, so that the mobile terminal such as the mobile phone can play the sketching video.
- the electronic device is a mobile terminal such as a mobile phone, the sketching video is directly played or the video sketching video is stored locally.
- the electronic device after acquiring the source image, the electronic device generates sketch images of different stages in the process of simulating the painter's drawing of the target sketch in the order from light to dark based on the source image, and for each target sketch image, Generate multiple sub-images in the process of simulating the process of drawing the sketch image by the painter according to the order of drawing completion degree from high to bottom, and then use each sub-image of each sketch image in the multiple sketch images as the video frame of the sketch drawing video, according to the color depth from The order of light to dark and the order of drawing completion from low to high set the order of video frames to generate a sketch drawing video.
- various stages of the painter's creation of sketch images are simulated through image processing, the process is simple, and no deep learning process is required, and the efficiency is high.
- the electronic device when the electronic device generates a plurality of sub-images of the target sketch image based on the target sketch image, the increasing order of the mask values of the pixels in the first mask is determined, and the initial mask value of the pixels in the first mask is determined.
- the film value is 0, the growth of pixels in the first mask is used to indicate that the mask value of the pixel is changed from 0 to 1, and the first mask is used to make the background of the target sketch image according to the The growing order gradually transitions to the target sketch image.
- the plurality of subgraphs are generated, and each increase in the increase sequence is one-to-one with the subgraphs in the plurality of subgraphs correspond.
- the electronic device sets a first mask (mask) for the target sketch image, and the initial mask of the pixels on the first mask is Membrane values are all 0.
- the electronic device determines the increasing order of the pixels in the first mask, and one increasing sequence is used to indicate which pixels have a mask value from 0 to 1, and each increase corresponds to one or more strokes in the painting process of the professional painter. For example, if a certain increase indicates that the mask value of the pixel point of the upper lip contour changes from 0 to 1, then the upper lip contour can be drawn according to this increase.
- the target sketch image and the background of the target sketch image first, for each increase in the increasing sequence, from all The first set of pixels corresponding to the increase is determined in the first mask.
- the latter sketch image is actually obtained by continuing to draw on the basis of the former sketch image in the real creation process.
- the process of creating the target sketch image is actually a process of drawing on the background of the target sketch image, so that the background is transformed into the target sketch image.
- it is necessary to control the variation of the mask value of the pixels in the first mask. For example, initially, the mask value of each pixel in the first mask is 0, and at this time, the background does not change.
- the electronic device determines the first pixel set from the first mask according to the growth. For example, during a certain increase, the mask value of the pixel representing the contour of the upper lip in the first mask changes to 1, and the pixel representing the contour of the upper lip is used as the first set of pixels.
- a second pixel set is determined from the target sketch image, and a third pixel set is determined from the background, the first pixel set, the second pixel set One-to-one correspondence with the pixels in the third pixel set.
- the pixels of the target sketch image, the first mask and the background are in one-to-one correspondence. Therefore, after the electronic device determines the first set of pixels from the first mask, it can determine the second pixel from the target sketch image. set, and a third set of pixels is determined from the background.
- the electronic device determines, according to the mask value of the first pixel in the first pixel set, the pixel value of the second pixel in the second pixel set, and the pixel value of the third pixel in the third pixel set,
- the pixel value of the fourth pixel, the pixel value of the fourth pixel the mask value of the first pixel ⁇ the pixel value of the second pixel+the pixel value of the third pixel ⁇ (1-the first pixel mask value for one pixel).
- the third pixel in the background is updated to the fourth pixel to obtain the sub-image corresponding to the increase.
- FIG. 5 is a schematic diagram of a process of drawing sub-images in the video generation method provided by the present disclosure.
- the pixel value of each pixel on the background is 255, that is, the background is a pure white background.
- the pixels corresponding to a certain increase of the first mask are 6 pixels with a mask value of 1
- a polyline containing the 6 pixels is finally drawn on the background, and the gray values of the 6 pixels are 90, 0, 23, 23, 255, 89.
- the background when the target sketch image is not the sketch image with the lightest color among the plurality of sketch images, the background is that the background is adjacent to the target sketch image among the plurality of sketch images. , and the sketch image before the target sketch image; when the target sketch image is the sketch image with the darkest color among the plurality of sketch images, the background is a white image with the same size as the target sketch image .
- the sketch image 2 is obtained by continuing to draw the sketch image 1, and the sketch image 3 is obtained on the basis of the sketch image 2.
- the sketch image 1 is actually drawn on a white background. That is to say, the background of the sketch image 1 is a white background, the pixel value of each pixel of the white background is 255, the background of the sketch image 2 is the sketch image 1, and so on.
- sketch images at different stages correspond to different backgrounds, so as to accurately simulate the purpose of drawing sketch images by painters.
- each increment in the growth sequence of the first mask corresponds to the sub-images in the multiple sub-images one-to-one, and the growth sequence includes the growth sequence of the facial contours of the characters in the source image.
- the electronic device determines the increasing order of the mask values of the pixels in the first mask, firstly, extracts the face key points in the source image to obtain a key point set. Afterwards, the electronic device determines a second mask of the hair region of the person in the source image according to the source image.
- the electronic device determines the first convex hull area of the person's face according to the set of key points; according to the intersection of the first convex hull area and the second mask, from the first mask A second convex hull region is defined in the film. Finally, the electronic device sequentially connects the face contour key points in the key point set on the second convex hull region according to the stroke speed, to obtain the growth sequence of the human face contour in the source image, and the stroke speed is based on The duration of the video is determined.
- FIG. 6 is a schematic process diagram of the video generation method provided by the present disclosure.
- the electronic device uses the facial key point detection model to extract the facial feature key points and the facial contour key points, and uses the hair segmentation model to determine the second mask of the hair region of the person .
- FIG. 7 is a schematic diagram of a face key point in the video generation method provided by the present disclosure.
- the key points of the human face include the key points of 7 parts of the left eyebrow, the right eyebrow, the left eye, the right eye, the nose, the mouth and the outline of the human face.
- the numbers next to the key points in the figure represent the serial numbers of the key points. For example, there are 34 key points in the face contour, and the key points represented by the serial numbers 1 to 34 are the key points of the face contour.
- FIG. 8 is a schematic diagram of the second mask of the hair area in the video generation method provided by the present disclosure. Please refer to FIG. 8.
- the function of the second mask of the hair area is to avoid the problem of disordered sequence in the process of simulating painting, such as actual painting. When drawing the outline of the face first, then the eyebrows, then the eyes, and finally the hair, instead of the hair appearing first. Therefore, the use of the second mask can block the hair area to avoid hair appearing first in the simulation painting process, resulting in an unnatural simulation process.
- the electronic device When drawing a face contour, first determine the growth order of the face contour in the first mask. During the determination process, the electronic device determines the first convex hull region from the first mask according to the set of key points. For example, the electronic device will connect the face contour key points, the left eyebrow key points and the right eyebrow key points in sequence, and the first convex hull region can be obtained. After that, the electronic device determines the intersection of the first convex hull region and the second mask described in FIG. 8 , and then the second convex hull region can be determined.
- FIG. 9 is a schematic diagram of the first convex hull area and the second convex hull area in the video generation method provided by the present disclosure. Please refer to FIG. 9. Since there is an area blocked by hair in the first convex hull area, in order to avoid the first convex hull area in the simulation drawing process When the hair problem occurs, it is necessary to remove the area occluded by the hair in the first convex hull area, so as to obtain the second convex hull area. The position of the face can be determined from the background according to the second convex hull region. Next, it is necessary to draw the outline of the face, facial features, etc. at the position of the face in the background, and then draw the area other than the hair and the character, so that the target sketch image appears little by little on the background, thereby simulating the process of drawing the target sketch image .
- the process of drawing the face contour is actually a process in which the mask values of the pixels representing the face contour in the first mask are sequentially changed from 0 to 1, that is, the growth process of the pixels representing the face contour in the first mask.
- the growth process includes the growth sequence and the stroke speed.
- the growth sequence is the process of connecting the key points of the face contour in sequence. For example, the 1-34th face contour key points are connected in sequence to obtain the growth sequence of the face contour.
- the growth rate is determined by the length of the video. For example, in a 450-frame video, if you want to draw the outline of the face in 3 strokes, it is equivalent to increasing the first mask 3 times during the process of drawing the outline of the face, each time increasing by about 11 pixels.
- the outline of the face is drawn by simulating three strokes of the painter, corresponding to three sub-images, namely three image frames.
- three strokes of the painter corresponding to three sub-images, namely three image frames.
- you want to draw the outline of a human face in one second it is equivalent to increasing the first mask 30 times, which is equivalent to simulating the process of drawing the outline of a human face with 30 strokes by an artist.
- the mask value of the face contour position on the first mask is 1.
- the mask value of the face contour continues to remain 1.
- the electronic device determines the growth sequence of the outline of the human face, it also needs to determine the growth sequence of each facial feature area.
- the electronic device determines the key points of different areas of the face from the set of key points; performs interpolation on each area of the face according to the key points of each area; According to the stroke speed on the second convex hull region, the growth order of the corresponding regions in the face of the person in the source image is determined.
- the face contour is relatively simple, and the discount obtained by sequentially connecting the key points of the face can represent the face contour.
- the image obtained by sequentially connecting the key points of the facial features area cannot satisfy the realistic sketch image.
- Figure 7. After the two lines of key points on the left eyebrow are connected in sequence, there is a blank part in the middle. Obviously, this cannot meet the requirements of the sketch image.
- the key points of each row are connected in turn, it is obviously unreasonable to form two strokes to draw the left eyebrow. Therefore, for the facial features area, it is not enough to connect the key points in sequence.
- different facial features need to be interpolated in different ways. According to the interpolated area, in the area of the second convex hull in the background according to The brush stroke speed determines the value-added sequence corresponding to the facial features area, and prepares for the subsequent sub-strategy drawing of facial features.
- the key points of the left eyebrow include two rows of key points, and the key points of the upper row correspond to the key points of the next row one by one.
- the electronic The device determines the average value to interpolate the keypoints of the left eyebrow region into 3 rows of keypoints. Further, the electronic device can also interpolate the key points of the left eyebrow region into 4 lines or 5 lines, which is not limited in the embodiment of the present disclosure, and the number of lines is related to the precision. After interpolation, it takes seven or eight strokes to draw the left eyebrow, which can more reasonably simulate the painting process of the artist.
- the eyeball region is interpolated according to the circular region according to the key points of the eyeball region.
- the pupil cannot be drawn horizontally and horizontally. Therefore, for the eyeball area, use the circular interpolation method, first draw a circle representing the eyeball, and then interpolate small circles one by one until a solid circle is formed. During the interpolation process, the average value of the center of the eyeball and the points on the circumference is determined, the average value is taken as the radius, and a circle is interpolated with the center of the eyeball as the center. From the perspective of the first mask, when the first mask grows, the image formed by each growing pixel is not a polyline, but one or more circles.
- a plurality of curves are vertically interpolated according to the key points of the mouth region.
- the key points in the mouth area can only draw the outline of the mouth. After the outline of the mouth is completed, the interpolation is performed according to the vertical line when filling. From the point of view of the first mask, after the first mask completes the growth of the mouth contour, for the lips part, the image formed by the pixels each time increases is a vertical line.
- the last few frames of images are used to supplement the background areas other than the hair and the characters.
- 90 frames are used to generate the sketch image 1 in Figure 3
- the first 88 frames of the 90 frames are used to draw the outline of the face and facial features
- the 89th frame is used to add hair
- the 90th frame is used for adding hair.
- background areas other than people For adding background areas other than people.
- the electronic device when the electronic device generates a plurality of sketch images according to the source image, first, a grayscale image is generated based on the source image. Then, the electronic device determines a plurality of Gaussian kernels, the Gaussian kernels in the plurality of Gaussian kernels correspond to the sketch images in the plurality of sketch images one-to-one, and the size of the corresponding Gaussian kernel of the sketch image with darker color is larger than that of the color Gaussian kernel for lighter sketch images. Next, the electronic device performs Gaussian blur on the grayscale image according to the multiple Gaussian kernels, respectively, to obtain a Gaussian blurred image corresponding to each of the multiple Gaussian kernels. Finally, the electronic device generates the multiple sketch images according to the Gaussian blur map corresponding to each of the multiple Gaussian kernels and the grayscale image.
- the electronic device after acquiring the source image, the electronic device performs a noise reduction process such as median filtering on the source image. Afterwards, the electronic device performs grayscale processing on each pixel of the source image, thereby converting the source image into a grayscale image. After obtaining the grayscale image, the electronic device determines a Gaussian kernel (kernel) corresponding to the Gaussian blur according to Gaussian convolution, etc., and uses the Gaussian kernel to perform Gaussian blurring on the grayscale image to obtain a Gaussian blurred image. The larger the size of the Gaussian kernel, the darker the color of the Gaussian blur image obtained by using the Gaussian kernel to perform the Gaussian model on the grayscale image.
- a noise reduction process such as median filtering on the source image.
- the electronic device performs grayscale processing on each pixel of the source image, thereby converting the source image into a grayscale image.
- the electronic device determines a Gaussian kernel (kernel) corresponding to the Gau
- the electronic device uses the Gaussian blur image and the grayscale image as materials to generate a black and white sketch image. For example, the electronic device fuses the pixels of the Gaussian blur image with the corresponding pixels in the grayscale image to generate a sketch image.
- the source image, grayscale image, Gaussian blur image and sketch image have the same size and one-to-one pixel correspondence. Therefore, the electronic device can determine the pixel value of the corresponding pixel in the black and white sketch image according to the pixel value of the pixel in the grayscale image and the Gaussian blur image, and then obtain the sketch image. For example, after the Gaussian blur map is generated, the electronic device uses the dodge mode to extract the effect, that is, the following formula (1) is used for extraction:
- image_target represents the pixel value of the pixel in the sketch image
- image_gray represents the pixel value of the pixel in the grayscale image
- gray_blur1 represents the pixel value of the pixel in the Gaussian blur image.
- dodges based on different Gaussian kernels can obtain sketch images of different levels.
- FIG. 10 is a structural block diagram of a video generation apparatus according to an embodiment of the present disclosure.
- the device includes: an acquiring unit 11 , a first generating unit 12 , a second generating unit 13 and a third generating unit 14 .
- the acquiring unit 11 is used to acquire the source image.
- the first generating unit 12 is configured to generate a plurality of sketch images according to the source image, where the plurality of sketch images respectively correspond to sketch images of the source image with different color depths.
- the second generating unit 13 is configured to generate, based on the target sketch image, multiple sub-images of the target sketch image, where the multiple sub-images respectively correspond to the sketch images of the target sketch image under different drawing completion degrees, and the target sketch image
- the sketch image is any one of the plurality of sketch images.
- the third generating unit 14 is configured to use each sub-image of each sketch image in the plurality of sketch images as the video frame of the sketch drawing video, in the order of color depth from light to dark and the order of drawing completion degree from low to high
- the sequence of the video frames is set, and the sketching video is generated.
- the second generating unit 13 is configured to determine the increasing order of the mask values of the pixels in the first mask, and the initial mask value of the pixels in the first mask is 0,
- the growth of pixels in the first mask is used to indicate that the mask value of the pixels is changed from 0 to 1, and the first mask is used to make the background of the target sketch image gradually change to the target sketch image; according to the growth sequence, the target sketch image and the background of the target sketch image, the multiple sub-images are generated, and each increase in the growth sequence is the same as that in the multiple sub-images.
- the subgraphs correspond one by one.
- the second generation unit 13 when the second generation unit 13 generates the plurality of sub-images according to the growth order, the target sketch image, and the background of the target sketch image, it is used for the growth
- a first set of pixels corresponding to the increase is determined from the first mask
- a second set of pixels is determined from the target sketch image
- a third pixel set is determined from the background, and the first pixel set, the second pixel set and the pixels in the third pixel set are in one-to-one correspondence;
- the third pixel is updated to the fourth pixel
- the background when the target sketch image is not the sketch image with the lightest color among the plurality of sketch images, the background is the same as the target sketch image among the plurality of sketch images.
- the source image includes characters
- the increasing order includes an increasing sequence of face contours of the characters in the source image
- the second generating unit 13 determines the pixels in the first mask
- it is used to extract the face key points in the source image to obtain a set of key points; according to the source image, determine the second mask of the hair area of the character in the source image;
- the set of key points determine the first convex hull area of the person's face; according to the intersection of the first convex hull area and the second mask, determine from the second mask
- the second convex hull area; on the second convex hull area connect the face contour key points in the key point set in turn according to the stroke speed to obtain the growth sequence of the human face contour in the source image, and the stroke The speed is determined according to the duration of the video.
- the second generating unit 13 sequentially connects the face contour key points in the key point set on the second convex hull region according to the stroke speed to obtain the characters in the source image After the growth sequence of the face contour, it is also used to determine the key points of different regions of the human face from the set of key points; according to the key points of each region, each region of the human face is interpolated; In the area after interpolation, according to the stroke speed on the second convex hull area, the growth order of the corresponding areas in the face of the person in the source image is determined.
- the second generation unit 13 when the second generation unit 13 performs interpolation on different regions of the human face according to the key points of the regions, for the eyebrow region in the human face, according to the eyebrows
- the key points of the area are horizontally interpolated with multiple curves; for the eyeball area in the face area, according to the key points of the eyeball area, the eyeball area is interpolated according to the circular area; for the mouth in the face area region, and vertically interpolate multiple curves according to the key points of the mouth region.
- the first generating unit 12 is configured to generate a grayscale image based on the source image;
- the sketch images in the sketch images are in one-to-one correspondence, and the size of the corresponding Gaussian kernel of the sketch image with darker color is larger than that of the sketch image with lighter color;
- Gaussian blurring is performed on the image to obtain a Gaussian blur map corresponding to each of the multiple Gaussian kernels; according to the Gaussian blur map corresponding to each of the multiple Gaussian kernels and the grayscale map, the multiple Gaussian blur maps are generated.
- FIG. 11 is a schematic structural diagram of an electronic device for implementing an embodiment of the present disclosure.
- the electronic device 200 may be a terminal device or a server.
- the terminal device may include, but is not limited to, such as mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (Personal Digital Assistant, referred to as PDA), tablet computers (Portable Android Device, referred to as PAD), portable multimedia players (Portable Media Player, PMP for short), mobile terminals such as in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, and the like.
- PDA Personal Digital Assistant
- PAD tablet computers
- PMP portable multimedia players
- PMP Portable Media Player
- mobile terminals such as in-vehicle terminals (such as in-vehicle navigation terminals), etc.
- fixed terminals such as digital TVs, desktop computers, and the like.
- the electronic device shown in FIG. 11 is only an example, and should not
- the electronic device 200 may include a processing device (such as a central processing unit, a graphics processor, etc.) 201, which may be stored in a read only memory (Read Only Memory, ROM for short) 202 according to a program or from a storage device 208 is a program loaded into a random access memory (Random Access Memory, RAM for short) 203 to execute various appropriate actions and processes.
- ROM Read Only Memory
- RAM Random Access Memory
- various programs and data required for the operation of the electronic device 200 are also stored.
- the processing device 201, the ROM 202, and the RAM 203 are connected to each other through a bus 204.
- An Input/Output (I/O for short) interface 205 is also connected to the bus 204 .
- the following devices can be connected to the I/O interface 205: input devices 206 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a Liquid Crystal Display (LCD for short) ), speaker, vibrator, etc. output device 207; storage device 208 including, eg, magnetic tape, hard disk, etc.; and communication device 202.
- Communication means 202 may allow electronic device 200 to communicate wirelessly or by wire with other devices to exchange data. While FIG. 11 shows the electronic device 200 having various means, it should be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.
- embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart.
- the computer program may be downloaded and installed from the network via the communication device 202, or from the storage device 208, or from the ROM 202.
- the processing apparatus 201 When the computer program is executed by the processing apparatus 201, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are executed.
- the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two.
- the computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above.
- Computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable Read-Only Memory (Erasable Programmable Read-Only Memory, referred to as EPROM, or flash memory), optical fiber, portable compact disk read-only memory (Compact Disc Read-Only Memory, referred to as CD-ROM), optical storage devices, magnetic storage devices, or Any suitable combination of the above.
- a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
- a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with computer-readable program code embodied thereon. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
- a computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device .
- the program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to: electric wire, optical cable, radio frequency (RF for short), etc., or any suitable combination of the above.
- the above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or may exist alone without being assembled into the electronic device.
- the aforementioned computer-readable medium carries one or more programs, and when the aforementioned one or more programs are executed by the electronic device, causes the electronic device to execute the methods shown in the foregoing embodiments.
- Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, including object-oriented programming languages—such as Java, Smalltalk, C++, but also conventional Procedural programming language - such as the "C" language or similar programming language.
- the program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server.
- the remote computer can be connected to the user's computer through any kind of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or, can be connected to an external A computer (eg using an Internet service provider to connect via the Internet).
- LAN Local Area Network
- WAN Wide Area Network
- each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions.
- the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.
- each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations can be implemented in dedicated hardware-based systems that perform the specified functions or operations , or can be implemented in a combination of dedicated hardware and computer instructions.
- the units involved in the embodiments of the present disclosure may be implemented in a software manner, and may also be implemented in a hardware manner.
- the name of the unit does not constitute a limitation of the unit itself under certain circumstances, for example, the first obtaining unit may also be described as "a unit that obtains at least two Internet Protocol addresses".
- FPGA Field Programmable Gate Array
- ASIC Application Specific Integrated Circuit
- ASSP Application Specific Standard Parts
- SOC System on Chip
- CPLD Complex Programmable Logic Device
- a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with the instruction execution system, apparatus or device.
- the machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium.
- Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing.
- machine-readable storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), fiber optics, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.
- RAM random access memory
- ROM read only memory
- EPROM or flash memory erasable programmable read only memory
- CD-ROM compact disk read only memory
- magnetic storage or any suitable combination of the foregoing.
- a video generation method comprising: acquiring a source image; and generating a plurality of sketch images according to the source image, where the plurality of sketch images respectively correspond to Sketch images of the source image at different color depths; based on the target sketch image, generate multiple sub-images of the target sketch image, and the multiple sub-images respectively correspond to the sketches of the target sketch image under different rendering completion degrees image, the target sketch image is any one sketch image in the plurality of sketch images; each sub-picture of each sketch image in the plurality of sketch images is used as the video frame of the sketch drawing video, according to the color depth from light to The order of depth and the order of drawing completion from low to high set the order of the video frames to generate the sketch drawing video.
- the generating a plurality of sub-images of the target sketch image based on the target sketch image includes: determining an increasing order of mask values of pixels in a first mask, the first The initial mask value of the pixels in the mask is 0, the growth of the pixels in the first mask is used to indicate that the mask value of the pixels is changed from 0 to 1, and the first mask is used to make the target
- the background of the sketch image is gradually transformed into the target sketch image according to the increasing order; according to the increasing order, the target sketch image and the background of the target sketch image, the plurality of sub-images are generated, and in the increasing order
- Each increase of is in a one-to-one correspondence with the subgraphs in the plurality of subgraphs.
- the background when the target sketch image is not a sketch image with the lightest color among the plurality of sketch images, the background is a sketch image that is the same as the target sketch among the plurality of sketch images
- the sketch image that is adjacent to the image and is located before the target sketch image; when the target sketch image is the sketch image with the lightest color among the plurality of sketch images, the background is the size of the target sketch image. Same white image.
- the source image includes a person
- the increasing order includes an increasing order of face contours of the person in the source image
- the mask for determining the pixels in the first mask includes: extracting the key points of the face in the source image to obtain a set of key points; according to the source image, determining the second mask of the hair area of the person in the source image; according to the key point A set of points to determine the first convex hull area of the person's face; according to the intersection of the first convex hull area and the second mask, determine the second convex hull from the second mask area; on the second convex hull area, connect the facial contour key points in the set of key points sequentially according to the stroke speed, and obtain the growth sequence of the human face contour in the source image, and the stroke speed is based on the The length of the video is determined.
- the face contour key points in the key point set are sequentially connected on the second convex hull region according to the stroke speed to obtain the face of the person in the source image
- the growth sequence of the contour also includes: determining the key points of different regions of the human face from the set of key points; performing interpolation on each region of the human face according to the key points of each region; , on the second convex hull region, according to the stroke speed, determine the growth order of the corresponding regions in the face of the person in the source image.
- performing interpolation on different regions of the human face according to key points of the regions includes: for the eyebrow region in the human face, performing interpolation according to the eyebrow region of the human face.
- the key points are horizontally interpolated with multiple curves; for the eyeball region in the face region, according to the key points of the eyeball region, the eyeball region is interpolated according to the circular region; for the mouth region in the face region, According to the key points of the mouth region, a plurality of curves are vertically interpolated.
- the generating a plurality of sketch images according to the source image includes: generating a grayscale image based on the source image; determining a plurality of Gaussian kernels, the plurality of Gaussian kernels
- the Gaussian kernels in the sketch images correspond one-to-one with the sketch images in the plurality of sketch images, and the size of the corresponding Gaussian kernel of the sketch image with darker color is larger than that of the sketch image with lighter color
- the multiple Gaussian kernels kernel respectively perform Gaussian blur on the grayscale image to obtain a Gaussian blur map corresponding to each of the multiple Gaussian kernels; according to the Gaussian blur map corresponding to each of the multiple Gaussian kernels and the grayscale A degree map is generated to generate the plurality of sketch images.
- a video generation apparatus comprising:
- a first generating unit configured to generate a plurality of sketch images according to the source image, where the plurality of sketch images respectively correspond to sketch images of the source image under different color depths;
- the second generating unit is configured to generate, based on the target sketch image, multiple sub-images of the target sketch image, where the multiple sub-images respectively correspond to the sketch images of the target sketch image under different drawing completion degrees, and the target sketch image the image is any one of the plurality of sketch images;
- the third generating unit is configured to use each sub-image of each sketch image in the plurality of sketch images as a video frame of the sketch drawing video, and set in the order of color depth from light to dark and the order of drawing completion degree from low to high The sequence of the video frames to generate the sketching video.
- the second generating unit is configured to determine an increasing order of mask values of pixels in a first mask, where an initial mask value of pixels in the first mask is 0 , the growth of pixels in the first mask is used to indicate that the mask value of the pixel is changed from 0 to 1, and the first mask is used to make the background of the target sketch image gradually change according to the growth order is the target sketch image; according to the growth sequence, the target sketch image and the background of the target sketch image, the plurality of subgraphs are generated, and each increase in the increase sequence is related to the plurality of subgraphs The subgraphs correspond one-to-one.
- a first pixel set corresponding to the growth is determined from the first mask; according to the first pixel set, a second pixel set is determined from the target sketch image, and determine a third pixel set from the background, the first pixel set, the second pixel set and the pixels in the third pixel set are in one-to-one correspondence;
- the third pixel is updated to the fourth pixel, and the sub-picture corresponding to
- the background when the target sketch image is not a sketch image with the lightest color among the plurality of sketch images, the background is a sketch image that is the same as the target sketch among the plurality of sketch images
- the sketch image that is adjacent to the image and is located before the target sketch image; when the target sketch image is the sketch image with the lightest color among the plurality of sketch images, the background is the size of the target sketch image. Same white image.
- the source image includes characters
- the increasing order includes an increasing sequence of face contours of the characters in the source image
- the second generating unit determines In the increasing order of the mask values of the pixels, it is used to extract the key points of the face in the source image to obtain a set of key points; according to the source image, determine the second mask of the hair area of the person in the source image ; According to the set of key points, determine the first convex hull area of the person's face; According to the intersection of the first convex hull area and the second mask, determine from the second mask A second convex hull area is obtained; on the second convex hull area, the face contour key points in the key point set are sequentially connected according to the stroke speed, and the growth sequence of the human face contour in the source image is obtained.
- the stroke speed is determined according to the duration of the video.
- the second generating unit sequentially connects the face contour key points in the key point set on the second convex hull region according to the stroke speed, and obtains the image in the source image. After the growth sequence of the human face contours of the characters, it is also used to determine the key points of the different regions of the human face from the set of key points; according to the key points of each region, each region of the human face is interpolated; According to the interpolated area, on the second convex hull area, according to the stroke speed, determine the growth order of the corresponding area in the face of the person in the source image.
- the second generation unit when the second generation unit performs interpolation on different regions of the human face according to key points of the regions, for the eyebrow region in the human face, according to the The key points of the eyebrow area are horizontally interpolated with multiple curves; for the eyeball area in the face area, according to the key points of the eyeball area, the eyeball area is interpolated according to the circular area; for the eyeball area in the face area For the mouth area, a plurality of curves are vertically interpolated according to the key points of the mouth area.
- the first generating unit is configured to generate a grayscale image based on the source image
- the Gaussian kernels in the plurality of Gaussian kernels correspond to the sketch images in the plurality of sketch images one-to-one, and the size of the corresponding Gaussian kernel of the sketch image with darker color is larger than that of the sketch with lighter color
- Gaussian blurring is performed on the grayscale image respectively to obtain a Gaussian blurred image corresponding to each of the multiple Gaussian kernels; according to the Gaussian blurring corresponding to each of the multiple Gaussian kernels the blur map and the grayscale map to generate the plurality of sketch images.
- an electronic device comprising: at least one processor and a memory;
- the memory stores computer-executable instructions
- the at least one processor executes computer-implemented instructions stored in the memory to cause the at least one processor to perform the video generation method as described in the first aspect and various possible designs of the first aspect above.
- a computer-readable storage medium where computer-executable instructions are stored in the computer-readable storage medium, and when a processor executes the computer-executable instructions, The video generation method described above in the first aspect and various possible designs of the first aspect is implemented.
- a computer program product comprising: a computer program, the computer program being stored in a readable storage medium, at least one processor of an electronic device The computer program is read from the readable storage medium, the computer program being executed by the at least one processor to cause the electronic device to perform the video generation method as described in the first aspect and various possible designs of the first aspect above.
- a computer program that, when executed by a processor, implements the video generation method described in the first aspect and various possible designs of the first aspect. .
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Signal Processing (AREA)
- General Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Human Computer Interaction (AREA)
- Processing Or Creating Images (AREA)
- Image Processing (AREA)
- Indexing, Searching, Synchronizing, And The Amount Of Synchronization Travel Of Record Carriers (AREA)
- Television Signal Processing For Recording (AREA)
Abstract
Description
Claims (13)
- 一种视频生成方法,其特征在于,包括:获取源图像;根据所述源图像,生成多个素描图像,所述多个素描图像分别对应于所述源图像在不同颜色深度下的素描图像;基于目标素描图像,生成所述目标素描图像的多个子图,所述多个子图分别对应于所述目标素描图像在不同绘制完成度下的素描图像,所述目标素描图像是所述多个素描图像中的任意一个素描图像;将所述多个素描图像中各个素描图像的各个子图作为素描绘制视频的视频帧,按照颜色深度从浅到深的顺序以及绘制完成度从低到高的顺序设置所述视频帧的顺序,生成所述素描绘制视频。
- 根据权利要求1所述的方法,其特征在于,所述基于目标素描图像,生成所述目标素描图像的多个子图,包括:确定第一掩膜中像素的掩膜值的增长顺序,所述第一掩膜中像素的初始掩膜值为0,所述第一掩膜中像素的增长用于指示所述像素的掩膜值从0变更为1,所述第一掩膜用于使得所述目标素描图像的背景根据所述增长顺序逐渐转变为所述目标素描图像;根据所述增长顺序、所述目标素描图像和所述目标素描图像的背景,生成所述多个子图,所述增长顺序中的每次增长与所述多个子图中的子图一一对应。
- 根据权利要求2所述的方法,其特征在于,所述根据所述增长顺序、所述目标素描图像和所述目标素描图像的背景,生成所述多个子图,包括:针对所述增长顺序中的每次增长,从所述第一掩膜中确定出所述增长对应的第一像素集合;根据所述第一像素集合,从所述目标素描图像中确定出第二像素集合,并从所述背景中确定出第三像素集合,所述第一像素集合、所述第二像素集合和所述第三像素集合中的像素一一对应;根据所述第一像素集合中第一像素的掩膜值、所述第二像素集合中的第二像素的像素值、所述第三像素集合中第三像素的像素值,确定第四像素的像素值,所述第四像素的像素值=所述第一像素的掩膜值×所述第二像素的像素值+所述第三像素的像素值×(1-所述第一像素的掩膜值);将所述背景中的所述第三像素更新为所述第四像素,得到所述增长对应的子图。
- 根据权利要求2或3所述的方法,其特征在于,当所述目标素描图像不是所述多个素描图像中的颜色最浅的素描图像时,所述背景是所述多个素描图像中与所述目标素描图像相邻、且位于所述目标素描图像之前的素描图像;当所述目标素描图像是所述多个素描图像中颜色最浅的素描图像时,所述背景是尺寸与所述目标素描图像尺寸相同的白色图像。
- 根据权利要求2-4中任意一项所述的方法,其特征在于,所述源图像中包含人物,所述增长顺序包括所述源图像中人物的人脸轮廓的增长顺序,所述确定第一掩膜中像素的掩膜值的增长顺序,包括:提取所述源图像中的人脸关键点,得到关键点集合;根据所述源图像,确定所述源图像中人物的头发区域的第二掩膜;根据所述关键点集合,确定出所述人物的人脸的第一凸包区域;根据所述第一凸包区域和所述第二掩膜的交集,从所述第二掩膜中确定出第二凸包区域;在所述第二凸包区域上按照笔触速度依次连接所述关键点集合中的人脸轮廓关键点,得到所述源图像中人物人脸轮廓的增长顺序,所述笔触速度根据所述视频的时长确定出。
- 根据权利要求5所述的方法,其特征在于,所述在所述第二凸包区域上按照笔触速度依次连接所述关键点集合中的人脸轮廓关键点,得到所述源图像中人物的人脸轮廓的增长顺序之后,还包括:从所述关键点集合中确定出所述人脸不同区域的关键点;根据各区域的关键点,对所述人脸的各区域进行插值;根据插值后区域,在所述第二凸包区域上按照笔触速度,确定所述源图像中人物的人脸中对应区域的增长顺序。
- 根据权利要求6所述的方法,其特征在于,所述对所述人脸的不同区域,根据所述区域的关键点进行插值,包括:对于所述人脸中的眉毛区域,根据所述眉毛区域的关键点横向插值多条曲线;对于所述人脸区域中的眼球区域,根据所述眼球区域的关键点,按照圆形区域对所述眼球区域插值;对于所述人脸区域中的嘴巴区域,根据所述嘴巴区域的关键点,竖向插值多条曲线。
- 根据权利要求1-7中任意一项所述的方法,其特征在于,所述根据所述源图像,生成多个素描图像,包括:基于所述源图像,生成灰度图;确定多个高斯核,所述多个高斯核中的高斯核和所述多个素描图像中的素描图像一一对应,颜色较深的素描图像的对应的高斯核的尺寸大于颜色较浅的素描图像对应的高斯核;根据所述多个高斯核,分别对所述灰度图进行高斯模糊,得到所述多个高斯核中各高斯核对应的高斯模糊图;根据所述多个高斯核中各高斯核对应的高斯模糊图和所述灰度图,生成所述多个素描图像。
- 一种视频生成装置,其特征在于,包括:获取单元,用于获取源图像;第一生成单元,用于根据所述源图像,生成多个素描图像,所述多个素描图像分别对应于所述源图像在不同颜色深度下的素描图像;第二生成单元,用于基于目标素描图像,生成所述目标素描图像的多个子图,所述多个子图分别对应于所述目标素描图像在不同绘制完成度下的素描图像,所述目标素描图像是所述多个素描图像中的任意一个素描图像;第三生成单元,用于将所述多个素描图像中各个素描图像的各个子图作为素描绘制 视频的视频帧,按照颜色深度从浅到深的顺序以及绘制完成度从低到高的顺序设置所述视频帧的顺序,生成所述素描绘制视频。
- 一种电子设备,其特征在于,包括:至少一个处理器和存储器;所述存储器存储计算机执行指令;所述至少一个处理器执行所述存储器存储的计算机执行指令,使得所述至少一个处理器执行如权利要求1-8中任意一项所述的视频生成方法。
- 一种计算机可读存储介质,其特征在于,所述计算机可读存储介质中存储有计算机执行指令,当处理器执行所述计算机执行指令时,实现如权利要求1-8中任意一项所述的视频生成方法。
- 一种计算机程序产品,其特征在于,包括计算机程序,所述计算机程序在被处理器执行时实现如权利要求1-8中任意一项所述的视频生成方法。
- 一种计算机程序,其特征在于,所述计算机程序被处理器执行时实现如权利要求1-8中任意一项所述的视频生成方法。
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP22749175.0A EP4277261A4 (en) | 2021-02-05 | 2022-01-29 | VIDEO GENERATING METHOD AND APPARATUS, AS WELL AS APPARATUS AND READABLE STORAGE MEDIUM |
BR112023015702A BR112023015702A2 (pt) | 2021-02-05 | 2022-01-29 | Método e aparelho de geração de vídeo, dispositivo e meio de armazenamento legível |
JP2023547370A JP2024506014A (ja) | 2021-02-05 | 2022-01-29 | 動画生成方法、装置、機器及び可読記憶媒体 |
US18/264,232 US20240095981A1 (en) | 2021-02-05 | 2022-01-29 | Video generation method and apparatus, device and readable storage medium |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110163139.8A CN112995534B (zh) | 2021-02-05 | 2021-02-05 | 视频生成方法、装置、设备及可读存储介质 |
CN202110163139.8 | 2021-02-05 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2022166896A1 true WO2022166896A1 (zh) | 2022-08-11 |
Family
ID=76348275
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2022/075037 WO2022166896A1 (zh) | 2021-02-05 | 2022-01-29 | 视频生成方法、装置、设备及可读存储介质 |
Country Status (6)
Country | Link |
---|---|
US (1) | US20240095981A1 (zh) |
EP (1) | EP4277261A4 (zh) |
JP (1) | JP2024506014A (zh) |
CN (1) | CN112995534B (zh) |
BR (1) | BR112023015702A2 (zh) |
WO (1) | WO2022166896A1 (zh) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116260976A (zh) * | 2023-05-15 | 2023-06-13 | 深圳比特耐特信息技术股份有限公司 | 一种视频数据处理应用*** |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112995534B (zh) * | 2021-02-05 | 2023-01-24 | 北京字跳网络技术有限公司 | 视频生成方法、装置、设备及可读存储介质 |
CN113747136B (zh) * | 2021-09-30 | 2024-03-22 | 深圳追一科技有限公司 | 视频数据处理方法、装置、设备及介质 |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP3114587U (ja) * | 2005-07-08 | 2005-10-27 | アートコレクションハウス株式会社 | デッサン用絵画教材 |
CN109087553A (zh) * | 2018-08-23 | 2018-12-25 | 广东智媒云图科技股份有限公司 | 一种临摹绘画方法 |
CN109448079A (zh) * | 2018-10-25 | 2019-03-08 | 广东智媒云图科技股份有限公司 | 一种绘画引导方法及设备 |
CN109993810A (zh) * | 2019-03-19 | 2019-07-09 | 广东智媒云图科技股份有限公司 | 一种智能素描绘画方法、装置、存储介质及终端设备 |
CN112995534A (zh) * | 2021-02-05 | 2021-06-18 | 北京字跳网络技术有限公司 | 视频生成方法、装置、设备及可读存储介质 |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6021417A (en) * | 1997-10-31 | 2000-02-01 | Foto Fantasy, Inc. | Method of stimulating the creation of an artist's drawing or painting, and device for accomplishing same |
US6619860B1 (en) * | 1997-11-14 | 2003-09-16 | Eastman Kodak Company | Photobooth for producing digitally processed images |
WO2012093856A2 (en) * | 2011-01-04 | 2012-07-12 | Samsung Electronics Co., Ltd. | Method and apparatus for creating a live artistic sketch of an image |
CN102496180B (zh) * | 2011-12-15 | 2014-03-26 | 山东师范大学 | 一种自动生成水墨山水画图像的方法 |
CN107967667A (zh) * | 2017-12-21 | 2018-04-27 | 广东欧珀移动通信有限公司 | 素描的生成方法、装置、终端设备和存储介质 |
CN110599437A (zh) * | 2019-09-26 | 2019-12-20 | 北京百度网讯科技有限公司 | 用于处理视频的方法和装置 |
CN110738595B (zh) * | 2019-09-30 | 2023-06-30 | 腾讯科技(深圳)有限公司 | 图片处理方法、装置和设备及计算机存储介质 |
CN110717919A (zh) * | 2019-10-15 | 2020-01-21 | 阿里巴巴(中国)有限公司 | 图像处理方法、装置、介质和计算设备 |
-
2021
- 2021-02-05 CN CN202110163139.8A patent/CN112995534B/zh active Active
-
2022
- 2022-01-29 WO PCT/CN2022/075037 patent/WO2022166896A1/zh active Application Filing
- 2022-01-29 EP EP22749175.0A patent/EP4277261A4/en active Pending
- 2022-01-29 US US18/264,232 patent/US20240095981A1/en active Pending
- 2022-01-29 BR BR112023015702A patent/BR112023015702A2/pt unknown
- 2022-01-29 JP JP2023547370A patent/JP2024506014A/ja active Pending
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP3114587U (ja) * | 2005-07-08 | 2005-10-27 | アートコレクションハウス株式会社 | デッサン用絵画教材 |
CN109087553A (zh) * | 2018-08-23 | 2018-12-25 | 广东智媒云图科技股份有限公司 | 一种临摹绘画方法 |
CN109448079A (zh) * | 2018-10-25 | 2019-03-08 | 广东智媒云图科技股份有限公司 | 一种绘画引导方法及设备 |
CN109993810A (zh) * | 2019-03-19 | 2019-07-09 | 广东智媒云图科技股份有限公司 | 一种智能素描绘画方法、装置、存储介质及终端设备 |
CN112995534A (zh) * | 2021-02-05 | 2021-06-18 | 北京字跳网络技术有限公司 | 视频生成方法、装置、设备及可读存储介质 |
Non-Patent Citations (1)
Title |
---|
See also references of EP4277261A4 * |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116260976A (zh) * | 2023-05-15 | 2023-06-13 | 深圳比特耐特信息技术股份有限公司 | 一种视频数据处理应用*** |
Also Published As
Publication number | Publication date |
---|---|
US20240095981A1 (en) | 2024-03-21 |
BR112023015702A2 (pt) | 2023-11-07 |
JP2024506014A (ja) | 2024-02-08 |
CN112995534B (zh) | 2023-01-24 |
EP4277261A1 (en) | 2023-11-15 |
EP4277261A4 (en) | 2024-07-24 |
CN112995534A (zh) | 2021-06-18 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2022166896A1 (zh) | 视频生成方法、装置、设备及可读存储介质 | |
US20180025506A1 (en) | Avatar-based video encoding | |
US20190287283A1 (en) | User-guided image completion with image completion neural networks | |
US9314692B2 (en) | Method of creating avatar from user submitted image | |
CN110766777A (zh) | 虚拟形象的生成方法、装置、电子设备及存储介质 | |
KR20240050463A (ko) | 얼굴 재연을 위한 시스템 및 방법 | |
CN112017257B (zh) | 图像处理方法、设备及存储介质 | |
US10783713B2 (en) | Transmutation of virtual entity sketch using extracted features and relationships of real and virtual objects in mixed reality scene | |
CN112581635B (zh) | 一种通用的快速换脸方法、装置、电子设备和存储介质 | |
JP2023549841A (ja) | ビデオ処理方法、装置、電子機器及び記憶媒体 | |
CN111340921B (zh) | 染色方法、装置和计算机***及介质 | |
WO2023197780A1 (zh) | 图像处理方法、装置、电子设备及存储介质 | |
WO2022166907A1 (zh) | 图像处理方法、装置、设备及可读存储介质 | |
CN114842120A (zh) | 一种图像渲染处理方法、装置、设备及介质 | |
CN110860084A (zh) | 一种虚拟画面处理方法及装置 | |
CN115953597B (zh) | 图像处理方法、装置、设备及介质 | |
WO2021155666A1 (zh) | 用于生成图像的方法和装置 | |
CN112508772B (zh) | 图像生成方法、装置及存储介质 | |
CN114422698A (zh) | 视频生成方法、装置、设备及存储介质 | |
CN116501901A (zh) | 一种数字音乐图像展示方法、***及装置 | |
CN115714888B (zh) | 视频生成方法、装置、设备与计算机可读存储介质 | |
WO2023158375A2 (zh) | 表情包生成方法及设备 | |
WO2023158370A2 (zh) | 表情包生成方法及设备 | |
CN112053450B (zh) | 文字的显示方法、装置、电子设备及存储介质 | |
CN115512193A (zh) | 面部表情生成方法和装置 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 22749175 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 18264232 Country of ref document: US |
|
WWE | Wipo information: entry into national phase |
Ref document number: 202327052480 Country of ref document: IN Ref document number: 2023547370 Country of ref document: JP |
|
ENP | Entry into the national phase |
Ref document number: 2022749175 Country of ref document: EP Effective date: 20230807 |
|
REG | Reference to national code |
Ref country code: BR Ref legal event code: B01A Ref document number: 112023015702 Country of ref document: BR |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
WWE | Wipo information: entry into national phase |
Ref document number: 11202305881X Country of ref document: SG |
|
ENP | Entry into the national phase |
Ref document number: 112023015702 Country of ref document: BR Kind code of ref document: A2 Effective date: 20230804 |