CN112669198A - Image special effect processing method and device, electronic equipment and storage medium - Google Patents
Image special effect processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112669198A CN112669198A CN202011199693.3A CN202011199693A CN112669198A CN 112669198 A CN112669198 A CN 112669198A CN 202011199693 A CN202011199693 A CN 202011199693A CN 112669198 A CN112669198 A CN 112669198A
- Authority
- CN
- China
- Prior art keywords
- image
- target
- dimensional
- key point
- processed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000000694 effects Effects 0.000 title claims abstract description 118
- 238000003672 processing method Methods 0.000 title description 9
- 238000012545 processing Methods 0.000 claims abstract description 69
- 238000000034 method Methods 0.000 claims abstract description 35
- 230000011218 segmentation Effects 0.000 claims description 69
- 239000011159 matrix material Substances 0.000 claims description 28
- 230000003287 optical effect Effects 0.000 claims description 15
- 238000000605 extraction Methods 0.000 claims description 11
- 238000009499 grossing Methods 0.000 claims description 7
- 238000012946 outsourcing Methods 0.000 claims description 7
- 230000006641 stabilisation Effects 0.000 claims description 5
- 238000011105 stabilization Methods 0.000 claims description 5
- 239000000284 extract Substances 0.000 abstract description 2
- 210000000282 nail Anatomy 0.000 description 68
- 238000010586 diagram Methods 0.000 description 13
- 238000004422 calculation algorithm Methods 0.000 description 10
- 238000004891 communication Methods 0.000 description 10
- 238000005516 engineering process Methods 0.000 description 6
- 210000003813 thumb Anatomy 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000005236 sound signal Effects 0.000 description 4
- 239000002131 composite material Substances 0.000 description 3
- 210000004905 finger nail Anatomy 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 230000001133 acceleration Effects 0.000 description 2
- 210000005252 bulbus oculi Anatomy 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 210000000720 eyelash Anatomy 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 210000003128 head Anatomy 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 210000003811 finger Anatomy 0.000 description 1
- 210000004247 hand Anatomy 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000877 morphologic effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 210000004906 toe nail Anatomy 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/107—Static hand or arm
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Human Computer Interaction (AREA)
- Image Analysis (AREA)
- Processing Or Creating Images (AREA)
Abstract
The disclosure relates to a method and a device for processing image special effects, an electronic device and a storage medium, wherein the method comprises the following steps: extracting two-dimensional key point information of a target object from an image to be processed; projecting the three-dimensional model to a target area image of an image to be processed according to the two-dimensional key point information and the three-dimensional key point information corresponding to the two-dimensional key point information in the three-dimensional model of the target object; and carrying out special effect processing on the target area image to obtain a target special effect image. The method and the device extract the two-dimensional key point information of the target object, determine and obtain the target area image projected with the three-dimensional model from the image to be processed according to the two-dimensional key point information and the three-dimensional key point information, improve the recognition accuracy of the position of the target object in the image to be processed, and further perform special effect processing on the target area image, so that the problem that the target object in the target special effect image is staggered with the target object in the image to be processed is avoided, and the special effect of the target object is improved.
Description
Technical Field
The present disclosure relates to the field of image processing technologies, and in particular, to a method and an apparatus for processing an image special effect, an electronic device, and a storage medium.
Background
The virtual nail art is a new function of a short video application program or a camera application program, and the virtual nail art performs beautifying processing on nails in images.
In the related art, the current virtual nail art usually analyzes an image to identify an approximate area of a nail in the image, and then performs beautification processing such as recoloring and replacing a pattern on the approximate area of the nail to achieve nail art. However, the existing virtual nail art can only roughly acquire the approximate area of the nail, which causes the problem that the beautified nail is dislocated with the real nail, and the beautification effect is not ideal.
Disclosure of Invention
The disclosure provides a method and a device for processing image special effects, electronic equipment and a storage medium, which are used for at least solving the problem that the special effect of a target object in the related art is not real. The technical scheme of the disclosure is as follows:
according to a first aspect of the embodiments of the present disclosure, there is provided a method for processing an image special effect, including: acquiring an image to be processed containing a target object; extracting two-dimensional key point information of the target object from the image to be processed; projecting the three-dimensional model to a target area image of the target object in the image to be processed according to the two-dimensional key point information and three-dimensional key point information corresponding to the two-dimensional key point information in the three-dimensional model of the target object; and carrying out special effect processing on the target area image to obtain a target special effect image.
Optionally, the projecting the three-dimensional model to the target object on the target area image of the image to be processed according to the two-dimensional key point information and the three-dimensional key point information corresponding to the two-dimensional key point information in the three-dimensional model of the target object includes: calculating a camera external parameter matrix of the image to be processed according to the two-dimensional key point information and the three-dimensional key point information; and projecting the three-dimensional model to a target area image of the target object in the image to be processed according to the camera external parameter matrix and the camera internal parameter matrix.
Optionally, the extracting two-dimensional key point information of the target object from the image to be processed includes: inputting the image to be processed into a region segmentation model to obtain a target region image, wherein the target region image comprises a minimum outer-wrapping rectangle of the target object; and inputting the target area image into a key point regression model to obtain the two-dimensional key point information.
Optionally, the inputting the image to be processed into a region segmentation model to obtain the target region image includes: inputting the image to be processed into a first region segmentation model to obtain a first target region image; inputting the first target area image into a second area segmentation model to obtain the target area image; wherein the first target area image comprises the target area image, the first target area image comprises a minimum bounding rectangle of a first object, the target object being located in the first object.
Optionally, the inputting the image to be processed into a region segmentation model to obtain the target region image includes: and inputting the image to be processed into the region segmentation model to obtain a target region initial image containing the minimum outsourcing rectangle, and performing time sequence smoothing on the target region initial image to obtain the target region image.
Optionally, the inputting the target region image into a keypoint regression model to obtain the two-dimensional keypoint information includes: and inputting the target area image into the key point regression model to obtain two-dimensional initial key point information of the target object, and performing optical flow stabilization processing on the two-dimensional initial key point information to obtain the two-dimensional key point information.
Optionally, after the performing the special effect processing on the target area image to obtain the target special effect image, the method further includes: and replacing the target object of the image to be processed with the target special effect image to obtain a final effect image.
Optionally, after the target object of the image to be processed is replaced with the target special effect image to obtain a final effect image, the method further includes: extracting a mask region image of the target object from the image to be processed; and adjusting the position of the target special effect image in the final effect image until the distance between the position of the target special effect image in the final effect image and the position of the mask area image in the image to be processed is smaller than a preset distance threshold.
According to a second aspect of the embodiments of the present disclosure, there is provided an image special effect processing apparatus, including: an acquisition module configured to acquire an image to be processed containing a target object; the extraction module is configured to extract two-dimensional key point information of the target object from the image to be processed; the projection module is configured to project the three-dimensional model to a target area image of the target object in the image to be processed according to the two-dimensional key point information and three-dimensional key point information corresponding to the two-dimensional key point information in the three-dimensional model of the target object; and the special effect module is configured to perform special effect processing on the target area image to obtain a target special effect image.
Optionally, the projection module includes: the external parameter matrix calculation module is configured to calculate a camera external parameter matrix of the image to be processed according to the two-dimensional key point information and the three-dimensional key point information; a model projection module configured to project the three-dimensional model onto a target area image of the target object on the image to be processed according to the camera external parameter matrix and the camera internal parameter matrix.
Optionally, the extraction module includes: a segmentation module configured to input the image to be processed into a region segmentation model to obtain the target region image, where the target region image includes a minimum outsourcing rectangle of the target object; and the regression module is configured to input the target area image into a key point regression model to obtain the two-dimensional key point information.
Optionally, the segmentation module is configured to input the image to be processed into a first region segmentation model, so as to obtain a first target region image; inputting the first target area image into a second area segmentation model to obtain the target area image; wherein the first target area image comprises the target area image, the first target area image comprises a minimum bounding rectangle of a first object, the target object being located in the first object.
Optionally, the segmentation module is configured to input the image to be processed into the region segmentation model, obtain an initial image of a target region including the minimum outsourcing rectangle, and perform time-series smoothing on the initial image of the target region to obtain the image of the target region.
Optionally, the regression module is configured to input the target area image into the keypoint regression model, obtain two-dimensional initial keypoint information of the target object, and perform optical flow stabilization processing on the two-dimensional initial keypoint information to obtain the two-dimensional keypoint information.
Optionally, the apparatus further comprises: and the replacing module is configured to replace the target object of the image to be processed with the target special effect image to obtain a final effect image after the special effect module performs the special effect processing on the target area image to obtain the target special effect image.
Optionally, the apparatus further comprises: the fine adjustment module is configured to extract a mask area image of the target object from the image to be processed after the target object of the image to be processed is replaced by the target special effect image through the replacement module to obtain a final effect image; and adjusting the position of the target special effect image in the final effect image until the distance between the position of the target special effect image in the final effect image and the position of the mask area image in the image to be processed is smaller than a preset distance threshold.
According to a third aspect of the embodiments of the present disclosure, there is provided an electronic apparatus including: a processor; a memory for storing the processor-executable instructions; wherein the processor is configured to execute the instructions to implement the method for processing image effects according to the first aspect.
According to a fourth aspect of embodiments of the present disclosure, there is provided a storage medium, wherein instructions, when executed by a processor of an electronic device, enable the electronic device to perform the processing method of image special effects according to the first aspect.
According to a fifth aspect of embodiments of the present disclosure, there is provided a computer program product comprising readable program code executable by a processor of an electronic device to perform the method for processing an image special effect of the first aspect.
The technical scheme provided by the embodiment of the disclosure at least brings the following beneficial effects:
in the embodiment of the disclosure, two-dimensional key point information of a target object is extracted from an image to be processed containing the target object, and then a three-dimensional model is projected onto a target area image of the target object in the image to be processed according to the two-dimensional key point information and three-dimensional key point information corresponding to the two-dimensional key point information in the three-dimensional object model of the target object, so that a target special effect image is obtained by performing special effect processing on the target area image. The embodiment of the disclosure combines the two-dimensional key point information of the target object and the corresponding three-dimensional key point information in the three-dimensional model, projects the three-dimensional model onto the target area image of the image to be processed, and further performs special effect processing on the target area image. The method comprises the steps of firstly extracting two-dimensional key point information of a target object, determining a target area image projected with a three-dimensional model from an image to be processed according to the two-dimensional key point information and the three-dimensional key point information of the target object, and further carrying out special effect processing on the target area image. The target area image is determined according to the two-dimensional key point information and the three-dimensional key point information, and the recognition accuracy of the position of the target object in the image to be processed is improved, so that the problem that the target object in the target special effect image is staggered with the target object in the image to be processed is avoided, and the special effect of the target object is improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure and are not to be construed as limiting the disclosure.
Fig. 1 is a flowchart illustrating a method for processing image special effects according to an exemplary embodiment.
FIG. 2a is a schematic illustration of a nail makeup protocol according to an exemplary embodiment.
FIG. 2b is a schematic flow diagram illustrating a three-dimensional model projected onto an original image in a nail makeup scenario, according to an exemplary embodiment.
FIG. 3a is an original image of a nail makeup scenario, according to an exemplary embodiment.
FIG. 3b is an image of a hand region in a nail makeup scenario, according to an exemplary embodiment.
FIG. 3c is a nail segmentation image in a nail makeup scheme, according to an exemplary embodiment.
FIG. 3d is a composite image of a nail segmentation image and a hand region image in a nail makeup scheme, according to an example embodiment.
FIG. 3e is an image of a nail region in a nail makeup protocol, according to an exemplary embodiment.
FIG. 3f is a key point diagram of a nail in a nail make-up program according to an exemplary embodiment.
FIG. 3g is a schematic diagram illustrating a three-dimensional model in a nail makeup scenario, according to an exemplary embodiment.
FIG. 3h is a diagram illustrating projected effects in a nail makeup protocol according to an exemplary embodiment.
Fig. 3i and 3j are two makeup effect diagrams in a nail makeup scenario according to an exemplary embodiment.
Fig. 4 is a block diagram illustrating an image special effect processing apparatus according to an exemplary embodiment.
FIG. 5 is a block diagram illustrating one type of image special effects processing electronics, according to an exemplary embodiment.
FIG. 6 is a block diagram illustrating an electronic device for processing image special effects in accordance with one illustrative embodiment.
Detailed Description
In order to make the technical solutions of the present disclosure better understood by those of ordinary skill in the art, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.
It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in sequences other than those illustrated or otherwise described herein. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.
Fig. 1 is a flowchart illustrating a method for processing an image special effect according to an exemplary embodiment, and the method for processing an image special effect may be applied to a terminal or a server, as illustrated in fig. 1. The image special effect processing method can comprise the following steps.
In step S11, an image to be processed containing the target object is acquired.
In an embodiment of the present disclosure, the target object may be a nail, and the nail may include a fingernail and a toenail. Besides, the target object may be an eyeball, an eyelash, a lip, and the like. The embodiments of the present disclosure take the target object as a nail as an example, and when the target object is an eyeball, an eyelash, a lip, or the like, the processing method of the image special effect may be performed with reference to the embodiments of the present disclosure. In practical applications, the image to be processed may include one or more target objects, and the image to be processed may be understood as a two-dimensional image to be processed.
In step S12, two-dimensional keypoint information of the target object is extracted from the image to be processed.
In the embodiment of the disclosure, a target object may be segmented from an image to be processed, and then feature extraction may be performed on the target object to obtain two-dimensional key point information.
In step S13, the three-dimensional model is projected onto the target area image of the target object in the image to be processed according to the two-dimensional keypoint information and the three-dimensional keypoint information corresponding to the two-dimensional keypoint information in the three-dimensional model of the target object.
In the embodiment of the present disclosure, the three-dimensional model may be a trained model, and the training process of the three-dimensional model and the like are not specifically limited in the embodiment of the present disclosure. The three-dimensional key point information of the two-dimensional key point information in the three-dimensional model may be preset key point information. Typically, one two-dimensional keypoint information corresponds to one three-dimensional keypoint information in the three-dimensional model. And combining the two-dimensional key point information and the three-dimensional key point information, and projecting the three-dimensional model to a target area image of a target object in the image to be processed by utilizing the corresponding relation between the position of the two-dimensional key point information in the image to be processed and the position of the three-dimensional key point information in the three-dimensional model.
In step S14, a special effect process is performed on the target area image to obtain a target special effect image.
In the embodiment of the present disclosure, the target object may be included in the target area image, and the target object exists in a three-dimensional manner in the target area image. The specific effect processing on the target area image can be understood as beautifying processing on a target object existing in a three-dimensional mode, and the target specific image is a beautified three-dimensional target object. In practical applications, the beautification processing may include changing a color of the target object, changing a pattern of the target object, and the like, and embodiments of the present disclosure do not specifically limit the content of the beautification processing and the technical means employed.
In the embodiment of the disclosure, two-dimensional key point information of a target object is extracted from an image to be processed containing the target object, and then a three-dimensional model is projected onto a target area image of the target object in the image to be processed according to the two-dimensional key point information and three-dimensional key point information corresponding to the two-dimensional key point information in the three-dimensional object model of the target object, so that a target special effect image is obtained by performing special effect processing on the target area image. The embodiment of the disclosure combines the two-dimensional key point information of the target object and the corresponding three-dimensional key point information in the three-dimensional model, projects the three-dimensional model onto the target area image of the image to be processed, and further performs special effect processing on the target area image. The method comprises the steps of firstly extracting two-dimensional key point information of a target object, determining a target area image projected with a three-dimensional model from an image to be processed according to the two-dimensional key point information and the three-dimensional key point information of the target object, and further carrying out special effect processing on the target area image. The target area image is determined according to the two-dimensional key point information and the three-dimensional key point information, and the recognition accuracy of the position of the target object in the image to be processed is improved, so that the problem that the target object in the target special effect image is staggered with the target object in the image to be processed is avoided, and the special effect of the target object is improved.
In an exemplary embodiment of the disclosure, in the execution process of the above step S13, when projecting the three-dimensional model onto the target object on the target area image of the image to be processed, the camera external parameter matrix may be calculated according to the two-dimensional key Point information and the three-dimensional key Point information based on a N-Point Perspective (PNP) algorithm. The PNP algorithm aims to solve the three-dimensional-two-dimensional point-to-point motion method. In short, how to estimate the pose of the camera (i.e. the pose of the camera in coordinate system a) is known for n three-dimensional space point coordinates (relative to some specified coordinate system a) and its two-dimensional projection position. The camera external reference matrix is used for describing the motion of the camera in a static scene or the rigid motion of a moving object when the camera is fixed. The camera external reference matrix may include a rotation matrix describing directions of coordinate axes of the world coordinate system relative to camera coordinate axes and a translation matrix describing a position of a spatial origin under the camera coordinate system. In practical applications, since the number of key points of the nail is small, the P3P algorithm (i.e., N — 3) may be used to generate the camera extrinsic parameter matrix. And then, projecting the three-dimensional model to a target object on a target area image of the image to be processed according to the camera external parameter matrix and the camera internal parameter matrix. The camera internal reference matrix is determined by a hardware structure of the camera and comprises a focal length and a principal point offset of the camera. The principal axis of the camera is the line perpendicular to the image plane and passing through the vacuum, and the focus of the principal axis and the image plane is called the principal point. The principal point offset is the position of the principal point location relative to the image plane. In practical application, a three-dimensional model can be projected onto a target area image of a target object in an image to be processed by means of a blender (a piece of open-source cross-platform all-around three-dimensional animation software which provides a series of animation short-film making solutions from modeling, animation, material, rendering, audio processing, video clipping and the like). And projecting the three-dimensional model onto a target area image of the target object on the image to be processed according to the camera internal parameter matrix and the camera external parameter matrix obtained by calculation, so that the accuracy of projecting the three-dimensional model onto the target area image can be ensured.
In an exemplary embodiment of the present disclosure, in the execution process of step S12, when the two-dimensional keypoint information of the target object is extracted from the image to be processed, the image to be processed may be input to a region segmentation model to obtain a target region image containing a minimum outsourcing rectangle of the target object, and then the target region image may be input to a keypoint regression model to obtain the two-dimensional keypoint information. The region segmentation model may include a first region segmentation model and a second region segmentation model that are trained in advance. In practical application, an image to be processed may be input into a first region segmentation model, a first region segmentation mask is obtained by performing semantic segmentation on the image to be processed by using the first region segmentation model, a first target region image is obtained by segmenting the image to be processed according to the first region segmentation mask, the first target region image is input into a second region segmentation model, the first target region image is subjected to semantic segmentation by using the second region segmentation model to obtain a second region segmentation mask, a target region image is obtained by segmenting the first target region image according to the second region segmentation mask, the target region image is input into a key point regression model, and feature extraction is performed on the target region image by using the key point regression model to obtain two-dimensional key point information of each target object. The first region segmentation model can perform semantic segmentation on the image to be processed to obtain a finger region image, the second region segmentation model can perform semantic segmentation on the hand region image to obtain a nail region image, and the key point regression model extracts two-dimensional key point information of each nail in the nail region image. And semantically segmenting the image to be processed into a target region image step by step through a first region segmentation model and a second region segmentation model, and extracting two-dimensional key point information from the target region image through a key point regression model. The image to be processed is segmented into the target area image step by step, so that the interference of a non-target object in the image to be processed on the extraction of the two-dimensional key points is reduced, and the accuracy of the extraction of the two-dimensional key points of the target object is improved.
In an exemplary embodiment of the disclosure, in order to make the output items of the region segmentation model and the keypoint regression model more stable, the image to be processed may be input into the region segmentation model to obtain a target region initial image, and the target region initial image may be smoothed to obtain the target region image. The target area image can be input into the key point regression model to obtain two-dimensional initial key point information of the target object, and the two-dimensional initial key point information is subjected to optical flow stabilization processing to obtain the two-dimensional key point information. Namely, according to the first region segmentation model, semantic segmentation is carried out on the image to be processed to obtain a first target region initial image, and then time sequence smoothing processing is carried out on the first target region initial image to obtain a first target region image. And similarly, performing semantic segmentation on the first target area image according to a second area segmentation model to obtain a second target area initial image, and then performing time sequence smoothing on the second target area initial image to obtain a target area image. When the two-dimensional key point information is obtained through extraction, feature extraction can be performed on the target area image according to a key point regression model and an optical flow algorithm to obtain the two-dimensional key point information. The optical flow algorithm can adopt a Lucas-Kanade optical flow algorithm (a two-frame differential optical flow estimation algorithm). The output item of the region segmentation model, namely the target region image, can be more stable through the time sequence smoothing processing. The output items of the key point regression model, namely the two-dimensional key point information, can be more stable through the optical flow algorithm.
In an exemplary embodiment of the present disclosure, after the execution of the above step S14, the target object of the image to be processed may be replaced with the target special effect image, resulting in the final effect image, and the final effect image may be further fine-tuned. In practical application, a mask region image of a target object can be extracted from an image to be processed, and the mask region image can be understood as an image containing a mask, wherein the mask is a binary image composed of 0 and 1. When a mask is applied in a certain function, the 1-value area is processed, and the masked 0-value area is not included in the calculation. The image mask is defined by specified data values, data ranges, limited or unlimited values, regions of interest, and annotation files, and any combination of the above options may also be applied as input to create the mask. And adjusting the position of the target special effect image in the final effect image until the distance between the position of the target special effect image in the final effect image and the position of the mask region image in the image to be processed is smaller than a preset distance threshold, wherein the finely adjusted target special effect image can enable the condition that the three-dimensional model shields the target object to have better robustness, and the final effect image is more vivid.
Based on the above description about an embodiment of a method for processing an image special effect, a nail makeup scheme is described below, which may involve a hand region segmentation model, a nail region segmentation model, and a nail key point regression model, as shown in fig. 2 a. And performing human hand semantic segmentation on the original image containing the fingernails by using a hand region segmentation model to obtain a hand region image. And carrying out nail semantic segmentation on the hand region image by using a nail region segmentation model to obtain a nail region image. And extracting the key point coordinates of each nail in the nail region image by using a nail key point regression model. And then applying a PNP algorithm, a blender tool and the like to make up the nail, and replacing the nail in the original image with the nail after the nail is made up to obtain a final nail beautifying image.
As shown in fig. 2b, a schematic flow chart of projecting a three-dimensional model to an original image in a nail makeup scheme is shown. And generating a camera external parameter matrix by utilizing a P3P algorithm according to the key point coordinates of the fingernails in the original images and the corresponding key point coordinates in the three-dimensional model. And then projecting the three-dimensional model to the nail region image of the nail on the original image by means of a blender tool according to the camera external parameter matrix, the camera internal parameter matrix and the three-dimensional model. In the above nail makeup scheme, the hand region image is obtained by segmenting the original image using the hand region segmentation model, and then the nail region image is obtained by segmenting the hand region image using the nail region segmentation model. That is to say, sequentially pass through the hand region segmentation model and the nail region segmentation model, divide the original image into the hand region image and the nail region image in turn, according to the position relation, the morphological relation, the affiliation relation and the like between the nail and the hand, divide step by step from the original image and obtain the nail region image, avoid directly identifying the approximate region of the nail from the original image, have promoted the accurate degree of nail in the identification original image.
Fig. 3a to 3j are schematic views showing a makeup scheme for a nail, and fig. 3a shows an original image including a head, an arm, a hand, a watch, a building, and the like. The original image is input to the hand region segmentation model, resulting in the hand region image shown in fig. 3 b. The hand region image mainly includes hands, and does not include the head, arms, watches, buildings, and the like in the original image. The hand region image is input to the nail region segmentation model, and the nail segmentation image shown in fig. 3c and the composite image of the nail segmentation image and the hand region image shown in fig. 3d are obtained. The composite image is cut to obtain a nail region image of the thumb nail as shown in fig. 3 e. Inputting the nail region image into the key point regression model to obtain a key point schematic diagram of the nail as shown in fig. 3f, wherein the key points are four points on the edge of the thumb nail: keypoint "0", keypoint "1", keypoint "2" and keypoint "3". Fig. 3g shows a schematic view of a three-dimensional model of the thumb nail. And according to the key points of the thumb nail and the corresponding three-dimensional key points of the key points in the three-dimensional model, projecting the three-dimensional model onto the target area image of the thumb nail in the original image to obtain a projection effect image as shown in fig. 3 h. The projected target area image is subjected to special effect processing to obtain two nail makeup effect diagrams as shown in fig. 3i and 3 j.
Fig. 4 is a block diagram illustrating an image special effect processing apparatus according to an exemplary embodiment. The image special effect processing device can be applied to a terminal or a server, and specifically, the image special effect processing device can include the following modules.
An acquisition module 41 configured to acquire an image to be processed containing a target object;
an extraction module 42 configured to extract two-dimensional key point information of the target object from the image to be processed;
a projection module 43 configured to project the three-dimensional model of the target object on a target area image of the image to be processed according to the two-dimensional key point information and three-dimensional key point information corresponding to the two-dimensional key point information in the three-dimensional model of the target object;
and the special effect module 44 is configured to perform special effect processing on the target area image to obtain a target special effect image.
In an exemplary embodiment of the present disclosure, the projection module 43 includes:
the external parameter matrix calculation module is configured to calculate a camera external parameter matrix of the image to be processed according to the two-dimensional key point information and the three-dimensional key point information;
a model projection module configured to project the three-dimensional model onto a target area image of the target object on the image to be processed according to the camera external parameter matrix and the camera internal parameter matrix.
In an exemplary embodiment of the present disclosure, the extraction module 42 includes:
a segmentation module configured to input the image to be processed into a region segmentation model to obtain the target region image, where the target region image includes a minimum outsourcing rectangle of the target object;
and the regression module is configured to input the target area image into a key point regression model to obtain the two-dimensional key point information.
In an exemplary embodiment of the present disclosure, the segmentation module is configured to input the image to be processed into a first region segmentation model, resulting in a first target region image; inputting the first target area image into a second area segmentation model to obtain the target area image; wherein the first target area image comprises the target area image, the first target area image comprises a minimum bounding rectangle of a first object, the target object being located in the first object.
In an exemplary embodiment of the disclosure, the segmentation module is configured to input the image to be processed into the region segmentation model, obtain an initial image of a target region including the minimum outsourcing rectangle, and perform time-series smoothing on the initial image of the target region to obtain the image of the target region.
In an exemplary embodiment of the disclosure, the regression module is configured to input the target area image into the keypoint regression model to obtain two-dimensional initial keypoint information of the target object, and perform optical flow stabilization processing on the two-dimensional initial keypoint information to obtain the two-dimensional keypoint information.
In an exemplary embodiment of the present disclosure, the apparatus further includes:
a replacing module, configured to replace the target object of the image to be processed with the target special effect image after the special effect module 44 performs the special effect processing on the target area image to obtain the target special effect image, so as to obtain a final effect image.
In an exemplary embodiment of the present disclosure, the apparatus further includes:
the fine adjustment module is configured to extract a mask area image of the target object from the image to be processed after the target object of the image to be processed is replaced by the target special effect image through the replacement module to obtain a final effect image; and adjusting the position of the target special effect image in the final effect image until the distance between the position of the target special effect image in the final effect image and the position of the mask area image in the image to be processed is smaller than a preset distance threshold.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
FIG. 5 is a block diagram illustrating one type of image special effects processing electronics, according to an exemplary embodiment. For example, the electronic device 500 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, an exercise device, a personal digital assistant, and the like.
Referring to fig. 5, electronic device 500 may include one or more of the following components: a processing component 502, a memory 504, a power component 506, a multimedia component 508, an audio component 510, an input/output (I/O) interface 512, a sensor component 514, and a communication component 516.
The processing component 502 generally controls overall operation of the electronic device 500, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing components 502 may include one or more processors 520 to execute instructions to perform all or a portion of the steps of the image effect processing methods described above. Further, the processing component 502 can include one or more modules that facilitate interaction between the processing component 502 and other components. For example, the processing component 502 can include a multimedia module to facilitate interaction between the multimedia component 508 and the processing component 502.
The memory 504 is configured to store various types of data to support operations at the electronic device 500. Examples of such data include instructions for any application or method operating on the electronic device 500, contact data, phonebook data, messages, images, videos, and so forth. The memory 504 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.
The power supply component 506 provides power to the various components of the electronic device 500. The power components 506 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the electronic device 500.
The multimedia component 508 includes a screen that provides an output interface between the electronic device 500 and a user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 508 includes a front facing camera and/or a rear facing camera. The front camera and/or the rear camera may receive external multimedia data when the electronic device 500 is in an operating mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
The audio component 510 is configured to output and/or input audio signals. For example, the audio component 510 includes a Microphone (MIC) configured to receive external audio signals when the electronic device 500 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may further be stored in the memory 504 or transmitted via the communication component 516. In some embodiments, audio component 510 further includes a speaker for outputting audio signals.
The I/O interface 412 provides an interface between the processing component 502 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.
The sensor assembly 514 includes one or more sensors for providing various aspects of status assessment for the electronic device 500. For example, the sensor assembly 514 may detect an open/closed state of the electronic device 500, the relative positioning of components, such as a display and keypad of the electronic device 500, the sensor assembly 514 may detect a change in the position of the electronic device 500 or a component of the electronic device 500, the presence or absence of user contact with the electronic device 500, orientation or acceleration/deceleration of the electronic device 500, and a change in the temperature of the electronic device 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 514 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
The communication component 516 is configured to facilitate wired or wireless communication between the electronic device 500 and other devices. The electronic device 500 may access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G, or 5G), or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
In an exemplary embodiment, the electronic device 500 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the above-described image effect processing methods.
In an exemplary embodiment, a storage medium comprising instructions, such as the memory 504 comprising instructions, executable by the processor 520 of the electronic device 500 to perform the method of processing an image effect described above is also provided. Alternatively, the storage medium may be a non-transitory computer readable storage medium, which may be, for example, a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.
In an exemplary embodiment, a computer program product is also provided, which comprises readable program code executable by the processor 520 of the electronic device 500 to perform the above-mentioned processing method of image special effects. Alternatively, the program code may be stored in a storage medium of the electronic device 500, which may be a non-transitory computer-readable storage medium, for example, ROM, Random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.
FIG. 6 is a block diagram illustrating an electronic device for processing image special effects in accordance with one illustrative embodiment. For example, the electronic device 600 may be provided as a server. Referring to fig. 6, electronic device 600 includes a processing component 622 that further includes one or more processors, and memory resources, represented by memory 632, for storing instructions, such as applications, that are executable by processing component 622. The application programs stored in memory 632 may include one or more modules that each correspond to a set of instructions. Further, the processing component 622 is configured to execute instructions to perform the processing method of image effects described above.
The electronic device 600 may also include a power component 626 configured to perform power management for the electronic device 600, a wired or wireless network interface 650 configured to connect the electronic device 600 to a network, and an input/output (I/O) interface 658. The electronic device 600 may operate based on an operating system stored in the memory 632, such as Windows Server, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, or the like.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims (10)
1. A method for processing image special effects is characterized by comprising the following steps:
acquiring an image to be processed containing a target object;
extracting two-dimensional key point information of the target object from the image to be processed;
projecting the three-dimensional model to a target area image of the target object in the image to be processed according to the two-dimensional key point information and three-dimensional key point information corresponding to the two-dimensional key point information in the three-dimensional model of the target object;
and carrying out special effect processing on the target area image to obtain a target special effect image.
2. The method according to claim 1, wherein the projecting the three-dimensional model onto the target object on the target area image of the image to be processed according to the two-dimensional keypoint information and corresponding three-dimensional keypoint information of the two-dimensional keypoint information in the three-dimensional model of the target object comprises:
calculating a camera external parameter matrix of the image to be processed according to the two-dimensional key point information and the three-dimensional key point information;
and projecting the three-dimensional model to a target area image of the target object in the image to be processed according to the camera external parameter matrix and the camera internal parameter matrix.
3. The method according to claim 1, wherein the extracting two-dimensional key point information of the target object from the image to be processed comprises:
inputting the image to be processed into a region segmentation model to obtain a target region image, wherein the target region image comprises a minimum outer-wrapping rectangle of the target object;
and inputting the target area image into a key point regression model to obtain the two-dimensional key point information.
4. The method according to claim 3, wherein the inputting the image to be processed into a region segmentation model to obtain the target region image comprises:
inputting the image to be processed into a first region segmentation model to obtain a first target region image;
inputting the first target area image into a second area segmentation model to obtain the target area image;
wherein the first target area image comprises the target area image, the first target area image comprises a minimum bounding rectangle of a first object, the target object being located in the first object.
5. The method according to claim 3, wherein the inputting the image to be processed into a region segmentation model to obtain the target region image comprises:
and inputting the image to be processed into the region segmentation model to obtain a target region initial image containing the minimum outsourcing rectangle, and performing time sequence smoothing on the target region initial image to obtain the target region image.
6. The method of claim 3, wherein the inputting the target region image into a keypoint regression model to obtain the two-dimensional keypoint information comprises:
and inputting the target area image into the key point regression model to obtain two-dimensional initial key point information of the target object, and performing optical flow stabilization processing on the two-dimensional initial key point information to obtain the two-dimensional key point information.
7. The method according to claim 1, wherein after the performing the special effect processing on the target area image to obtain the target special effect image, the method further comprises:
and replacing the target object of the image to be processed with the target special effect image to obtain a final effect image.
8. An apparatus for processing a special effect of an image, comprising:
an acquisition module configured to acquire an image to be processed containing a target object;
the extraction module is configured to extract two-dimensional key point information of the target object from the image to be processed;
the projection module is configured to project the three-dimensional model to a target area image of the target object in the image to be processed according to the two-dimensional key point information and three-dimensional key point information corresponding to the two-dimensional key point information in the three-dimensional model of the target object;
and the special effect module is configured to perform special effect processing on the target area image to obtain a target special effect image.
9. An electronic device, comprising:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the method of processing image effects of any of claims 1 to 7.
10. A storage medium, characterized in that instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the method of processing image effects according to any one of claims 1 to 7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011199693.3A CN112669198A (en) | 2020-10-29 | 2020-10-29 | Image special effect processing method and device, electronic equipment and storage medium |
| PCT/CN2021/105334 WO2022088750A1 (en) | 2020-10-29 | 2021-07-08 | Image generation method and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011199693.3A CN112669198A (en) | 2020-10-29 | 2020-10-29 | Image special effect processing method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112669198A true CN112669198A (en) | 2021-04-16 |
Family
ID=75402841
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011199693.3A Pending CN112669198A (en) | 2020-10-29 | 2020-10-29 | Image special effect processing method and device, electronic equipment and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN112669198A (en) |
| WO (1) | WO2022088750A1 (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113421182A (en) * | 2021-05-20 | 2021-09-21 | 北京达佳互联信息技术有限公司 | Three-dimensional reconstruction method and device, electronic equipment and storage medium |
| CN114359522A (en) * | 2021-12-23 | 2022-04-15 | 阿依瓦（北京）技术有限公司 | AR model placing method and device |
| WO2022088750A1 (en) * | 2020-10-29 | 2022-05-05 | 北京达佳互联信息技术有限公司 | Image generation method and electronic device |
| WO2024041637A1 (en) * | 2022-08-26 | 2024-02-29 | 北京字跳网络技术有限公司 | Special effect image generation method and apparatus, device, and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090262989A1 (en) * | 2007-07-12 | 2009-10-22 | Tatsuo Kozakaya | Image processing apparatus and method |
| CN109272579A (en) * | 2018-08-16 | 2019-01-25 | Oppo广东移动通信有限公司 | Makeups method, apparatus, electronic equipment and storage medium based on threedimensional model |
| CN110675489A (en) * | 2019-09-25 | 2020-01-10 | 北京达佳互联信息技术有限公司 | Image processing method and device, electronic equipment and storage medium |
| CN111047526A (en) * | 2019-11-22 | 2020-04-21 | 北京达佳互联信息技术有限公司 | Image processing method and device, electronic equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112669198A (en) * | 2020-10-29 | 2021-04-16 | 北京达佳互联信息技术有限公司 | Image special effect processing method and device, electronic equipment and storage medium |
-
2020
- 2020-10-29 CN CN202011199693.3A patent/CN112669198A/en active Pending
-
2021
- 2021-07-08 WO PCT/CN2021/105334 patent/WO2022088750A1/en active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090262989A1 (en) * | 2007-07-12 | 2009-10-22 | Tatsuo Kozakaya | Image processing apparatus and method |
| CN109272579A (en) * | 2018-08-16 | 2019-01-25 | Oppo广东移动通信有限公司 | Makeups method, apparatus, electronic equipment and storage medium based on threedimensional model |
| CN110675489A (en) * | 2019-09-25 | 2020-01-10 | 北京达佳互联信息技术有限公司 | Image processing method and device, electronic equipment and storage medium |
| CN111047526A (en) * | 2019-11-22 | 2020-04-21 | 北京达佳互联信息技术有限公司 | Image processing method and device, electronic equipment and storage medium |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022088750A1 (en) * | 2020-10-29 | 2022-05-05 | 北京达佳互联信息技术有限公司 | Image generation method and electronic device |
| CN113421182A (en) * | 2021-05-20 | 2021-09-21 | 北京达佳互联信息技术有限公司 | Three-dimensional reconstruction method and device, electronic equipment and storage medium |
| CN113421182B (en) * | 2021-05-20 | 2023-11-28 | 北京达佳互联信息技术有限公司 | Three-dimensional reconstruction method, three-dimensional reconstruction device, electronic equipment and storage medium |
| CN114359522A (en) * | 2021-12-23 | 2022-04-15 | 阿依瓦（北京）技术有限公司 | AR model placing method and device |
| CN114359522B (en) * | 2021-12-23 | 2024-06-18 | 阿依瓦（北京）技术有限公司 | AR model placement method and device |
| WO2024041637A1 (en) * | 2022-08-26 | 2024-02-29 | 北京字跳网络技术有限公司 | Special effect image generation method and apparatus, device, and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2022088750A1 (en) | 2022-05-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112669198A (en) | Image special effect processing method and device, electronic equipment and storage medium | |
| CN110991327A (en) | Interaction method and device, electronic equipment and storage medium | |
| CN107464253B (en) | Eyebrow positioning method and device | |
| CN104156915A (en) | Skin color adjusting method and device | |
| CN111047526A (en) | Image processing method and device, electronic equipment and storage medium | |
| CN109840939B (en) | Three-dimensional reconstruction method, three-dimensional reconstruction device, electronic equipment and storage medium | |
| CN110889382A (en) | Virtual image rendering method and device, electronic equipment and storage medium | |
| CN111241887B (en) | Target object key point identification method and device, electronic equipment and storage medium | |
| CN111626183B (en) | Target object display method and device, electronic equipment and storage medium | |
| WO2023045207A1 (en) | Task processing method and apparatus, electronic device, storage medium, and computer program | |
| CN107657590B (en) | Picture processing method and device and storage medium | |
| CN110211211B (en) | Image processing method, device, electronic equipment and storage medium | |
| CN111144266B (en) | Facial expression recognition method and device | |
| CN110490164B (en) | Method, device, equipment and medium for generating virtual expression | |
| CN111368796A (en) | Face image processing method and device, electronic equipment and storage medium | |
| CN110782532B (en) | Image generation method, image generation device, electronic device, and storage medium | |
| CN112509005B (en) | Image processing method, image processing device, electronic equipment and storage medium | |
| WO2023142645A1 (en) | Image processing method and apparatus, and electronic device, storage medium and computer program product | |
| CN112669381A (en) | Pose determination method and device, electronic equipment and storage medium | |
| CN110728621B (en) | Face changing method and device of face image, electronic equipment and storage medium | |
| CN114067085A (en) | Virtual object display method and device, electronic equipment and storage medium | |
| CN112347911A (en) | Method and device for adding special effects of fingernails, electronic equipment and storage medium | |
| CN114581525A (en) | Attitude determination method and apparatus, electronic device, and storage medium | |
| CN111988522B (en) | Shooting control method and device, electronic equipment and storage medium | |
| CN110502993B (en) | Image processing method, image processing device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |